Added compile time testing support

[Teskann]

Added compile-time testing macros

Discussion: https://discord.com/channels/415851350384181248/529055331880271884

Description

Why should I test code at compile-time ?

Testing code during compilation is the only way for ensuring the code does
not lead to undefined behavior.
This is critical to make your program safe, prevent unexpected crashes and worse
consequences of undefined behavior.

How ?

Catch2 made it easy to test code at compile-time.

The usage is pretty similar to testing at runtime:

Runtime macro	Compile time macro
SECTION	CONSTEXPR_SECTION
REQUIRE	CONSTEXPR_REQUIRE
REQUIRE_FALSE	CONSTEXPR_REQUIRE_FALSE
CHECK	-
CHECK_FALSE	-

TEST_CASE("My compile time test")
{
    CONSTEXPR_SECTION("std::array subscript and size")
    {
        std::array v = {1, 2, 3, 4, 5};
        CONSTEXPR_REQUIRE( v.size() == 5 );
        CONSTEXPR_REQUIRE( v[2] == 3 );

    }; // <-- Don't forget the semicolon here!
}

All code inside the CONSTEXPR_SECTION will be evaluated during compilation.
If any of the CONSTEXPR_REQUIRE or CONSTEXPR_REQUIRE_FALSE fails,
it will cause a compilation error.

Warning

You cannot use REQUIRE or CHECK inside a CONSTEXPR_SECTION.
Be careful not to mistake CONSTEXPR_REQUIRE with STATIC_REQUIRE.
They both concern compilation-time testing, but they do not have the same
purpose.

Warning

You cannot nest CONSTEXPR_SECTIONs or put a SECTION inside a
CONSTEXPR_SECTION

Note

The code inside the CONSTEXPR_SECTION is also evaluated at
runtime. This way, it remains debuggable, and it contributes to the code
coverage analysis.

What can I test ?

You can test anything that can be evaluated in a constexpr function.
This will depend on your compiler and the C++ standard you are using.
C++20, C++23 and C++26 improved a lot the support for constexpr.
Take a look at the C++ compiler support on cppreference
to see what you can use.

Debug failing compile-time tests

Investigate from the compiler output

You can check the output of your compiler to find the failing assertion.
The failing line should be highlighted somewhere. If you cannot see any
line:

• Ensure the CONSTEXPR_SECTION runs only code that can be evaluated
  at compile-time. This will depend on your compiler and the C++ standard you
  are using.
• Ensure you did not use unsupported Catch2 macros inside the CONSTEXPR_SECTION.
  The only supported macros are CONSTEXPR_REQUIRE and CONSTEXPR_REQUIRE_FALSE.
• Ensure the code you wrote doesn't produce undefined behavior (UB cannot
  compile). Reading a range outside its bounds, dereferencing an invalid pointer,
  reading a variable after it has been destroyed, are widespread undefined
  behaviors. They will be all caught by these tests.

Investigate at runtime

Tip

If you want to debug the code in a CONSTEXPR_SECTION, you can simply replace
the CONSTEXPR_SECTION with a SECTION and the code will be evaluated at runtime
only instead. Remember to come back to the CONSTEXPR_SECTION once you are
done !

Compile time test report

At runtime, a CONSTEXPR_SECTION will add a section in the test report. These sections
provide a way to see what was tested during compilation in the test report.
For instance, given the following test case:

TEST_CASE("My other compile time test")
{
    SECTION("Any runtime section")
    {
        SECTION("Nested section")
        {
            CONSTEXPR_SECTION("First constexpr section")
            {
                // ...
            };
        };
        
        CONSTEXPR_SECTION("Second constexpr section")
        {
            // ...
        };
        CONSTEXPR_SECTION("Third constexpr section")
        {
            // ...
        }; 
        CONSTEXPR_SECTION("Fourth section")
        {
            // ...
        };           
    }
    
    SECTION("Another runtime section")
    {
        SECTION("Nested runtime section")
        {
            // ...
        }
    }    
}

The report would look like this:

• ✅ My other compile time test
  • ✅ Any runtime section
    • ✅ Nested section
      • ✅ [Passed during compilation] First constexpr section
    • ✅ [Passed during compilation] Second constexpr section
    • ✅ [Passed during compilation] Third constexpr section
    • ✅ [Passed during compilation] Fourth constexpr section
  • ✅ Another runtime section
    • ✅ Nested runtime section

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.08%. Comparing base (cd93d20) to head (c22969c).
⚠️ Report is 25 commits behind head on devel.

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #3027      +/-   ##
==========================================
+ Coverage   90.93%   91.08%   +0.14%     
==========================================
  Files         201      204       +3     
  Lines        8681     8731      +50     
==========================================
+ Hits         7894     7952      +58     
+ Misses        787      779       -8     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ZXShady]

Suggested change

	static_assert(
	[&] // error: CONSTEXPR_SECTION failure. Check the compiler
	// output to find the cause. You can make the section a
	// SECTION to investigate at runtime.
	{
	callable();
	return true;
	}(),
	static_assert((callable(),true));

I don't like lamdbas when a simple expression can do it should be simpler fpr the compiler

if we are really pendatic we can use void() to always use builtin comma

[Teskann]

Very good suggestion, thanks. I didn't think about this, it's definitely better 👍

[Teskann]

Fixed. I kept a message in the static assert to make the error clearer for the user. Even if the assert technically never fails (it's either true or unevaluable at compile time). At least compilers will spot this line and the users will see the error message in the code.

Regarding the cast to void, I considered it useless as callable is always a lambda that does not overload the comma operator.

Thanks for this suggestion !

[ZXShady]

Regarding the cast to void, I considered it useless as callable is always a lambda that does not overload the comma operator.

You are right

[ZXShady]

why the else?

couldn't it be if(false)?

[Teskann]

The idea here was to avoid side effects of the macro expansion. If it expands to a simple if, user can write an else right after the macro usage which should not be valid in my opinion.

It doesn't really matter here as the static assert fires. I can remove it

[Teskann]

@ZXShady fixed