Support for templated tests?

[jalfd]

Does something like this exist? Or would it be possible to add?

Suppose I need to run the same set of tests on a number of different types, effectively templating the tests on a type parameter.

I'm not sure what would be an appropriate syntax for it (Boost.Test uses Boost.MPL to create type lists, but adding such a dependency might be awkward, and it requires you to be comfortable with metaprogramming just to define your tests)

Perhaps something like this?

// indicate I want to define tests that can be templated on a number of different types
TEST_TEMPLATE()
{
    // specify a test case for some type T
    TEST_CASE_T(name, description, T)
    {
        // inside the test case, I can use T as a placeholder for the specific type
        std::vector<T> vec;
        REQUIRE(vec.size() == 0);
    }
    // Register the types I want to run the tests for
    FOR_TYPE(int);
    FOR_TYPE(float)
}

Or perhaps you can come up with a better syntax (or even better, the feature already exists ;))
In any case, the feature would be very useful.

[philsquared]

Hmm... I had responded to this by email previously and thought it had updated here. Apparently not.

Anyway, I had been saying that I've been toying with such an idea - and proposed some syntax for it.

However I'm still not sure if it is needed - or at least not at the TEST_CASE level.

Where I've needed templated tests I've just written a separate function template that contains a SECTION. Then I write a TEST_CASE that calls that template for all types I'm interested in.

There are some things that could be cleaner - such as getting the name of the type in the SECTION name/ description.

Recasting the example you gave this way gives us:

template
void verifyMyType( const std::string& typeName )
{
SECTION( "verify/" + typeName, "" )
{
std::vector vec;
REQUIRE( vec.size() == 0 );
}
}

TEST_CASE( "test my types", "instantiates templates for all supported types and verifies them" )
{
verifyMyType( "int" );
verifyMyType( "float" );
}

you could use a macro to wrap the verify function - so you didn't have to repeat the type name.
That's something that could possibly be moved into the library.
However I like the fact that you can do this stuff fairly naturally - without too much boilerplate - and all with no explicit framework support.

Of course if you are doing this sort of thing a lot the boilerplate does start to become significant. Maybe I just don't come across the need enough?

Thoughts?

[jcockhren]

I agree with the premise of the suggestion. Also, there's currently nothing stopping a user from adding their own utility macro like:

#define CATCH_CONFIG_MAIN
#include<catch.hpp>
#include<vector>

#define TEST_TYPE(NAME,DESCRIPTION,T) \
    SECTION(NAME,DESCRIPTION) {\
        T b = 0;\
        T a = 1;\
        CHECK(a!=b);\
        REQUIRE((b+a)==a);\
        std::vector<T> vec (4,100);\
        REQUIRE(vec.size()==4);\
   }

TEST_CASE("Case/1","Auxiliary Macro") {
        CHECK(1==0);
        TEST_TYPE("sub-section/1","For integer",myInteger);     
}

I think the power comes if there's a way to provide a list of types to generate tests. I'm not sure how this could be achieved using the variadic macro capability of C99: http://stackoverflow.com/questions/2632300/looping-through-macro-varargs-values

However, even when combined with variadic functions, it'll fall short given in order to extract an argument, one must already know the type. GCC officially-unofficially allows for variadic template lists (with 0x switch). If having boost as a dependent isn't a big deal, this could be simulated with boost::variant and doing a static visitation to extract the correct type.

Personally, I like the fact that Catch is self-contained and that the compile time is minimal. Also, just imagine if previous issues spit out compile time errors like those from boost! Don't get me wrong, boost is awesome but with all that metaprogramming debugging is a nightmare if you're just trying to use the library. end mini-rant

All that to say: I vote for waiting for 0x. Clearly, I've thought too hard about this.

[philsquared]

Thanks Jurnell,

Yes, I'm definitely against a dependency on boost - although pulling the pseudo-typelist out of boost::variant isn't that hard - especially if you hardcode the limit (so don't have to roll your own Boost.PP too). Actually I need something like this elsewhere (to flesh out the implementation of value generators).

But, going back to your suggestion - why do you need the type list anyway? Is it so you can use the same list of types in more than one place? Assuming you can't roll them all into one TEST_CASE with SECTIONs that may be a motivation.

But other than that my proposal effectively gives you type lists, albeit not as compactly.
But a (boost-variant-like) pseudo-typelist is not incompatible with that approach and could be used to forward on.

[jcockhren]

Well, I wouldn't go as far as calling it a suggestion. I was trying to point out a convenience that could stem from taking jalfd's idea a bit further and was probably reaching. Personally, I don't need any of that. I'm perfectly ok with calling my utility macros over and over.

[jalfd]

I agree that it's pretty easy to templatize individual tests.The problem is that it doesn't really scale. You effectively need to define two functions per test, instead of just one.

For larger groups of tests (in my case, I have three separate implementations of a particular component (all exposing the same interface, but with very different implementation strategies internally), and I'd like to write a suite of tests which can be applied to each of these implementations to verify that it has the expected semantics.

Imagine a test suite verifying that user-defined iterator classes, or perhaps container classes, behave as required by the standard/STL.
You've got a known, fixed API, and want to verify that some custom implementation follows the rules. That might be 20, 40, 100+ tests, all of which should be applied to the same couple of types.

I first tried writing the tests in a similar way to your example, but the overhead just made me give up halfway (and request this feature from you instead ;))

When I write unit tests, convenience really is king. If I have to write too much boilerplate code, I end up not writing tests at all.

So far, Catch has really been amazing in this respect. Being able to get rid of all my fixture classes from Boost.Test, and instead just put setup/teardown code inside a TEST_CASE with a bunch of SECTIONS inside it really makes things more convenient. But as it is now, I need to write a separate function, outside the test case, to handle the template case. And I need to manually maintain this coupling between two separate locations in the code (make sure that each function template is actually called from a corresponding TEST_CASE).

And when you have a few dozen tests, all of which you'd like to apply to the same common of types, there's just too much boilerplate code.

Just like Catch lets me write setup code in the test case, to be applied to every sections inside it, I'd like a solution which lets me define a single scope which specifies the set of types that should be applied to all tests defined inside it.

Regarding typelists, I considered something like it in my original suggestion, but I'd actually prefer a simpler solution. Boost.Test uses MPL type lists, and sure, it works, and it's powerful and elegant, but it adds more complexity to my test code than I'm comfortable with.

[philsquared]

Reducing/ eliminating boilerplate is certainly one of the central goals of Catch. I'm completely with you there.
However I'm also against making Catch a Kitchen Sink of all possible testing conveniences. That's one of the things that sunk cppunit in the end.

So my thinking had been, if you can do something without framework support with a little boilerplate, but which is, itself, fairly rare - then I prefer to not add explicit support to the framework.

However your last comments suggest (a) it may not be that rare and (b) may be more than a little boilerplate.

So I will look at this again. I just need to find some time to explore it a bit more. I'm not yet totally convinced that there's not a non-framework way to make this easier.

[jalfd]

It might help if you have tailor the feature for a specific use case (to avoid ending up with a "kitchen sink" solution). I'm with you in that "normal" ad-hoc/one-off template tests can easily be created as it is now, but for the specific case where you have a group of tests you'd like to all run on the same set of types (API conformance tests, perhaps), some Catch plumbing would be helpful.

But yeah, I'm not sure how it should be implemented or what the syntax should be.

In terms of boilerplate, I don't mind having to write a bit extra code, but losing locality really kills me. The moment I have to write code in two separate places (and make calls from one to the other), it becomes messy.

[philsquared]

How about this:

TEMPLATE_TEST_CASE_2( "proposal/template", "test a vector for all supported types", int, float )
{    
      std::vector<T> vec;
      REQUIRE(vec.size() == 0);
}

You'd have to have a different numbered macro for each number of templates (e.g. TEMPLATE_TEST_CASE_3( "name", "desc", int, float, std::string ) )

But it's a pretty straightforward implementation

[jalfd]

Yeah, that could work. Of course, it's kind of ugly and clumsy to have to specify the number of template types in the macro name itself, but if it significantly simplifies the implementation, that's worth a lot too.

[samaursa]

I hope I am not intruding. Unless I have misunderstood, you want to run the same test but on different template types? I ran into a similar issue and the simple solution was to make a template function that has all the test and have the test case call it with different types. I believe that is what Phil put down above but the syntax confused me.

I am with Phil on this one and I would like to see CATCH to be light without any dependencies on external libraries (one of my main reasons for not using any other library). I think what I just said was discussed above but I thought I would put another vote in :)

The TEMPLATE_TEST_CASE_# looks interesting. As long as the debugger is OK with it, I personally have no complaints, although I still prefer the explicit template function solution. No need to make CATCH one size fits all.

[philsquared]

Of course you're not intruding, @samaursa. Thanks for your input.
Unfortunately I've got behind on this (and other) issue(s) for various reasons.
I'm going to try and do a big catch-up (no pun intended) in the next week or so.

[dirvine]

Did you get anywhere with this Phil ?

[philsquared]

Unfortunately I've not really had a chance to look back this far for some time.
It's not completely forgotten and, now that 1.0 is finally out, I'm hoping to get back to some of these.

[dirvine]

That's Great Phil I converted some of our tests from Gtest to catch to see how simple it would be a for a reasonably large codebase to do this. It's all great but for value and type parametised tests. These are a good bit more difficult in catch and I wonder what your thoughts are. You can see a small example in this branch of our code if it helps, many of the other libs use parametised tests as you will see if it helps. Here is the branch https://github.com/maidsafe/MaidSafe-Common/tree/catch/src/maidsafe/common/tests (this has some tests that use catch and the remaining still configured for gtest)

[philsquared]

Hi David,

I had a browse through. I found a parameterised test: KeyValueBufferTestDiskMemoryUsage. Is that what you meant?
This thread is about templatised tests, which is slightly different. Did I miss any examples of this?

KeyValueBufferTestDiskMemoryUsage should be doable in Catch as it is.
I don't know if you have seen Generators? They're undocumented as yet as they're not really finished. With that in mind if you're happy to try them they should actually be simpler than the current gtest impl (no separate instantiation).

Something like:

https://gist.github.com/philsquared/5990252

[edit: this gist wouldn't current work due to: https://github.com//issues/21)]

Let me know if you need more help

[dirvine]

Hi Phil, this is really good and looks better and cleaner than the gtest mechanism. Gtest has 2 parametrisation types value parametrised (which this is and you have that covered by the looks of it, cool!) and type parametrised, it's both I was looking at thinking template tests would be more c++ than type parametrisation which seems like a non c++ way of doing things. The generators were a nice surprise though.

I will have a play with the gist you did for us (thanks a lot for that) and get back with the template / type parametrised issues shortly. Again great lib !!

[philsquared]

Thanks. As I say - bear in mind that Generators are still an experimental feature and may be subject to change (probably not drastically - any changes are likely to simplify the sort of usage I showed).

And, as I said earlier, type parameterised tests are on the radar at some point.

[philsquared]

Oh wait - I forgot about the biggest limitation with generators at the moment!

#21

Generators are not currently compatible with sections. The gist, as written, would have issues! Sorry about that.
If you don't use sections it should be fine.

[dirvine]

Thanks again Phil, we will certainly be testing this out and if possible switching to catch, happy to write that up in our blog when we can do it. Work like this deserves to be mentioned to get as many using it as possible. I really like your approach to this being a test suite and that's it, no mock etc. it will keep it clean and accurate. I will keep you posted on progress, good and bad as we try this out (btw conversion from gtest is very simple really).

[garethsb-ghost]

I've just been looking at converting a small suite of about 100 test cases from Boost.Test to Catch as an experiment, initially using shim macros that can be switched to either use Boost.Test or Catch, to prove things to the rest of the team. The use of type-lists to test a template with a range of types, with BOOST_AUTO_TEST_CASE_TEMPLATE (roughly 3% of test cases in this suite), is one of only two stumbling blocks I found, everything else has been reassuringly very smooth.

I agree with the comment from @samaursa on Dec 3, 2011 that TEMPLATE_TEST_CASE_# looks interesting. In the interests of keeping Catch core small, I'd be perfectly happy to to keep its implementation in my own code - @philsquared, do you have an implementation lying around?! (The alternative mapping in my shim layer to Boost.Test way of doing things would be easy I think.)

Best regards!

[garethsb-ghost]

OK, here's a first attempt of a complete implementation of CATCH_TEMPLATE_TEST_CASE_n in case it's useful to someone else. @philsquared, is this the sort of thing you were suggesting?

#define INTERNAL_CATCH_TEMPLATE_TEST_CASE_DECL( name, description, T ) \
    template<typename T> \
    void INTERNAL_CATCH_UNIQUE_NAME( ____C_A_T_C_H____T_E_M_P_L_A_TE____T_E_S_T____ )(); \
    CATCH_TEST_CASE( name, description )

#define INTERNAL_CATCH_TEMPLATE_TEST_CASE_SECTION( Tn ) \
        CATCH_SECTION( #Tn ) \
        { \
            INTERNAL_CATCH_UNIQUE_NAME( ____C_A_T_C_H____T_E_M_P_L_A_TE____T_E_S_T____ )<Tn>(); \
        }

#define INTERNAL_CATCH_TEMPLATE_TEST_CASE_DEFN( T ) \
    template<typename T> \
    void INTERNAL_CATCH_UNIQUE_NAME( ____C_A_T_C_H____T_E_M_P_L_A_TE____T_E_S_T____ )()

#define CATCH_TEMPLATE_TEST_CASE_1( name, description, T, T1 ) \
    INTERNAL_CATCH_TEMPLATE_TEST_CASE_DECL( name, description, T) \
    { \
        INTERNAL_CATCH_TEMPLATE_TEST_CASE_SECTION( T1 ) \
    } \
    INTERNAL_CATCH_TEMPLATE_TEST_CASE_DEFN( T )

#define CATCH_TEMPLATE_TEST_CASE_2( name, description, T, T1, T2 ) \
    INTERNAL_CATCH_TEMPLATE_TEST_CASE_DECL( name, description, T ) \
    { \
        INTERNAL_CATCH_TEMPLATE_TEST_CASE_SECTION( T1 ) \
        INTERNAL_CATCH_TEMPLATE_TEST_CASE_SECTION( T2 ) \
    } \
    INTERNAL_CATCH_TEMPLATE_TEST_CASE_DEFN( T )

// And so on...

For info, the largest type-list I found for a template test case in the (Boost.Test) test suite I'm looking at right now has 14 types, which is constructed from two other type-lists (used for earlier tests) of 8 and 6 types. So this is also an example of type-list reuse which @jalfd mentioned.

[philsquared]

I'm closing this issue in favour of a dedicated generators/ property based testing ticket, #850.
(That ticket covers type-parameterised tests, too).
Be sure to watch that issue for notifications if you're interested,