Implement Property Based Testing/ Data Generators

[philsquared]

This has been a long-standing promise/ point of discussion and a number of issues have been raised that amount to this, so I thought I'd write up the definite feature request to direct them all to.

There are two strands to this:

1. Data generators, or parametrised tests: i.e. you want to re-use the same core set of assertions with a range (possible a wide range) or different inputs - including the cross product of multiple possible inputs. In some cases you want to specifically control the inputs, in others generate it as an increasing range, or randomly. There's a variation where different types are used to parameterise. While strictly speaking that's a different feature I suspect it's all going to get tied up in them same solution, so I'll include that here too. We can always unbundle it later, if necessary.

2. Building on (1) is the idea of Property-Based Testing. This is a more formally structured approach to working with generated ranges of inputs and also includes features such as shrinking (where failing inputs are automatically reduced to the simplest/ smallest possible failure). The tests themselves are for properties which are usually invariants that should always hold - although sometimes an alternate (e.g. simple or reference) implementation is compared against.

Support for generators was experimentally added in the very early days of Catch, but were never progressed. In fact quite a lot of work was done in support of them, and a few alternate implementations started. The main obstacle was interaction with the SECTION tracking infrastructure. That code was reworked with generators in mind a couple of years ago, and a new proof-of-concept generators implementation was successfully written against it. However by that time the goal was full property-based testing and the path there seemed tortuous in a C++03 constrained context, so the effort was deferred to such a time as we could at least rebase on C++11 (and ideally would leverage C++14 and beyond to allow for things like Ranges v3 to be supported in the generators part). C++11 rebasing was decided for Catch2 - with generators/ PBT being one of the main motivators. At time of this writing a proof-of-concept implementation of Catch2 exists which consists of a mostly rewritten core. That work has paused while the "Catch Classic" backlog is tamed but will resume again soon - with generators and PBT being one of the first big features to be worked on next.

I'll keep this issue open until the feature is ready so others can be closed in favour of it.

[capsocrates]

You asked me this in #558

I believe I have answered this query now (even if not entirely satisfactorily) - and objection to me closing it? (@capsocrates, if you're still watching?)

If you meant that Catch2 is an answer to the query, I suppose that's satisfactory enough. :)

[Quincunx271]

I found myself needing this, so I made a workaround:

template <typename F, typename... Args>
void parameterized_test(F fn, Args&&... args)
{
    // this for_each should do the equivalent of Boost Hana's hana::for_each
    // Alternatively, see implementation after
    detail::for_each(std::forward_as_tuple(std::forward<Args>(args)...), fn);
}

And it is used like so:

parameterized_test(
    [](std::string parameter) {
        // rest of test code here - haven't tried using SECTION, though
    }, "arguments", "that", "are", "passed"
);

This is working because the assertion macros work even inside a function. The error message on where a macro failed is also useful because it shows the file/line of where the lambda is defined, rather than in my utility header. I also know that CAPTURE works.

This might be useful in figuring out how to do parameterized tests.

One drawback is that this doesn't allow multiple parameters, but that can be implemented through some heavier TMP, or worked around with tuples and C++17's bind expressions / std::tie

---

An implementation of for_each:

    template <typename Tuple, typename F, std::size_t... Is>
    void for_each_impl(Tuple&& tuple, F&& fn, std::index_sequence<Is...>)
    {
        int unused[] = {
            (std::invoke(std::forward<F>(fn), std::get<Is>(std::forward<Tuple>(tuple))), 0)...};
        (void) unused;
    }

    template <typename Tuple, typename F>
    void for_each(Tuple&& tuple, F&& fn)
    {
        for_each_impl(std::forward<Tuple>(tuple), std::forward<F>(fn),
                                    std::make_index_sequence<std::tuple_size_v<Tuple>>{});
    }

[Trass3r]

Looking forward to this! Especially now that Catch2 finally surfaces.

[ArekPiekarz]

@Quincunx271 Your version doesn't work with SECTIONs, because each section is executed only once.
So if you nest parameterized_test with multiple GIVEN, WHEN, THEN, some of them will be invoked for only the first parameter.

@philsquared Do you have any fuzzy estimation when this feature may be available in Catch2? Is it in a ballpark of a few months or closer to a year?

[mlimber]

If you tell me how to do this with the catch2 branch, I will try it out on my project and contribute fixes and documentation if possible.

[horenmar]

@ArekPiekarz So far most of our estimates were wildly wrong, so I am going to give "when it is done". Sorry.

@mlimber You can't yet.

[ThirtySomething]

Added myself to #850 after closing #1177

[johnthagen]

This is really the one remaining feature I'd love to see in Catch, and I think it would round out the framework really well in terms of the Don't Repeat Yourself (DRY) philosophy.

One framework that does parametrizing well is pytest. While Python is a much different beast than C++11 (dynamically typed, monkey patching, etc), the overall API style that they use for parameterization could be helpful to look at.

An example:

import pytest

@pytest.mark.parametrize("test_input,expected", [
    ("3+5", 8),
    ("2+4", 6),
    ("6*9", 42),
])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Properties of this API I like:

1. Minimal repetition when adding extra input data
2. Clearly associating the input data with a test case (in this case through a @property)
3. The input data can be named in a meaningful way

[Leandros]

Any progress here? Might also want to take a look at how GoogleTest does it: https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#how-to-write-value-parameterized-tests

[greenrobot]

There are so many approaches to this. Not sure if this was discussed yet: Like SECTION, Catch2 could offer something like INIT_SECTION, which would add another section dimension for init (setting up the test). Thus, each INIT_SECTION would run all SECTIONs.

[philsquared]

I know it's been a while, but I've started looking at this again.
I've got an initial implementation up and running on a local branch - but I wanted to bikeshed the syntax a bit before going further.

Here's one of my test cases:

TEST_CASE("Generators") {

    auto i = GENERATE( values( { "a", "b", "c" } ) );

    SECTION( "one" ) {
        auto j = GENERATE( range( 8, 11 ) << 2 );
        std::cout << "one: " << i << ", " << j << std::endl;
    }
    SECTION( "two" ) {
        auto j = GENERATE( 3.141 << 1.379 );
        std::cout << "two: " << i << ", " << j << std::endl;
    }
}

particular things to note:

1. Use of << as a "sequencing operator. It means: after generating the values on the left, continue by generating the values on the right
2. The syntax for the values generator. It takes an initialiser list - which means you get the double-braced syntax.
3. Sequencing individual values effectively adds a single value generator. Is this too magic? Or a big convenience for a common case?
4. Would improving (1) make (3) a complete substitute for (2)?
5. GENERATE acts similarly to SECTION in that the whole test case is re-entered for each value generated (and the cross-product of all other GENERATEs and SECTIONs) - in fact they run on the same underlying mechanism.
6. values and range are instances of generators which can be composed (in this case by sequencing with <<)
7. All generators (including composite, sequenced, generators) allow random ordering and random sub-ranges.
8. Additional generators can be user supplied (although this may initially be prohibited to allow more implementation flexibility until it beds in).

[ArekPiekarz]

@philsquared Could you make it clear what is the output of your proposed solution?

[philsquared]

Excellent point, @ArekPiekarz - that would help, wouldn't it - thanks :-)

one: a, 8
one: a, 9
one: a, 10
one: a, 11
one: a, 2
two: a, 3.141
two: a, 1.379
one: b, 8
one: b, 9
one: b, 10
one: b, 11
one: b, 2
two: b, 3.141
two: b, 1.379
one: c, 8
one: c, 9
one: c, 10
one: c, 11
one: c, 2
two: c, 3.141
two: c, 1.379

[johnthagen]

@philsquared

Use of << as a "sequencing operator

If the generated values have overloads for the ostream operator (for example, user defined types), does this change the behavior?

[philsquared]

No, the << overloads are on a Generator type:

template<typename T>
auto operator << ( Generator<T>&& g1, T const& val ) -> Generator<T> {
    return { std::move(g1), value( val ) };
}

[philsquared]

@johnthagen no - no change to the table examples

[philsquared]

Should work with any types that are copy/ movable.
You might need to specialise the range generator if your type doesn't support +

[ekrieger001]

how would the report look like for such testcases? What would be the testcasenames?

[philsquared]

@ekrieger001 currently there is no change to the test names - but this relates to the open question, above, about capturing variable names.

I'm thinking that generated variable values should appear like section names. If we capture variable names. too, we could report them as "<name> = <value>". Otherwise we could do just "<value>", or something like, "For generated value(s): <first>, <second> ..." - where <first> etc would be the generated values.

One complication is that you might want to use the variable in a section name (as in the Cucumber example, earlier). In that case it would be a shame to have the value reported twice. Not sure there's an easy way around that.

[myrgy]

Hi guys,
would it be possible to use generators to test multiple implementations. which has no virtual interface - so I had to specify types.

like

TEST_CASE_T("Templated", std::vector<int>, std::queue<char>) {
  SECTION("push_back") {
    T s;
   s.push_back(1);
   }  
}

Thanks!

[Quincunx271]

There's currently no built-in generator that just takes something like a std::vector<T>. That would be helpful, since currently the only easy way to reuse the values from one GENERATE(...) in another is to extract out a std::initializer_list<T>. Doing so, I've been running into lifetime bugs on clang-6.0 or less.

I'm imagining something like this: GENERATE(each(container.begin(), container.end()))

[atomgalaxy]

Phil, are there plans for the "" bits mentioned above?

[horenmar]

@Quincunx271 That's a good idea that sadly got lost -- I'll give it a try later.

@atomgalaxy Not sure which bits you mean.

[atomgalaxy]

@horenmar Sorry, github seems to have eaten the markup.

I mean the printing the current value idea that @philsquared details above:

@ekrieger001 currently there is no change to the test names - but this relates to the open question, above, about capturing variable names.

I'm thinking that generated variable values should appear like section names. If we capture variable names. too, we could report them as "<name> = <value>". Otherwise we could do just "<value>", or something like, "For generated value(s): <first>, <second> ..." - where <first> etc would be the generated values.

One complication is that you might want to use the variable in a section name (as in the Cucumber example, earlier). In that case it would be a shame to have the value reported twice. Not sure there's an easy way around that.

[horenmar]

There have been no changes to how generators are handled in regards to stringification, so they are still anonymous until the user stringifies them manually (e.g. via CAPTURE).

[matthew-limbinar]

@horenmar: Can you identify what is left on this feature request? Seems like it's potentially closable.

[horenmar]

@matthew-limbinar Everything that makes it actually property testing. There is data generation, but no case reduction, and no property testing loop support.

[tsondergaard]

@horenmar: Can you identify what is left on this feature request? Seems like it's potentially closable.

You cannot address a particular combination that you would like to run. Lets say that I break a particular case in a large set of generated combinations.

1. The reporter doesn't report the virtual section the test failure - if it did it would be easier to identify the failure
2. I cannot specify on the command line the specific test I would like to run. This can be a big disadvantage if running the test across all parameters take much longer than running just the single case that is failing