When are test cases run without sections?

[joto]

I am a bit unclear on when the code in a TEST_CASE is run when there are nested SECTIONs. I can't find anything about the details in the documentation so I am not sure what the intended action is here and whether there is a bug or not.

Here is some example code:

TEST_CASE("run once with section") {
    printf("a0\n");
    SECTION("foo") {
        printf("a1\n");
    }
    printf("a2\n");
}

TEST_CASE("run twice with and without section") {
    int error = 0;

    printf("b0\n");
    try {
        printf("b1\n");
        SECTION("bar") {
            printf("b2\n");
            throw 1;
        }
        printf("b3\n");
    } catch (int e) {
        printf("b4\n");
        error = e;
    }
    printf("b5\n");

    REQUIRE(error > 0);
}

This prints out:

a0
a1
a2
b0
b1
b2
b4
b5
b0
b1
b3
b5

indicating that the first example was only run once, but the second was run twice, once without and once with the SECTION("bar") code.

See also #191.

[philsquared]

The way SECTION discovery works it has to execute the code up to a SECTION before it knows about it (under the hood SECTIONs are just if statements which look up in a table of what's run and what hasn't).
In your first example, after executing the SECTION Catch knows that it has run all the code in the test case, so does not need to re-enter.

In the second example, because you throw from the first SECTION the test case is ended early. Catch now doesn't know if there are any further SECTIONs that need to be executed, so it has to run it through again (skipping over the first SECTION) to see if it encounters any more. In this case it doesn't - so it executes all the code in the test case outside the SECTION.

At one point I had it so it always did the final, sectionless, run-through - for consistency (in fact it did the sectionless run-through first - that's the behaviour being discussed in the linked issue).
I had quite a bit of feedback to say that was unexpected - and sometimes a problem - and I found it inconvenient myself. So I pruned it back to the "only when necessary" case (with sectionless run, when necessary, only at the end) we have now.

I hope that clears it up?

[joto]

Thanks @philsquared. This clears up the behaviour, but it isn't really intuitive, so I think it should be in the documentation. Maybe in an extra section titled something like "How this all works internally. Reads this if you get unexpected interaction between your code and Catch macros."

[philsquared]

It's mostly covered here:
https://github.com/philsquared/Catch/blob/master/docs/tutorial.md#test-cases-and-sections

But I agree it doesn't specifically mention that this implies a run through with no sections under some circumstances. I'll try and get something added.
It may also be worth covering that ground in the ref docs, not just the tutorial.

Thanks for the feedback.