Introduce extension API for dynamic test generation

[nipafx]

As it stands the creation of dynamic tests via test factory methods is, from a coding experience perspective, no improvement over simple looping:

    // should be executed for different points and distances
    void testDistanceComputation(Point p1, Point p2, double distance) {
        assertEquals(distance, p1.distanceTo(p2));
    }

    @Test
    void testDistanceComputations() {
        createTestData().forEach(testData -> 
            testDistanceComputation(datum.point1(), datum.point2(), datum.distance()));
        }
    }

    @TestFactory
    Stream<DynamicTest> testDistanceComputations() {
        return DynamicTest.stream(
            createTestData(),
            datum -> "Testing " + datum,
            datum -> testDistanceComputation(datum.point1(), datum.point2(), datum.distance()));
    }

I see a couple of ways in which extensions could improve this situation but for that an extension point for dynamic test generation is needed.

My intuition is to create an extension interface, whose implementations must be registered on the test class and would be called at exactly the same point where @TestFactory is resolved now. They would have to return a stream of dynamic tests, which were then processed like those of test factories. Ideally @TestFactory would use the same mechanism.

This should give extensions plenty of room to create tests, especially parameterized ones, much more conveniently.

Related Issues

• Allow hierarchical parameterized/dynamic tests #604
• Introduce Extension API for registering dynamic tests #1444

[smoyer64]

@nicolaiparlog

I felt the same frustration with dynamic tests ... at least they're reported within a container described by the @testfactory.

I'd like to be able to create dynamic parameterized tests declaratively in a manner similar to how JUnitParams and RestFuse operate. In JUnit 4, these have to be Runners to overcome the rule against parameterized test methods but the real magic is that the Runner can use the test method as a container and create more than one instance as determined by the number of "parameter sets".

I've put together a PoC for an AfterDiscoveryCallback that allows test extensions to alter the TestPlan before it becomes immutable. Please take a look at #354 and critique/comment. It also takes care of changing the discovered Set from a HashSet to one of the ordered implementations, so it would allow Extensions for scenario, flow, use-case tests (any test class that requires methods to be strictly ordered).

[jbduncan]

It seems to me that this discussion would benefit with input from the Guava team, just like how I suggested in #330. They use JUnit 3's TestSuite - which is, by my understanding, equivalent to JUnit 5's dynamic tests - to implement the testers found in guava-testlib, so I think their input would be invaluable.

However I'm still rather new to participating in open-source projects, so I don't know if there's a socially accepted way of inviting them for feedback.

@sbrannen, may I have your thoughts on this please?

[nipafx]

It looks like #276 tries to achieve the same goal but takes a different route. I like this proposal better (obviously right 😉 ) because it does not alter existing tests.

It is connected to #354, as well, which also works via mutation (which I generally view with suspicion) but takes a much broader approach and is much more general.

[smoyer64]

@nicolaiparlog

I originally rejected mutating the engine's test descriptor by using the following signature:

TestDescriptor afterDiscovery(TestDescriptor testDescriptor) throws Exception;

I still prefer passing immutable objects around and would still have no problem chaining these methods together as you just need to feed the results of one into the next. I can't point to it now but I decided (for the PoC) to just mutate the TestDescriptor based on a the way some of the other JUnit 5 code was written.

Issue #354 was simply an implementation of what I was describing in #276. For JUnitParams, I don't think it would actually alter the test but rather would take the provided test method and "multiply it inside a new container". Fortunately, JUnit 5 is very hierarchy friendly!

[nipafx]

Since test templates were introduced in #642, a major use cases for a dynamic test extension point (parameterized tests) has a better solution. This begs the question whether this extension point is still needed. I think it is.

One important reason is that I really like how Jupiter uses extension points to implement its features. It makes the entire engine very customizable and it would be weird if this would not extend to dynamic tests. Rather than arguing for that I would expect an argument why dynamic tests are so unique that an extension point is not feasible.

The major technical reason is that test templates are bound to individual methods, which is a considerable limitation. Take, for example, a hypothetical extension that wanted to organize tests by the state they operate on:

• state A
  • test X
  • test Y
  • test Z
• state B
  • test X
  • test Y
  • test Z

While test templates allow running X, Y, and Z on A and B the appearance would be traversed:

• test X
  • state A
  • state B
• test Y
  • state A
  • state B
• test Z
  • state A
  • state B

After thinking about this for a while, it feels like there is a generalization of test templates and dynamic tests hidden somewhere. It joins the test template's getAdditionalExtensions (that was an awesome idea!) and lifecycle integration with dynamic test's freedom via Executable.

[nipafx]

I had a discussion with @sormuras, in which he convinced me that dynamic tests are not quite what I thought they were. Instead of seeing them as a way to easily create complex test plans at run time (which they are currently not and can't be without implementing this issue and #378) it makes more sense to use them to break longer tests into smaller pieces - a nice-to-have, not important infrastructure.

As such I am inclined to close this issue. Comments?

[nipafx]

Looks like nobody strongly disagrees, so I'll close this.