Add mechanism for registering dynamic tests

[marcphilipp]

Further Resources

• Related to Introduce support for parameterized tests #14.

[jlink]

@mmerdes and I are starting on some experiments with it.

See branch https://github.com/junit-team/junit5/tree/issue58-dynamic-tests

[sbrannen]

Awesome!

Just skimmed the work in the branch, and... it's great that you guys are making progress here.

Looking forward to the result...

Good Luck!

[jlink]

I don't think we'll go much further with branch (
https://github.com/junit-team/junit5/tree/issue58-dynamic-tests) unless we
agree in the team that's the direction we want to follow.

Here's a summary of what we learned so far:

• Adding a lambda-based approach for registering and executing dynamic
  tests was rather straight-forward. We introduced a new annotation
  @Dynamic for methods that return Stream<DynamicTest>,
  Collection<DynamicTest> or Iterator<DynamicTest>. Those tests are
  created, registered and executed on the fly and annotated methods follow
  the same behaviour as test methods as for Before/AfterEach and parameter
  injection. We also adapted 'junit4-runner' accordingly and the tests are being shown by IntelliJ - we didn't check Eclipse, though.
• The lambda-based however do currently not have a life cycle / extension
  model of their own. Just reusing the standard model from test methods did
  not seem intuitive nor possible for various reasons (e.g. early binding of
  the test instance to a lambda at creation time). But maybe we don't need
  that.
• Re-running such a lambda-based test by using its unique ID has not been
  implemented (yet). We started doing it but realized that the resulting code
  (as we imagined it) would be rather involved. We think, that's due to the
  current engine's focus on a statically created test hierarchy which does
  not lend itself well to dynamically created tests. However, it would be
  possible to add this functionality.
• The biggest drawback in the branch's implementation is that we don't
  think parameterized tests through parameter injection (see #l14) can be
  done with the same approach and the same life cycle/extension model as
  @test annotated methods. Same goes for quick-check-like tests of the same
  kind. But maybe we're wrong.

Our suggestion is that you guys look at the programming model and the
implementation and tell us what alternatives you see.

[schauder]

Had a quick look. I like it for one big (at least for me) reason: It seems to easily allow writing Tests as closures. With a simple class using these dynamic tests, it should be able write tests like:

class MyTests extends ClosureBasedTestRegistry {{
    test("my test name"){() -> {
        // actual test goes here
    });
}}

[ttddyy]

This looks great. I played around a bit.
Just for showcasing, parameterized test with single param can be written like this:

@Dynamic
public Stream<DynamicTest> dynamicTest() {
  return DynamicTest.streamFrom(parameters(), param -> "test-" + param, this::testWithSingleParam);
}

public Iterator<String> parameters() {
  return Arrays.asList("foo", "bar", "baz").iterator();
}

public void testWithSingleParam(String param) {
  // parameterized test here...
}

this way, multiple parameter generation methods can be in a single class.
It's limited to only single param(unless generating collection params), but interesting way to write test. :)

[aburmeis]

Would be cool to have the opportunity to create dynamic tests by an extension. I thought about it and would like to allow an extension to replace a void method annotated @Dynamic to be replaced using an extension point (where replace can return either a Stream, Collection or Iterator of DynamicTest:

interface DynamicTestExtensionPoint extends ExtensionPoint {
    boolean supports(MethodInvocationContext methodInvocationContext, ExtensionContext extensionContext);
    Object replace(MethodInvocationContext methodInvocationContext, ExtensionContext extensionContext);
}

This would be very useful to migrate libraries like junit-dataprovider and make them an extension from the former JUnit4 Runner.

[sbrannen]

@aburmeis, thanks for bringing this up. I agree: JUnit 5 will likely need to provide support for two types of dynamic tests:

1. lambdas dynamically provided as a stream or collection (as is the case with the status quo for this PR)
2. method descriptors provided dynamically as a stream/collection or via a extension point

The latter would allow for a certain category of dynamic tests that are executed analogous to methods annotated with @Test, thereby benefiting from all standard lifecycle callbacks and extensions (including method parameter injection).

[marcphilipp]

Pull request for current work-in-progress submitted: #278.