Introduce support for parameterized tests

[elygre]

Deliverables

• Introduce @TestTemplate and accompanying extension point.
• Determine if Parameter resolvers should be able to access the invocation context for test templates #704 should be included in M4.
• Determine if we need an @InvocationIndex annotation or TestTemplateInfo type for injecting the current invocation index, etc. into @Test, @BeforeEach, and @AfterEach methods.
  • See d2e3a4d
• Introduce junit-jupiter-params module
  • Merge initial contribution (Introduce junit-jupiter-params #728)
  • Javadoc
    • @since tags
  • API annotations
  • Add unit tests where missing (see TODOs)
  • Resolve TODOs
• Document @TestTemplate and accompanying extension point in User Guide and Release Notes
  • Provide example(s) in the documentation module and include in the User Guide. Will be addresses in Document @TestTemplate and accompanying extension point in User Guide #762
• Document parameterized tests in User Guide and Release Notes
  • Document the junit-jupiter-params module in the Dependency Metadata section.
  • Document the junit-jupiter-params module in the Dependency Diagram.
  • Provide example(s) in the documentation module and include in the User Guide.

---

Original Issue Description

At the time of writing, the https://github.com/junit-team/junit-lambda/wiki/Prototype-Test-Decorators page does not have all that much detail, so this might already be considered.

It would be good if the library shipped with ready-to-use simple parameter annotations that could be used inside a single test instance. It would in many ways look like JUnit 4's @Parameterized Runner, but for use inside a single class. When running the test, each @Parameterized method would appear many times, once for each data set.

class MyTest {

   // A simple test with a specified data set
   @Test @Parameterized("mod10")
   void testMod10(String value) { ... }

   @Parameterized("mod10") 
   Collection<Object[]> getMod10Values { ... }

   // Also including a method naming convention
   @Test @Parameterized("mod11", name="{0}")
   void testMod11(String value) { ... }

   @Parameterized("mod11")
   Collection<Object[]> getMod11Values { ... }

   // In the absense of a @Parameterized data provider, use reflection
   @Test @Parameterized("getMod12Values", name="{0}")
   void testMod12(String value) { ... }

   Collection<Object[]> getMod12Values{ ... }
}

---

Related Issues

• Enhance support for dynamic tests #378
• Improve documentation of DynamicTest lifecycle #386
• Allow hierarchical parameterized/dynamic tests #604
• Parameter resolvers should be able to access the invocation context for test templates #704
• Introduce junit-jupiter-params #728

[sbrannen]

@elygre, thanks for raising this issue.

We will certainly provide a mechanism for parameterized tests. We just haven't gotten to it yet in the prototype.

However, we do already have generic support for supplying parameters to methods. This can be achieved by implementing a custom MethodParameterResolver.

Thus, the MethodParameterResolver support is the first building block required for parameterized test support. The next step will be to provide TestExtensions a mechanism for registering tests dynamically.

We will post back here once we begin work on that front.

Regards,

Sam

[sbrannen]

Update: I have opted to remove support for @InvocationIndex in the branch for repeated tests. As an alternative I have introduced RepetitionInfo which can be injected into methods, analogous to TestInfo. Details here: 6f615a8

[sbrannen]

@marcphilipp, to answer my previous question on my own...

I discovered this example:

@ParameterizedTest
@CsvSource("foo")
@MethodSource(names = { "first", "second" })
@ArgumentsSource(MyArgumentsProvider.class)
@CsvFileSource(resources = { "/single-column.csv", "two-column.csv" })
void testWithMultipleDifferentSources(String parameter) {
}

So that obviously works. 😉

I don't think I would personally use that feature (combining multiple sources for the same parameterized test), but I suppose some people will find it useful.

Out of curiosity, is this use case something you've seen in the wild or had a feature request for?

[sbrannen]

@junit-team/junit-lambda, we need to decide if it makes sense to have something like @InvocationIndex or to just scrap the idea.

[sbrannen]

FYI: #704 is a follow-up to this issue.

[sbrannen]

FYI: as a follow up to the discussion regarding whether or not we need supports() for providers, the answer is a definitive "yes". Otherwise, the support for @RepeatedTest (and any other use cases where a provider is registered but not used for a particular test template method) will be broken.

[sbrannen]

Regarding the need for an @InvocationIndex annotation, I have personally decided against it. However, we could provide an alternative means to access similar information about the current test template invocation -- something like TestTemplateInfo perhaps.

See comments here: #704 (comment)

[marcphilipp]

Thanks for all the hard work, guys!

Remaining work (@TestTemplate documentation) will be addressed in #762.