Support for custom JUnit rule instead of RunWith

[thekalinga]

I am planning to use

1. Spring MVC - For Integration testing
2. NestedRunner - For nesting test cases
3. JUnitParams - For DDT
4. Assumes - for specifying test dependencies
5. Mockito - For mocking
6. PowerMockito - For any mocking abuse I may need to do (I mean class/static method mocking)
7. DBUnit - For fiddling with DB data

Spring MVC in the current milestone version (4.2.0.RC1 or > 4.2 whenever the release happens) uses rules instead of runner to support integration with other runners. PowerMock currently supports Rules. Mockito supports Rule in the current Beta version (2.0.17-beta)

However, AFAIK, all others does not support rules. It would be better if this library can implement the runner using Junit Rule instead of Runner so that it can be compatible with all other runners to make a full blown testing stack. Else, people need to hack around to get most out of Java test frameworks.

Reference:

1. Creating a custom runner for mockito
2. Nested Runners Example

[Macroz]

How would you like it to work? Do you have an example in mind of what it would look like when you use all of these together?

JUnit & test running is a bit complicated, and full of annotational magic. What would be the KISS solution?

[xkr47]

An alternative to using JUnitParams is WrappingParameterizedRunner which is in the junit-runners module in this git repository. It is a runner that supports wrapping another runner while adding the functionality of the standard Parameterized runner (with a slightly different (hopefully better) api).

[xkr47]

With JUnit rules, the following things are not possible (afaik):

• affect the choice of test class being run (e.g. for loading through a different classloader or replacing it with something else)
• enumerate (create a list of) the test methods to run in a class (and the order of them)
• decide which constructor to call for the test class for each individual test run

The first bullet can be circumvented (as proven by PowerMockRule), apparently by using java agents or similar to transform the classes being tested. It might be possible to use this technique for the third bullet also, haven't looked deeply enough into how PowerMockRule does it.

Luckily most "features" that you might want to plug in (for example the list in the issue description) will do just fine without these bullets. In these cases it's probably even easier to implement the feature as a JUnit rule than as a runner.

Of the features listed in the issue description I think all except the following should be possible to implement as JUnit rules:

• NestedRunner - enumerates test methods
• JUnitParams - enumerates test methods

An alternative for these is to make the runners wrapping runners (i.e. the runners calls another runner in a chain, for example like the WrappingParameterizedRunner written by me does, or like the standard Enclosed runner). This allows to some extent affect all of the three bullets for example by using classloading techniques, extending the testclass on the fly etc.

Since you can still choose one runner to be the "bottom runner", it is enough that one of the above two "cannot be implemented as JUnit rule" features is implemented as a wrapping runner. Personally I would perhaps evaluate how hard it would be to do that for the NestedRunner first.

Feel free to challenge any of my hypotheses above.

[thekalinga]

@Macroz
I have created a rough sketch of what I have my mind here

@xkr47 My assumption has been that WrappingParameterizedRunner is tied to junit Parameterized api. IMO, JUnitParams is much more clean and better option (just my view).

Please note that I'm not well versed in the internals of Junit, am just someone trying find a comprehensive framework combination

[xkr47]

@thekalinga: The api is similar to Parameterized but not the same, there is no tie. It's a complete rewrite.

I completely agree that as many "features" should be implemented as JUnit rules as possible. Everything else would benefit from supporting runner wrapping.

I would love to see something new to replace the current JUnit that took out all these limitations in a single blow and allowed easy combination of features without crazy classloading stuff being required to combine them to get e.g. JUnitParams-style features. I wouldn't mind if it was the next major release of JUnit, as long as the current Runner concept and associated responsibility constraints would be replaced with open and collaborative concepts - like JUnit rules but for the whole process.

[thekalinga]

@xkr47 Yes, could not agree with you more.. To me it seems everything boils down to lack of JUnit to chain Runners (my assumption was that the whole point of @Rule was that, but after reading your earlier comment and looking at the @TestRule api its obvious that Rule feature seems to be band-aid). Sure, we certainly need some level sanity in the junit framework. To me the current version of jUnit reminds me of bean validation framework [JSR 303 & JSR 349] that cant handle chained validators & makes your life extremely painful whenever you deviate from basic template.

I just went over the testcases for WrappingParameterizedRunnerTest and am thinking of gluing all the pieces mentioned in the original issue report into a working prototype via Enclosed (for Assumes) & WrappingParameterizedRunner see if everything fares well.

Update
Looks like WrappingParameterizedRunnerTest always expects @ParameterizedSuite. No luck in using NestedRunner with junitparams

@xkr47 @Macroz Any suggestions on integrating these two frameworks?

Thanks