Add JUnit 5 Support

[reinhapa]

Summary

Provide a JUnit 5 extension for cucumber.

Expected Behavior

When writing a test with JUnit 5 in our environment, we need to replicate a base environment in a way that we use different Extensions in order to to run tests. So we like to have a Scenario something like this:

@ExtendWith(CucumberExtension.class)
@ExtendWith(MockitoExtension.class)
@ExtendWith(BaseBusinessEnvironmentExtension.class)
class MyBusinessTest {
   @BeforeEach
    void init(@Mock Person person) {
        when(person.getName()).thenReturn("Dilbert");
    }

    @Test
    void simpleTestWithInjectedMock(@Mock Person person) {
        assertEquals("Dilbert", person.getName());
    }
}

Current Behavior

At the moment under JUnit 4 we needed to tweak the Cucumber runner in order to call default JUnit 4 annotations.

Possible Solution

We like to help create an CucumberExtension that handles the Cucumber specific annotations for JUnit 5 as an JUnit 5 extension, so that it can be combined with other extensions as for example mocking frameworks.

[mpkorstanje]

Awesome! I don't have any experience with JUnit5 so I can't help too much. Perhaps you can use #1103 as a starting point, see if it builds, meets your needs and review the code.

[reinhapa]

I will take a look into it

[reinhapa]

@mpkorstanje I looked into it briefly and it would be great to open a feature branch for this extension. I will try to contact @tenwit to see if I can help furter extend/test it...

[mpkorstanje]

Excellent! If you need help with anything let me know. I'll do what I can.

[reinhapa]

@mpkorstanje I just trying to merge the the current master and the existing pull request into my local repo right know. I will come back to you as soon I got it working.

[tenwit]

I'm here :) How can I help? My solution works using extensions; I see there's an alternative solution using the engine at https://github.com/signed/junit-cucumber-engine. I hadn't considered using an engine when I did my work. I think it's more idiomatic to use an engine.

[reinhapa]

@tenwit I give just started to merge your master code with actual 2.0.0-SNAPSHOT version on my repo https://github.com/reinhapa/cucumber-jvm/tree/jupiter as it seems, there was introduced a new API layer, which I was starting to switch over to by taking a look into the existing JUnit code. I'm not actually completed that work so far, but you may be take a look at the current state..

[mica16]

Is it currently recommended to use Cucumber with JUnit 4?

[aslakhellesoy]

@mica16, JUnit 4 is required until JUnit 5 support is merged and released.

[mica16]

@aslakhellesoy Ok, thanks :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[sherl0cks]

any word on this? I hope JUnit 5 async means that frameworks like vert.x will become cucumber compatible. https://developers.redhat.com/blog/2018/01/23/vertx-junit5-async-testing/

[aslakhellesoy]

Somebody needs to write the code @sherl0cks.

We've recently discussed a code generation approach, which would make it easier to support JUnit5 (and other test runners). See https://cucumberbdd.slack.com/archives/C6RLMP3C4/p1520605709000469

[sherl0cks]

looks like slack requires an invite, but I get the general idea. unfortunately I have no cycles for something like thing. I'll watch the issue

[mpkorstanje]

There was some experimentation in #1258 to setup cucumber as dynamic tests. It created quite a bit of boiler plate and seemed like the wrong solution and didn't allow for extensions.

After that I've done some research and I came to the following conclusion:

JUnit 5 consists of the JUnit Platform + JUnit Jupiter + JUnit Vintage.

JUnit Jupiter

Cucumber itself doesn't fit into JUnit Jupiters extension model. To be an extension it would have to yield control of test discovery, test instantiation and test execution to Jupiter. This is not feasible as cucumber follows an entirely different process for each.

While the TestInstancePostProcessor looks promising, it seems as if it could just as easily be applied to a glue class as it could be applied to Jupiter test instance, it is strongly tied to the assumption that a single class provides the context for a single test. In cucumber the test context consists of all glue classes.

This in my opinion makes it unlikely that we'll ever use JUnit Jupiter for Cucumber.

Even if we do through for example code generation, then we'll still be unable to use any extensions as Jupiter is only able to process the generated code of the test instance, not the glue classes that would benefit from the post processing.

Junit Platform

One of the problems of JUnit4 was that @RunWith required runners to generate test instances before they could enrich them. This made it impossible to combine different runners. They were all generating instances and unable to enrich those generated by others. JUnit Jupiter was designed to make these extensions possible. What remains is handled JUnit Platform.

As a generic platform for test execution it provides a common interface for test discovery, selection and filtering. This means that by implementing a Cucumber Engine for the Junit Platform any user who places it on class path can use the Junit Platform to discover and execute feature files.

Junit Vintage

Our current JUnit Runner should continue to work with JUnit Vintage. I reckon this will be the way to execute Cucumber for a long time to come.

[aslakhellesoy]

@sherl0cks sorry - slack joining instructions are here

[luolong]

As a generic platform for test execution it provides a common interface for test discovery, selection and filtering. This means that by implementing a Cucumber Engine for the Junit Platform any user who places it on class path can use the Junit Platform to discover and execute feature files.

This sounds like a proper approach to me. What are the issues with this one?

[mpkorstanje]

Until v4 test discovery, selection filtering and execution were hopelessly intertwined. This has recently been improved but I am not certain that what we have now is decoupled enough.

The other issue being that cucumber is open source and developed by volunteers. So far there has been no one who contributed a junit 5 module.

Feel free to help out!

[luolong]

Well, sure. I would love to help out at some point.

I suppose, first step would be to decouple test execution from any kind of test framework and then use framework to guide the test execution in some way.

But I suppose that this is not going to happen over a weekend?

[tauras007]

We were able to use the cucumber cli runners io.cucumber.core.cli.Main.run(..) inside a junit 5 test to execute specific features and used the exitStatus returned to determine if the feature failed or passed, i am sure this is not the ideal approach but something better than having both junit 5 and 4 in the project classpath.

Main.run(new String[]{"--glue ", "x.x.x.x.x", featurePath}, contextClassLoader)

[moshneagaOleg]

hello how can i run tests in parallel use jvm 5.4.0 ?

[mpkorstanje]

https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine#parallel-execution

[davidghiurco]

Has this been released with Cucumber 5.6.0? I'm not able to find any official Cucumber documentation that doesn't say I still need to include io.cucumber:cucumber-junit in my classpath, and that's JUnit 4.

What I'm trying to ask is, which, if any, cucumber artifacts should I include in my classpath to take advantage of the originally described desired functionality at the very top of this issue? e.g.

@ExtendWith(CucumberExtension.class)
class MyCucumberWithJUnit5Test

[mpkorstanje]

This seems to be a common point of confusion. It is worth reading the introduction of the documentation for JUnit 5.

But in brief Cucumber implements the TestEngine interface provided by the JUnit Platform. On the other hand @ExtendWith is a JUnit Jupiter annotation. These are two different parts of JUnit 5.

If you want to use JUnit 5 with Cucumber you can find all information you need here:

https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine

You will not find JUnit 5 advertised in other documentation yet. The tooling support for Junit Platform hasn't quite reached the point yet where I'd be comfortable changing that. And given that Cucumber JUnit works perfectly fine in combination with JUnit Vintage I see no reason too rush the upgrade either.

Internally Surefire and Gradle assume the old class based execution model. This means that Cucumber works less then stellar with these integrations. So I'm currently waiting for a few issues to be resolved here:

• https://issues.apache.org/jira/browse/SUREFIRE-1724
• JUnit 5: Allow resource-based testing gradle/gradle#4773

With respect to IDEA, there is no good way to use the JUnit Platform to run individual scenarios and so I'm waiting for:

• https://youtrack.jetbrains.com/issue/IDEA-227508
• Support FilePosition in discovery selectors junit-team/junit5#2146

You may want to upvote/star/+1 these issues to show that you have an interested in this feature. Or if you have the ability and capacity; contribute to a solution to these issues.

[helder-ciandt]

@mpkorstanje, do you have any example of junit-platform-engine + spring boot application?

[mpkorstanje]

• https://github.com/cucumber/cucumber-jvm/tree/main/examples/spring-txn
• https://github.com/mpkorstanje/cucumber-spring-boot-parallel

Neither is very exhaustive or explanatory but you can find all information you need here:

• https://github.com/cucumber/cucumber-jvm/tree/master/junit-platform-engine
• https://github.com/cucumber/cucumber-jvm/tree/main/spring
• https://github.com/cucumber/cucumber-jvm/tree/main/java

[viktozhu]

I can see that current cucumber version is 6.5.0 and this is 6.1.2-Snapshot. Is this part expected to be supported and released with later versions?

And second question is usage, currently proposed usage is @Cucumber while for Junit5 I would expect @ExtendWith(CucumberExtension.class) - in this case cucumber will not have to handle other possible extensions. What is planned direction?

[mpkorstanje]

@viktozhu it's in v6.5 and you seem to misunderstand what JUnit 5 is. Consider reading the discussion, I'd rather not repeat myself. See #1149 (comment).