Skip to content

Maven plugin that simplifies running Cucumber Scenarios in parallel.

License

Notifications You must be signed in to change notification settings

daczczcz1/cucable-plugin

 
 

Repository files navigation

cucable logo

Run Cucumber Scenarios in Parallel with Maven

Apache V2 License Maven Central Build Status codecov Twitter URL

Cucumber compatible

Cucable Maven Plugin

Cucable is a Maven plugin for Cucumber scenarios that simplifies fine-grained and efficient parallel test runs.

This plugin does the following:

  • Generate single Cucumber features containing one single scenario each
  • Convert scenario outlines into separate scenarios
  • Generating Cucumber runners for every generated "single scenario" feature file

Those generated runners and features can then be used with Maven Failsafe in order to parallelize test runs.

Note: From version 0.1.7 on this also works for non-english feature files!

Repository Structure

  • plugin-code contains the full plugin source code.
  • example-project contains an example Maven project to see the plugin in action.

Changelog

All changes are documented in the full changelog.

Maven dependency

<dependency>
    <groupId>com.trivago.rta</groupId>
    <artifactId>cucable-plugin</artifactId>
    <version>(check version on top of the page)</version>
</dependency>

Typical workflow

  1. Generation of runners and features
  2. Running the generated tests with Maven failsafe
  3. Aggregation of a single test report after all test runs

The following sections break down the above steps.

1. Generation of runners and features

<plugin>
    <groupId>com.trivago.rta</groupId>
    <artifactId>cucable-plugin</artifactId>
    <version>${cucable-plugin.version}</version>
    <executions>
        <execution>
            <id>generate-test-resources</id>
            <phase>generate-test-resources</phase>
            <goals>
                <goal>parallel</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <!-- Required properties -->
        <sourceRunnerTemplateFile>src/test/resources/parallel/cucable.template</sourceRunnerTemplateFile>
        <sourceFeatures>src/test/resources/features</sourceFeatures>
        <generatedFeatureDirectory>src/test/resources/parallel/features</generatedFeatureDirectory>
        <generatedRunnerDirectory>src/test/java/parallel/runners</generatedRunnerDirectory>
        
        <!-- Optional properties -->
        <numberOfTestRuns>1</numberOfTestRuns>
        <includeScenarioTags>
            <param>@includeMe</param>
            <param>@includeMeAsWell</param>
        </includeScenarioTags>                                
        <excludeScenarioTags>
            <param>@skip</param>
        </excludeScenarioTags>
        <logLevel>compact</logLevel>                                
    </configuration>    
</plugin>

Required Parameters

sourceRunnerTemplateFile

The specified file will be used to generate runners for every generated feature file.

This can be either a text file or a Java class. The difference is outlined below:

Using a java file as a runner template

If you use a java file (e.g. src/test/java/some/template/CucableJavaTemplate.java), all [FEATURE_FILE_NAME] placeholders as well as the class name will be substituted for the generated feature file name. Also, the package declaration will be stripped.

Example:

package some.template;

import cucumber.api.CucumberOptions;

@CucumberOptions(
        features = {"target/parallel/features/[FEATURE_FILE_NAME].feature"},
        plugin = {"json:target/cucumber-report/[FEATURE_FILE_NAME].json"}
)
public class CucableJavaTemplate {
}

will turn into

import cucumber.api.CucumberOptions;

@CucumberOptions(
        features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"},
        plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"}
)
public class MyFeature_scenario001_run001_IT {
}
Using a text file as a runner template

If you use a text file (e.g. src/test/resources/cucable.template), all [FEATURE_FILE_NAME] placeholders will be substituted for the generated feature file name.

Example:

import cucumber.api.CucumberOptions;

@CucumberOptions(
        features = {"target/parallel/features/[FEATURE_FILE_NAME].feature"},
        plugin = {"json:target/cucumber-report/[FEATURE_FILE_NAME].json"}
)
public class [FEATURE_FILE_NAME] {
}

will turn into

import cucumber.api.CucumberOptions;

@CucumberOptions(
        features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"},
        plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"}
)
public class MyFeature_scenario001_run001_IT {
}

sourceFeatures

This can specify

  • the root path of your existing Cucumber .feature files (e.g. src/test/resources/features)
  • the path to a specific existing Cucumber .feature file (e.g. src/test/resources/features/MyFeature.feature)
  • the path to a specific existing Cucumber .feature file including line numbers of specific scenarios/scenario outlines inside this file (e.g. src/test/resources/features/MyFeature.feature:12:19 would only convert the scenarios starting at line 12 and 19 inside MyFeature.feature)

generatedFeatureDirectory

The path where the generated Cucumber .feature files should be located (e.g. src/test/resources/parallel).

Note: This directory should be located under a valid resource folder to be included as a test source by Maven. If you want to use a directory inside Maven's target folder, check this example.

Caution: This directory will be wiped prior to the feature file generation!

generatedRunnerDirectory

The path where the generated runner classes should be located (e.g. src/test/java/parallel/runners).

Note: This directory should be located under a valid source folder to be included as a test source by Maven. If you want to use a directory inside Maven's target folder, check this example.

Caution: This directory will be wiped prior to the runner file generation!

Optional Parameters

numberOfTestRuns

Optional number of test runs. If it is not set, its default value is 1. For each test run, the whole set of features and runners is generated like this:

  • MyFeature_scenario001_run001_IT.feature
  • MyFeature_scenario001_run002_IT.feature
  • MyFeature_scenario001_run003_IT.feature
  • etc.

Note: Characters other than letters from A to Z, numbers and underscores will be stripped out of the feature file name.

includeScenarioTags

Optional scenario tags that should be included in the feature and runner generation. To include multiple tags, just add each one into as its own <param>:

<includeScenarioTags>
    <param>@scenario1Tag1</param>
    <param>@scenario1Tag2</param>
</includeScenarioTags>

Note: When using includeScenarioTags and excludeScenarioTags together, the excludeScenarioTags will override the includeScenarioTags. This means that a scenario containing an included tag and an excluded tag will be excluded!

excludeScenarioTags

Optional scenario tags that should not be included in the feature and runner generation. To include multiple tags, just add each one into as its own <param>:

<excludeScenarioTags>
    <param>@tag1</param>
    <param>@tag2</param>
</excludeScenarioTags>

Note: When using includeScenarioTags and excludeScenarioTags together, the excludeScenarioTags will override the includeScenarioTags. This means that a scenario containing an included tag and an excluded tag will be excluded!

logLevel

By default, Cucable logs all information including

  • its own name and version
  • all passed property values
  • a list of processed feature paths

This can be configured by passing the logLevel property:

<logLevel>default|compact|minimal|off</logLevel>
  • default will log all the mentioned information
  • compact will only log the plugin name, version, properties and one line of summary
  • minimal will only log a summary line
  • off will prevent any logging

Generating runners and features inside target directory

It may be desirable for you to generate the Cucable features and runners in Maven's target directory. The advantage of this is that this directory is wiped by the mvn clean command and older generated files do not reside in your src directory.

In order to achieve this, you can specify subdirectories under target (${project.build.directory}) for Cucable, e.g. ${project.build.directory}/parallel/runners and ${project.build.directory}/parallel/features

After this step, use the build-helper-maven-plugin in your POM file in order to consider the generated runner classes test sources:

<plugins>
    <plugin>
        <groupId>com.trivago.rta</groupId>
        <artifactId>cucable-plugin</artifactId>
        <version>${cucable.plugin.version}</version>
        <executions>
            <execution>
                <id>generate-test-resources</id>
                <phase>generate-test-resources</phase>
                <goals>
                    <goal>parallel</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <sourceRunnerTemplateFile>path_to_template_file</sourceRunnerTemplateFile>
            <sourceFeatures>path_to_features</sourceFeatures>
            <generatedFeatureDirectory>${project.build.directory}/parallel/features</generatedFeatureDirectory>
            <generatedRunnerDirectory>${project.build.directory}/parallel/runners</generatedRunnerDirectory>
        </configuration>    
    </plugin>
    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>${build.helper.plugin.version}</version>
        <executions>
            <execution>
                <id>add-test-source</id>
                <phase>generate-test-sources</phase>
                <goals>
                    <goal>add-test-source</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <sources>
                <source>${project.build.directory}/parallel/runners</source>
            </sources>
        </configuration>    
    </plugin>
</plugins>

Complete Example

Below, you can see a full example of what Cucable does.

Source feature file

This is our source feature file. It contains a scenario and a scenario outline with two examples.

MyFeature.feature

Feature: This is the feature name

    Scenario: First scenario
        Given I am on the start page
        And I click the login button
        Then I see an error message

    Scenario Outline: Second scenario with an amount of <amount>
        Given I am on the start page
        And I add <amount> items
        And I navigate to the shopping basket
        Then I see <amount> items
        Examples:
            | amount |
            | 12     |
            | 85     |

Runner template file

This is the runner template file that is used to generate single scenario runners. The placeholder [FEATURE_FILE_NAME] will be replaced with generated feature names by Cucable.

import cucumber.api.CucumberOptions;

@CucumberOptions(
        features = {"target/parallel/features/[FEATURE_FILE_NAME].feature"},
        plugin = {"json:target/cucumber-report/[FEATURE_FILE_NAME].json"}
)
public class [FEATURE_FILE_NAME] {
}

Note: The specified plugin generates Cucumber JSON files which are needed for custom aggregated test reports.

Generated Scenarios

For each scenario, a single feature file is created:

MyFeature_scenario001_run001_IT.feature

Feature: This is the feature name

Scenario: First scenario
Given I am on the start page
And I click the login button
Then I see an error message

Note that for the scenario outlines, each example is converted to its own scenario and feature file:

MyFeature_scenario002_run001_IT.feature

Feature: This is the feature name

Scenario: Second scenario with an amount of 12
Given I am on the start page
And I add 12 items
And I navigate to the shopping basket
Then I see 12 items

MyFeature_scenario003_run001_IT.feature

Feature: This is the feature name

Scenario: Second scenario with an amount of 85
Given I am on the start page
And I add 85 items
And I navigate to the shopping basket
Then I see 85 items

Generated runners

The generated runners point to each one of the generated feature files.

This is an example for one of the generated runners - note how the placeholders are now replaced with the name of the feature to run:

MyFeature_scenario001_run001_IT.java

import cucumber.api.CucumberOptions;

@CucumberOptions(
        features = {"target/parallel/features/MyFeature_scenario001_run001_IT.feature"},
        plugin = {"json:target/cucumber-report/MyFeature_scenario001_run001_IT.json"}
)
public class MyFeature_scenario001_run001_IT {
}

2. Running the generated tests with Maven failsafe

This will skip the unit tests (if any) and run the generated runner classes with Failsafe. Since all generated runner classes from the step before end with _IT, they are automatically considered integration tests and run with failsafe.

Note: If all tests should be run regardless of their result, it is important to set <testFailureIgnore>true</testFailureIgnore> for Failsafe - otherwise the plugin execution will stop on failing tests. However, if this is specified, the build will not fail in case of failing tests!

To circumvent that, it is possible to specify a custom rule for Maven enforcer that passes or fails the build depending on custom conditions.

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
            <skipTests>true</skipTests>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <executions>
            <execution>
                <id>Run parallel tests</id>
                <phase>integration-test</phase>
                <goals>
                    <goal>integration-test</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <testFailureIgnore>true</testFailureIgnore>
            <forkCount>${maven.fork.count}</forkCount>
            <reuseForks>false</reuseForks>
            <argLine>-Dfile.encoding=UTF-8</argLine>
            <disableXmlReport>true</disableXmlReport>
        </configuration>
    </plugin>
</plugins>

3. Aggregation of a single test report after all test runs

We use the Cluecumber plugin to aggregate all generated .json report files into one overall test report.

<plugins>
    <plugin>
        <groupId>com.trivago.rta</groupId>
        <artifactId>cluecumber-report-plugin</artifactId>
        <version>${cluecumber.report.version}</version>
        <executions>
            <execution>
                <id>report</id>
                <phase>post-integration-test</phase>
                <goals>
                    <goal>reporting</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <sourceJsonReportDirectory>${project.build.directory}/cucumber-report</sourceJsonReportDirectory>
            <generatedHtmlReportDirectory>${project.build.directory}/test-report</generatedHtmlReportDirectory>
        </configuration>    
    </plugin>
</plugins>

Example project

You can test the complete flow and POM configuration by checking out the Cucable example project.

Additional Information

Building

Cucable requires Java >= 8 and Maven >= 3.3.9. It is available in Maven central.

License

Copyright 2017 trivago N.V.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Credits

This plugin was inspired by the Cucumber Slices Maven Plugin.

This project is being tested on

Browserstack

About

Maven plugin that simplifies running Cucumber Scenarios in parallel.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 96.8%
  • Gherkin 3.2%