Ability to rerun a specific test N times or until failure

[rieske]

Sometimes, when developing, I expect some test might be flaky due to a code defect - especially when working with concurrent code. Such tests pass most of the time, masking a problem.
When I have such suspect, I end up rerunning that test manually until I get the failure from where I can explore further.
Other times, the failure can be tricky to expose and might take quite a few retries and can easily be missed.

IntelliJ IDEA has a handy feature in their JUnit test runner that I find extremely useful in such cases during development: retry until failure, until stopped or N times.

This is a great help to catch such rarely-failing-because-of-defect suspects by allowing them to run hands-off for a while and only get disturbed when they fail.

Gradle test runner does not expose such ability.

Since test-retry-gradle-plugin deals with retrying flaky tests that failed (false negatives), perhaps it could also provide an aid for dealing with potential false-positive flakiness?

Alternatively, or in addition to, perhaps Gradle could expose APIs in both CLI and TAPI to run potentially filtered out tests in a specific task until-failure or ntimes - ./gradlew test --tests "..." --until-failure.

[jkschneider]

@rieske Open to more concrete suggestions on how such a thing should work, since a plugin can't add a new top-level flag to the Gradle CLI.

Would running a specific task like ./gradlew testUntilFailure --tests "..." satisfy the case?

[eriwen]

@jkschneider note that plugins can declare CLI options this way starting in Gradle 4-dot-something: https://docs.gradle.org/current/userguide/custom_tasks.html#sec:declaring_and_using_command_line_options

We would need to create a separate task to run N times or until failure and we could make CLI flags to configure it.

[jkschneider]

@eriwen So do you think ./gradlew test -- tests "..." --until-failure is how it should be done? Or maybe --retry-until-failure, --retry-forever, and --retry-times=N?

[eriwen]

@jkschneider I think these are some use cases worth supporting:

• Repeat each test N times, don't fail-fast.
• Repeat each test N times, but stop after first failure.

Silly question: can we leverage Gradle's --fail-fast CLI flag here or do we need to create our own?

Can you think of other use cases? If not, perhaps --tests-repeat=N + --fail-fast OR --tests-continue-repeat=N + --tests-fail-fast-repeat=N would be the best. This may be better than having multiple CLI flags that affect each other (e.g. ./gradlew test -- tests "..." --retry-forever --retry-times=N --retry-until-failure is ambiguous)

[ldaley]

note that plugins can declare CLI options this way starting in Gradle 4-dot-something: https://docs.gradle.org/current/userguide/custom_tasks.html#sec:declaring_and_using_command_line_options

Task extensions can't currently do this. Only the implementation class is introspected. So, we could not do this for any released Gradle version.

I think there are several good feature ideas raised here, but I think we can wait until this functionality is implemented in Gradle.

[mwos-sl]

+100000 for retrying given test as long as a result is the same as previous one, but no more than N times. Example: N = 6

x x x x x x-> FAILLNG
o o o o o o -> PASSING
x x o -> FLAKY
o o o x -> FLAKY

So as soon as flakyness is detected - stop. It would be cool to have this done by this plugin.
We REALLY need this.

[mwos-sl]

..or at least do not stop retrying on failing ones.

[ldaley]

This is going to be awkward to do from this plugin. Having this in Gradle itself would provide a much better experience. In particular, it could support invocation time setting such as…

./gradlew --tests SomeTest --retry-until-failure 20

I have raised gradle/gradle#15179 for this.

[mdzhigarov]

The Maven plugin for IntelliJ has had this for a very long time. Keep up, Gradle!

[safaorhan]

This would be great to test for flaky tests in the newer Android Studio builds.