[BUG] Gradle check retry not working as expected

[kotwanikunal]

Describe the bug

• As a part of Retry failed flakey tests #2547 and Gradle check retry #2638, a retry mechanism was introduced to rerun failed tests during gradle check
• With the increased flakiness in tests, one issue that was noticed is the retries do not always run for tests on the build server

Sample runs -

• https://build.ci.opensearch.org/job/gradle-check/6825/consoleFull
• https://build.ci.opensearch.org/job/gradle-check/6782/

To Reproduce

1. Run gradle check for a PR with a test failure

Expected behavior

• Retries to be present within the console output with x tests completed, y failed with y > 1 in case of failed tests with a retry

Plugins

• N/A

Screenshots

• https://build.ci.opensearch.org/job/gradle-check/6825/consoleFull
• https://build.ci.opensearch.org/job/gradle-check/6782/

Host/Environment (please complete the following information):

• Jenkins build server for OpenSearch

Additional context

• [BUG] Test failure exists but Gradle 'check' task passes #2878

[andrross]

Just as an example, from this test run back in April, if we drill into the report we can see 3 failures of the same test:

But it succeeded on the 4 attempt so the overall test run is marked as successful. Not conclusive, but I have never seen an example of a test failing but succeeding on retry since we moved to the public Jenkins runner.

[andrross]

Ok, now I do have an example of a test succeeding after retry from just a few minutes ago: https://build.ci.opensearch.org/job/gradle-check/6876/testReport/org.opensearch.index/ShardIndexingPressureConcurrentExecutionTests/ You can see that the test testCoordinatingPrimaryThreadedUpdateToShardLimitsAndRejections failed twice but succeeded on the third attempt. You can also observe the retries in the console output. (The overall test failed for a different reason causing a test to fail 4 times).

[dblock]

I think this is working as expected. I'm closing this, reopen if you can show an example where all retries failed and we still reported success.

[andrross]

reopen if you can show an example where all retries failed and we still reported success

@dblock I think we have the opposite problem. The retries are working, but even when a test succeeds on retry we still fail the overall test run. See here for an example. AwarenessAttributeDecommissionIT.testInvariantsAndLogsOnDecommissionedNodes both failed and succeeded in the same run. Don't we expect the overall test result to be success in such a case?

[andrross]

Again looking at 6825, we see the "BUILD SUCCESSFUL" result in the output (tests succeeded on retry):

BUILD SUCCESSFUL in 25m 35s
2626 actionable tasks: 2612 executed, 2 from cache, 12 up-to-date
...
Finished: UNSTABLE

The gradle check jenkins run completes with the result "UNSTABLE" and results in a failure of the GitHub action. Is this the right behavior? I think the point of the retries is to pass the gradle check when tests pass on retry so this seems wrong to me.

[kotwanikunal]

I'll look at other UNSTABLE gradle checks to check if it's the same cause.

[kotwanikunal]

Build 6895 - https://build.ci.opensearch.org/job/gradle-check/6895/testReport/
Passed on retry - https://build.ci.opensearch.org/job/gradle-check/6895/testReport/org.opensearch.cluster.coordination/AwarenessAttributeDecommissionIT/

Build 6879 - https://build.ci.opensearch.org/job/gradle-check/6879/testReport/
Passed after 1 retry - https://build.ci.opensearch.org/job/gradle-check/6879/testReport/org.opensearch.repositories.s3/RepositoryS3ClientYamlTestSuiteIT/

Build 6828 - https://build.ci.opensearch.org/job/gradle-check/6828/testReport/
Passed on retry - https://build.ci.opensearch.org/job/gradle-check/6828/testReport/org.opensearch.index/ShardIndexingPressureConcurrentExecutionTests/, https://build.ci.opensearch.org/job/gradle-check/6828/testReport/org.opensearch.snapshots/DedicatedClusterSnapshotRestoreIT/

[kotwanikunal]

@dblock @andrross
Verified that UNSTABLE is essentially success on retry. Should we start treating UNSTABLE as a successful run on PRs?

[kotwanikunal]

When gradle check was not a jenkins step, we had binary output directly from the ./gradlew check run which was either FAILED or SUCCESS.
Jenkins, in case of any test failure, marks the step as UNSTABLE, which is what's leading to the difference in retry based successful runs.

From Jenkins docs - https://www.jenkins.io/doc/book/glossary/

Publisher
Part of a Build after the completion of all configured Steps which publishes reports, sends notifications, etc. A publisher may report Stable or Unstable result depending on the result of its processing and its configuration. For example, if a JUnit test fails, then the whole JUnit publisher may report the build result as Unstable.

[dblock]

I think that if a retry succeeded, the build should be a SUCCESS.

[kotwanikunal]

Fix: opensearch-project/opensearch-build#2902