Separate Jenkins gradle check job for runs against committed code

[andrross]

Is your feature request related to a problem? Please describe

The opensearch-project/OpenSearch repo has a well documented problem with non-reproducible test failures (i.e. flaky tests). It would be good to have an easily accessible place to observe trends over time with regard to test failures, test runtimes, etc. The existing Jenkins gradle-check can kind of give us that information, but it is very noisy as it has failures from PRs where the uncommitted code contains problems.

Describe the solution you'd like

We already run the gradle-check job after every commit to main. Ideally this code should never have failures (they should be caught in the PR stage). Having a separate job to observe the health of check runs against committed code will eliminate the noise from failures related to code still under review.

To accomplish this, I believe we'd need to set up another job in Jenkins, and then modify this GHA script to differentiate between runs triggered by pull_request_target versus push.

I'm also interested in other ideas on how to show the health of gradle-check over time with PR noise filtered out.

Describe alternatives you've considered

No response

Additional context

No response

[bbarani]

@rishabh6788 is working on a POC to automate the health dashboard for all repos under OpenSearch project.

[rishabh6788]

The plan is to index all the failing tests to an opensearch cluster for each and every gradle-check run.
The sample document would look like below.

{
  "build_number": 116,
  "pull_request": "PR #12503",
  "build_description": "removing unused fetch sub phase processor initialization during fetch",
  "test_class": "LibraryTest",
  "test_name": "test.java.LibraryTest.testNewLibraryMethod",
  "test_status": "FAILED",
  "build_result": "FAILURE",
  "test_fail_count": 2,
  "test_skipped_count": 0,
  "test_passed_count": 0,
  "build_duration": 10071,
  "build_start_time": 1710541261641
}

To be able to separate out gradle-check runs for each PR merge instead of each commit run to an open PR the user can filter on description field.
Each record for failing tests against a post-merge action will have a string similar to push trigger main/2x/1x <commit-id>. This will help user to focus on tests that fail when a PR is merged.
@andrross @peternied thoughts?

[andrross]

@rishabh6788 Looks very promising, I'm excited to see a dashboard with this information!

[peterzhuamazon]

Need to replace curl with higher version on gradle check host to support aws-sigv4.

• Replace default curl 7.68 on Ubuntu 20.04 with 7.75+ version to support aws-sigv4 opensearch-ci#406
• Resolve curl mv permission issues on ubuntu opensearch-ci#407
• Replace ubuntu ami with updated curl from #406 #407 opensearch-ci#408

[peternied]

Each record for failing tests against a post-merge action will have a string similar to push trigger main/2x/1x . This will help user to focus on tests that fail when a PR is merged.

@rishabh6788 Is there a way to query this data? Seems like information is getting stored as part of GitHub's metadata around commits, not sure how accessible this will be

[rishabh6788]

@peternied @andrross The PR to index failed test results along with associated PR # and PR Description has been merged and is actively running for each gradle-check job. See sample job with single test failure, do a web-page search with File Content is string and upon expanding you will be able to see what exact record got indexed. Here is the record from the mentioned job:

{
  "build_number": 35918,
  "pull_request": "12914",
  "pr_description": "backport 2x bump orgapachecommonscommonsconfiguration2 from 2100 to 2101 in pluginsrepositoryhdfs",
  "test_class": "RemoteIndexRecoveryIT",
  "test_name": "org.opensearch.remotestore.RemoteIndexRecoveryIT.testCancelRecoveryWithAutoExpandReplicas",
  "test_status": "FAILED",
  "build_result": "FAILURE",
  "test_fail_count": 1,
  "test_skipped_count": 677,
  "test_passed_count": 30892,
  "build_duration": 3028345,
  "build_start_time": 1711417450740
}

I will work on creating a dashboard over the results and share with you. We are also working to grant community members readonly access to this data.
Will keep everyone posted.
cc: @prudhvigodithi @bbarani

[peternied]

There is an API that produces json from the test results that is part of the Jenkins instance, for your example https://build.ci.opensearch.org/job/gradle-check/35918/testReport/api/json?tree=suites[cases[status,className,name]] "status":"FAILED" will find that test case you've mentioned.

By inspecting from PRs -> failures, these can be discovered and inspected such as with tool contribution-rate to create such as these top test failures, reports/20240325/top_test_failures.csv

If that data was pushed into a read only OpenSearch instance for consumption or easier yet a git repository rather than on Jenkins it would be far more useful.

[rishabh6788]

@peternied I am using Jenkins groovy pipeline methods to fetch the failed tests from the current build that is running, which gives the exact same data that you posted, and pushing it to OpenSearch cluster.
Is there anything else that you are expecting?
Wrt to other type of contribution reports that you mentioned, @prudhvigodithi is already ingesting similar github metrics across all the repositories into an OpenSearch cluster and planning it to open it to public post appsec review.

[peternied]

@rishabh6788 Thanks for giving me that quick overview earlier today, here are some items that I'd think would be really useful to include:

• Git Commit SHA
• Branch
• Any GitHub id associated with a person that is part of that pull request.

Depending on the workflow either

• Git Commit Body or
• Pull Request Description Body

Ideally also get failure information:

• Test callstack
• Std Output
• Err Output