[BUG] 'Increment Version' workflow doesn't add new 1.x BWC Version for 2.0 and 2.x branches

[tlfeng]

Describe the bug
The GitHub Action workflow https://github.com/opensearch-project/OpenSearch/blob/2.0.0-rc1/.github/workflows/version.yml doesn't create Pull Requests to add new version number to "Version.java" and .ci/bwcVersions for 2.0 and 2.x branches.

Take adding version number 1.3.3 for example, it only create PRs for main, 1.x and 1.3 version:
https://github.com/opensearch-project/OpenSearch/search?q=%5BAUTO%5D+1.3.3&type=issues
The situation is the same when adding version number 1.3.2:
https://github.com/opensearch-project/OpenSearch/search?q=%5BAUTO%5D+1.3.2&type=issues

Gradle check will fail for changes to 2.0 and 2.x branches, if not adding the new 1.x version to 2.x branches.
For exmaple: #3208 (comment)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':distribution:bwc:bugfix:buildBwcLinuxTar'.
> Building 1.3.2 didn't generate expected file /var/CITOOL/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/search/distribution/bwc/bugfix/build/bwc/checkout-1.3/distribution/archives/linux-tar/build/distributions/opensearch-min-1.3.2-SNAPSHOT-linux-x64.tar.gz

To Reproduce
No need to reproduce: push a tag in the format of ..*
https://github.com/opensearch-project/OpenSearch/blob/2.0.0-rc1/.github/workflows/version.yml#L6

Expected behavior
Pull Request is created automatically to add new version number to 2.0 and 2.x branches.

Additional context
The 'Increment Version' GHA workflow is introduced by PR #1816

[dblock]

Yeah, not sure it's worth fixing. https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/version.yml

[peterzhuamazon]

@tlfeng this would fail the main branch gradle check:

> Task :distribution:bwc:minor:buildBwcLinuxTar FAILED
 [2.1.0] 
 [2.1.0] FAILURE: Build failed with an exception.
 [2.1.0] 
 [2.1.0] * What went wrong:
 [2.1.0] Execution failed for task ':buildSrc:reaper:compileJava'.
 [2.1.0] > Could not create service of type UserHomeScopedCompileCaches using UserHomeScopeServices.createCompileCaches().
 [2.1.0]    > Timeout waiting to lock Java compile cache (/home/ec2-user/.gradle/caches/7.4.2/javaCompile). It is currently in use by another Gradle instance.
 [2.1.0]      Owner PID: 8658
 [2.1.0]      Our PID: 20754
 [2.1.0]      Owner Operation: 
 [2.1.0]      Our operation: 
 [2.1.0]      Lock file: /home/ec2-user/.gradle/caches/7.4.2/javaCompile/javaCompile.lock
 [2.1.0] 
 [2.1.0] * Try:
 [2.1.0] 2 actionable tasks: 2 executed
 [2.1.0] > Run with --stacktrace option to get the stack trace.
 [2.1.0] > Run with --info or --debug option to get more log output.
 [2.1.0] > Run with --scan to get full insights.
 [2.1.0] 
 [2.1.0] * Get more help at https://help.gradle.org
 [2.1.0] 
 [2.1.0] BUILD FAILED in 2m 9s
Output for /var/jenkins/workspace/gradle_check/search/distribution/bwc/minor/build/bwc/checkout-2.x/gradlew:

This does not happen in 2.0 which uses gradle 7.3.3, the issued main branch is using gradle 7.4.2.

Thanks.

[peterzhuamazon]

Hi @reta ever since gradle upgrade to 7.4.2 I have seen such issue above on 3.0.0/main when running gradle check.
This does not happen in 2.0 branch with 7.3.3 gradle.

Running ./gradlew check -Dtests.coverage=true --no-daemon --no-scan.

Thanks.

[CEHENKLE]

Hey @dreamer-89 Would you mind picking this up? I understand this is one of the blocking issues to help get coverage out which I know we're all keen to see

[dreamer-89]

Hey @dreamer-89 Would you mind picking this up? I understand this is one of the blocking issues to help get coverage out which I know we're all keen to see

Looking into this.

[dreamer-89]

[2.1.0] > Could not create service of type UserHomeScopedCompileCaches using UserHomeScopeServices.createCompileCaches().
[2.1.0] > Timeout waiting to lock Java compile cache (/home/ec2-user/.gradle/caches/7.4.2/javaCompile). It is currently in use by another Gradle instance.
[2.1.0] Owner PID: 8658
[2.1.0] Our PID: 20754
[2.1.0] Owner Operation:
[2.1.0] Our operation:
[2.1.0] Lock file: /home/ec2-user/.gradle/caches/7.4.2/javaCompile/javaCompile.lock
[2.1.0]

This failure is different than the one this issue is intially reported (auto update Version.java for 2.x branches). From failure trace above, seems like lock contention among different gradle processes is causing the failure.

This failure is not reproducible locally (ubuntu) with latest changes from main; which makes me think, this is related to environment/setup related issue. This is coming from opensearch-project/opensearch-build#2249

[reta]

Hi @reta ever since gradle upgrade to 7.4.2 I have seen such issue above on 3.0.0/main when running gradle check.
This does not happen in 2.0 branch with 7.3.3 gradle.

I have not seen it yet, but I will take a look, thanks @dreamer-89 for troubleshooting

[reta]

Seems to be gradle/gradle#20055

[peterzhuamazon]

@reta Update: resolved if you dont use AL2/CentOS but only use Ubuntu.

[dreamer-89]

Seems to be gradle/gradle#20055

Seems happening on AL2 specific instances that is what used here as well.

[reta]

Thanks a lot @peterzhuamazon !

[dreamer-89]

This issue appears during releases where a version bump on one branch fails the gradle check on all branches one version above. It needs prioritization. Need to update the workflow which identifies the new version tag and raises the PR. The recent one #4381