Generate new input manifest with previous latest version

[peterzhuamazon]

Description

Generate new input manifest with previous latest version

Issues Resolved

Closes #4389

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[peterzhuamazon]

Hi,

There are several changes in this PR:

1. Keep the original creating input manifest from template method as a fallback. If there is no known versions exist or the version for the new manifest is smaller than the min(known_versions), use templates.
2. Else, always use the latest version that is lower than the new version as base to create new version. Ex: Assuming the latest version is 2.12.0, and you want to create 2.12.1/2.13.0/3.0.0/4.0.0. Use 2.12.0 to create 2.12.1, use 2.12.1 to create 2.13.0, use 2.13.0 to create 3.0.0, use 3.0.0 to create 4.0.0.
3. Follow up to 2, if 2.12.0 and 2.13.0 co-exist, but you want to create a 2.12.1, then it will always use 2.12.0 as base even if there is a 2.13.0 within the same major version, due to 2.12.0 is the highest verion before 2.12.1.
4. Fully compatible with legacy manifests so that if you want to create a 2.2.2, it will try to find 2.2.1 from manifests, if not, it will find 2.2.1 from legacy-manifests and use it as base.
5. Fully compatible with both 1.0 and 1.1 version schema of input manifests.
6. Ability to use newly created manifest as base for the next new version, see 2.

Thanks.

Result logs:

$ rm -rf legacy-manifests/2.2.2 manifests/2.2.2 manifests/2.12.1 manifests/2.13.0 && ./manifests.sh update
......
2024-03-11 22:10:29 INFO     Known versions: ['1.0.0', '1.0.1', '1.1.0', '1.1.1', '1.2.0', '1.2.1', '1.2.2', '1.2.3', '1.2.4', '1.2.5', '1.3.0', '1.3.1', '1.3.2', '1.3.3', '1.3.4', '1.3.5', '1.3.6', '1.3.7', '1.3.8', '1.3.9', '1.3.10', '1.3.11', '1.3.12', '1.3.13', '1.3.14', '1.3.15', '1.4.0', '2.0.0', '2.0.1', '2.0.2', '2.1.0', '2.1.1', '2.2.0', '2.2.1', '2.3.0', '2.3.1', '2.4.0', '2.4.1', '2.4.2', '2.5.0', '2.5.1', '2.6.0', '2.6.1', '2.7.0', '2.7.1', '2.8.0', '2.8.1', '2.9.0', '2.9.1', '2.10.0', '2.10.1', '2.11.0', '2.11.1', '2.11.2', '2.12.0', '3.0.0']
2024-03-11 22:10:29 INFO     Checking out components into /tmp/tmp7jq2n86a
2024-03-11 22:10:29 INFO     Checking OpenSearch ['2.2', '2.12', '2.x'] branches
2024-03-11 22:10:29 INFO     Executing "git init" in /tmp/tmp7jq2n86a/OpenSearch/2.2
2024-03-11 22:10:29 INFO     Executing "git remote add origin https://github.com/opensearch-project/OpenSearch.git" in /tmp/tmp7jq2n86a/OpenSearch/2.2
2024-03-11 22:10:29 INFO     Executing "git fetch --depth 1 origin 2.2" in /tmp/tmp7jq2n86a/OpenSearch/2.2
2024-03-11 22:10:34 INFO     Executing "git checkout FETCH_HEAD" in /tmp/tmp7jq2n86a/OpenSearch/2.2
2024-03-11 22:10:35 INFO     Executing "git rev-parse HEAD" in /tmp/tmp7jq2n86a/OpenSearch/2.2
2024-03-11 22:10:35 INFO     Checked out https://github.com/opensearch-project/[email protected] into /tmp/tmp7jq2n86a/OpenSearch/2.2 at 75b48671562254158f70f55b9d474665443f15f5
2024-03-11 22:10:35 INFO     Executing "./gradlew properties -Dbuild.snapshot=false" in /tmp/tmp7jq2n86a/OpenSearch/2.2
2024-03-11 22:10:42 INFO     OpenSearch#2.2 is version 2.2.2
2024-03-11 22:10:42 INFO     Executing "git init" in /tmp/tmp7jq2n86a/OpenSearch/2.12
2024-03-11 22:10:42 INFO     Executing "git remote add origin https://github.com/opensearch-project/OpenSearch.git" in /tmp/tmp7jq2n86a/OpenSearch/2.12
2024-03-11 22:10:42 INFO     Executing "git fetch --depth 1 origin 2.12" in /tmp/tmp7jq2n86a/OpenSearch/2.12
2024-03-11 22:10:49 INFO     Executing "git checkout FETCH_HEAD" in /tmp/tmp7jq2n86a/OpenSearch/2.12
2024-03-11 22:10:49 INFO     Executing "git rev-parse HEAD" in /tmp/tmp7jq2n86a/OpenSearch/2.12
2024-03-11 22:10:49 INFO     Checked out https://github.com/opensearch-project/[email protected] into /tmp/tmp7jq2n86a/OpenSearch/2.12 at 8f3e2697b789708f0847f48d1480b47f5ac4050c
2024-03-11 22:10:49 INFO     Executing "./gradlew properties -Dbuild.snapshot=false" in /tmp/tmp7jq2n86a/OpenSearch/2.12
2024-03-11 22:10:58 INFO     OpenSearch#2.12 is version 2.12.1
2024-03-11 22:10:58 INFO     Executing "git init" in /tmp/tmp7jq2n86a/OpenSearch/2.x
2024-03-11 22:10:58 INFO     Executing "git remote add origin https://github.com/opensearch-project/OpenSearch.git" in /tmp/tmp7jq2n86a/OpenSearch/2.x
2024-03-11 22:10:58 INFO     Executing "git fetch --depth 1 origin 2.x" in /tmp/tmp7jq2n86a/OpenSearch/2.x
2024-03-11 22:11:06 INFO     Executing "git checkout FETCH_HEAD" in /tmp/tmp7jq2n86a/OpenSearch/2.x
2024-03-11 22:11:07 INFO     Executing "git rev-parse HEAD" in /tmp/tmp7jq2n86a/OpenSearch/2.x
2024-03-11 22:11:07 INFO     Checked out https://github.com/opensearch-project/[email protected] into /tmp/tmp7jq2n86a/OpenSearch/2.x at df9016121cc15e5af9f454c6f360bace0191bb93
2024-03-11 22:11:07 INFO     Executing "./gradlew properties -Dbuild.snapshot=false" in /tmp/tmp7jq2n86a/OpenSearch/2.x
2024-03-11 22:11:15 INFO     OpenSearch#2.x is version 2.13.0
2024-03-11 22:11:15 INFO     New Versions: ['2.2.2', '2.12.1', '2.13.0']
2024-03-11 22:11:15 INFO     Creating new version: 2.2.2
2024-03-11 22:11:15 INFO     Base Version: 2.2.1 is the highest version before 2.2.2
2024-03-11 22:11:15 INFO     Using /local/home/zhujiaxi/opensearch-build-peterzhuamazon/legacy-manifests/2.2.1/opensearch-2.2.1.yml as the base manifest
2024-03-11 22:11:15 INFO     Wrote /local/home/zhujiaxi/opensearch-build-peterzhuamazon/manifests/2.2.2/opensearch-2.2.2.yml as the new manifest
2024-03-11 22:11:15 INFO     Creating new version: 2.12.1
2024-03-11 22:11:15 INFO     Base Version: 2.12.0 is the highest version before 2.12.1
2024-03-11 22:11:15 INFO     Using /local/home/zhujiaxi/opensearch-build-peterzhuamazon/manifests/2.12.0/opensearch-2.12.0.yml as the base manifest
2024-03-11 22:11:15 INFO     Wrote /local/home/zhujiaxi/opensearch-build-peterzhuamazon/manifests/2.12.1/opensearch-2.12.1.yml as the new manifest
2024-03-11 22:11:15 INFO     Creating new version: 2.13.0
2024-03-11 22:11:15 INFO     Base Version: 2.12.1 is the highest version before 2.13.0
2024-03-11 22:11:15 INFO     Using /local/home/zhujiaxi/opensearch-build-peterzhuamazon/manifests/2.12.1/opensearch-2.12.1.yml as the base manifest
2024-03-11 22:11:16 INFO     Wrote /local/home/zhujiaxi/opensearch-build-peterzhuamazon/manifests/2.13.0/opensearch-2.13.0.yml as the new manifest
2024-03-11 22:11:16 INFO     Removing /tmp/tmp7jq2n86a

[codecov]

Codecov Report

Attention: Patch coverage is 97.29730% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 92.16%. Comparing base (3549bdb) to head (bfa8739).
Report is 9 commits behind head on main.

Files	Patch %	Lines
src/manifests_workflow/input_manifests.py	97.29%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4523      +/-   ##
==========================================
- Coverage   92.19%   92.16%   -0.03%     
==========================================
  Files         192      192              
  Lines        6292     6282      -10     
==========================================
- Hits         5801     5790      -11     
- Misses        491      492       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rishabh6788]

Also, given we are adding all the components to the new manifest, whether they have updated the next version or not, we should add --continue-on-error flag to the cron entry that is made as part of this job, else, even if OS and critical components bump versions on their end the daily build job would keep failing due to other components that are yet to bump the versions on their repo.

[rishabh6788]

Also, given we are adding all the components to the new manifest, whether they have updated the next version or not, we should add --continue-on-error flag to the cron entry that is made as part of this job, else, even if OS and critical components bump versions on their end the daily build job would keep failing due to other components that are yet to bump the versions on their repo.

Ignore this, just realized default is true for --continue-on-error.

[dblock]

Good stuff! I recommend documenting what's going on in some .md files, this is pretty deep :)

[peterzhuamazon]

Good stuff! I recommend documenting what's going on in some .md files, this is pretty deep :)

Hi @dblock , I have updated the manifest_workflow readme now.

Thanks

[rishabh6788]

LGTM!

[peterzhuamazon]

LGTM!

Thanks!

[peterzhuamazon]

New:

• [AUTO] Update input manifests #4527

[peterzhuamazon]

Runs well, will close this issues for now.