Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-generate next version manifest(s) #461

Merged
merged 12 commits into from
Sep 15, 2021

Conversation

dblock
Copy link
Member

@dblock dblock commented Sep 13, 2021

Signed-off-by: dblock [email protected]

Description

Automatically generate next version's manifest.

  1. Locate the latest manifest.
  2. Attempt to get gradle properties from each component.
  3. Generate a manifest for any new version.
  4. Make a pull request, e.g. [AUTO] Updated input manifests. dblock/opensearch-build#2

Now generates OpenSearch 1.0.1, 1.2.0 and 2.0.0 manifests.

I'm still writing more tests, but otherwise this is ready.

Issues Resolved

Closes #388.

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dblock dblock mentioned this pull request Sep 13, 2021
1 task
@dblock dblock force-pushed the generate-manifests branch 2 times, most recently from 324a8e6 to d88a549 Compare September 13, 2021 21:07
@codecov-commenter
Copy link

codecov-commenter commented Sep 13, 2021

Codecov Report

Merging #461 (b26d20e) into main (378f171) will increase coverage by 3.13%.
The diff coverage is 91.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #461      +/-   ##
==========================================
+ Coverage   65.18%   68.32%   +3.13%     
==========================================
  Files          51       59       +8     
  Lines        1307     1487     +180     
==========================================
+ Hits          852     1016     +164     
- Misses        455      471      +16     
Impacted Files Coverage Δ
bundle-workflow/src/manifests.py 0.00% <0.00%> (ø)
...workflow/src/manifests_workflow/input_manifests.py 95.65% <95.65%> (ø)
bundle-workflow/src/git/git_repository.py 91.11% <100.00%> (+0.41%) ⬆️
bundle-workflow/src/manifests/input_manifests.py 100.00% <100.00%> (ø)
bundle-workflow/src/manifests/manifests.py 100.00% <100.00%> (ø)
...undle-workflow/src/manifests_workflow/component.py 100.00% <100.00%> (ø)
...low/src/manifests_workflow/component_opensearch.py 100.00% <100.00%> (ø)
...src/manifests_workflow/component_opensearch_min.py 100.00% <100.00%> (ø)
...-workflow/src/manifests_workflow/manifests_args.py 100.00% <100.00%> (ø)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 378f171...b26d20e. Read the comment docs.

@dblock dblock changed the title WIP: Auto-generate next version manifest. Auto-generate next version manifest(s) Sep 14, 2021
@dblock dblock marked this pull request as ready for review September 14, 2021 16:18
peternied
peternied previously approved these changes Sep 14, 2021
def get_root_branches(self):
branches = ["main"]
remote_branches = (
subprocess.check_output(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to GitRepository? Or some kind of git actions class

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, will revisit in another PR.

run: |
./bundle-workflow/manifests.sh update
- name: Create Pull Request
uses: peter-evans/create-pull-request@v3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this pile up duplicate pull requests? Do we have any control over managing duplication?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plugin assumes a branch name (which here I didn't specify so it will just be called "patch") and if it sees a PR from a branch called "patch", it will update it instead of creating a new one. It's pretty smart like that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Could we add a branch name that is based off of the version number? This way if there is a patch release and a minor release updates at the same time separate pull requests are created.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not so easy to know what's being added, would require returning something meaningful from the build that can be parsed by the workflow. I did update the branch name to manifests/auto so that the auto-pr doesn't walk over other PRs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting branch: didn't work on my fork with a permissions error that doesn't happen with defaults, so I undid that.

peternied
peternied previously approved these changes Sep 15, 2021
super().__init__("OpenSearch", repo, snapshot)

@classmethod
def get_root_branches(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should document what it means to be a 'root branch' so that it is clear feature branches with the same versions of a 'root branch' are excluded.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

from system.properties_file import PropertiesFile


class ComponentOpenSearch(Component):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit - OpenSearchComponent & OpenSearchMinComponent?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that file names sort :)


### Auto-Generate Manifests

The [manifests workflow](src/manifests) reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is identified, it creates a new input manifest in [manifests](../manifests) and [opens a pull request](../.github/workflows/manifests.yml).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we specify which branches of core & components are scanned?

If I'm reading this correctly, we are only scanning branches of core (with the specific 'root' pattern) and the 'main' branch of component repos. Is that correct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. We’re looking at anything that seems like a version number. It’s really not that important though, we could be examining all branches.

We should probably not actually be looking at branches like 1.0 to save time - in our branching strategy we always increment on main or 1.x first.

@dblock dblock merged commit 9a5843f into opensearch-project:main Sep 15, 2021
@dblock dblock deleted the generate-manifests branch September 15, 2021 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatically generate manifests for future OpenSearch versions
4 participants