OpenSearch Dashboards: Automate increment to next development iteration

[prudhvigodithi]

Is your feature request related to a problem? Please describe

Coming from #1375
We begin incrementing versions after we decide that we want a release. This takes a day and makes it that nightly distribution builds do not include full bundle until T-14 days to release. Instead, prepare the project to make the next release after the previous release is done, so that we can save that day and have release candidates always available for the next development iteration.

Describe the solution you'd like

A workflow that notices new version, and increments versions accordingly and makes a PR.

Acceptance Criteria

• Dashboard plugin-helpers task [PROPOSAL] Incorporate a generic build system (something like Gradle). OpenSearch-Dashboards#1801
• POC Concept/Solution to Automate the version increment to next development iteration.
• Add/modify existing version increment workflow to onboard OpenSearch Dashboards to version increment automation.

[peterzhuamazon]

Not too much question just that we need to clear up the existing OS bumping method.
It is still a bit cumbersome to use and it could be better.
By introducing OSD the same workflow will have too many subtasks to run in GHA, which is very hard to find the one we need.

Example: https://github.com/opensearch-project/opensearch-build/actions/runs/4463409293
See side panel it is really hard to figure out which plugin get bumped to which version unless you click them manually one by one.

Thanks.

[gaiksaya]

Some enhancements while we design this:

1. +1 to @peterzhuamazon 's comment
2. Need to see if we can dynamically pick up plugins from manifests. From most recent 1.x , 2.x version. Reason being more and more plugins are getting added causing a maintenance overhead in GH workflow. This would also avoid adding exclusions for 1.x series.

[msfroh]

Does this necessarily need to be a GH workflow?

Could this be an offline script that loops through repos, checks them out locally, creates and pushes the next branch, does the version bump on the upstream branch, and creates a PR?

[dblock]

That would not be "automated" @msfroh. It works very well for OS already.

[prudhvigodithi]

Thanks @gaiksaya and @peterzhuamazon yes good thought on picking up plugins list from the manifest.
@msfroh as @dblock said the existing setup for OS works well we dont need to maintain an external offline script, the existing workflow already does the same finally raises a PR.
Thank you

[Divyaasm]

We can create a seperate increment-version workflow for OSD and call the Plugin Helper - opensearch-project/OpenSearch-Dashboards#2398

[prudhvigodithi]

The PR to onboard the osd workflow, works (sample run) in creating the AUTO version increment PR's for the OSD Plugins
Sample PR's
opensearch-project/security-dashboards-plugin#1413
opensearch-project/dashboards-query-workbench#68

But For 1.3 versions it failed with the following error, I think the plugin_helpers is not backported

error Command "plugin-helpers" not found. Did you mean "plugin_helpers"?
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.

@AMoo-Miki @kavilla can you please take a look if the osd-plugin-helpers have any issues with 1.3?

[ohltyler]

@prudhvigodithi typically dashboards plugins have other workflows that need to be updated as well, to run integration tests for example (see AD integ test workflow file here)

Is there a way this could be included in the workflow somehow? Or maybe a standard that plugins could follow to ensure that all of their tests are updated to run against the correct versions?

[prudhvigodithi]

Hey @ohltyler as per the plugin helper created by @AMoo-Miki the version should only be part of the JSON files, the helper used plugin-helpers version could be extended I believe?
@AMoo-Miki please add your thoughts.
Thank you

[AMoo-Miki]

I was going to recommend using plugin-helpers --sync but I see that the workflow already exists.

But For 1.3 versions it failed with the following error, I think the plugin_helpers is not backported

@prudhvigodithi, sadly some plugins have aliased this as plugin-helpers and others as plugin_helpers. For that reason, the workflows should use ./../scripts/plugin_helpers to avoid those differences.

@ohltyler, the workflow can use the version from its own JSON file to deduce what versions it needs to fetch or run against.

[prudhvigodithi]

Thanks @AMoo-Miki, using node ../../scripts/plugin_helpers version --sync legacy to make sure the version does not fail, but should we finalize plugin-helper for all plugins?

[AMoo-Miki]

should we finalize plugin-helper for all plugins?

I would have loved the plugins to have a unified name for this. Sadly they haven't for a long time and changing some abruptly would be problematic with users. I would suggest updating ALL plugins to have both plugin-helpers and plugin_helpers.

[prudhvigodithi]

Closing this issue as the OpenSearch Dashboard version increment implementation is completed.
Thank you