[PROPOSAL] Incorporate a generic build system (something like Gradle).

[prudhvigodithi]

Is your feature request related to a problem? Please describe.

Coming from opensearch-project/opensearch-build#1375.
Adding gradle to the project will help ease the automation part by working with simple gradle tasks, gradle also supports a bunch of node plugins that run node specific tasks, example https://plugins.gradle.org/plugin/com.github.node-gradle.node. Using gradle it's also easy to incorporate any future automation tasks.
One being auto increment the development iteration solution. Having gradle as build systems the automation will be consistent for OpenSearch core and OpenSearch Dashboards as well.

Describe the solution you'd like
To start with gradle build system, implement the solution to increment the development iteration upon every release. Right now since OpenSearch-Dashboards do not have gradle, there is some limitation to perform the automation that will launch a workflow which will trigger gradle tasks to auto-increment the version.

Describe alternatives you've considered

For node, using npm or yarn or any other packages could fullfil the the version increment, but this is only limited to JSON file and some with only strict semantic version. Looking at this example PR, its beyond just a JSON file and more that just having a version in the JSON file, the version increment is required in multiple files.
The other alternate solutions particularly for version increment problem is to create a workflow with linux parsing tools like awk, sed, but this so brittle and requires developer effort to modify the logic when ever the file changes.

Additional context

Considering for current and future automations, gradle would help ease to implement them with custom tasks and plugins, once gradle build system is part of OpenSearch-Dashboards, its associated plugins can implement the gradle build system.

[ashwin-pc]

I'm not entirely sure why we need a full Gradle system just to increment a version bump. The current build system uses webpack and is responsible for a lot of the bundling and build processes. Migrating all of them to Gradle would be a huge lift. It would be much easier to just have a script that uses Dashboards existing build system to trigger a version upgrade that the build repo can call when releasing.

I'd really love to weigh the pros and cons of having a unified build system vs letting each repo (atleast the node and java repos) have separate build systems tailored to each of their use cases.

We can also consider having a unified build contract that each build system needs to support irrespective of the underlying tool, giving us more flexibility for each project while keeping the build/release process simple.

[prudhvigodithi]

Hey @ashwin-pc, it can be the best supported one, other than gradle, my point is it should be a best fit for any other automations in future (build system).

I have tried few solutions that related to npm
there are some npm cli's and packages that support version increment but targets only semver (semver/semver#340)
Example:
https://jstools.dev/version-bump-prompt/
https://docs.npmjs.com/cli/v8/using-npm/scripts

Considering the version increment automation:
https://github.com/opensearch-project/security-dashboards-plugin/pull/1005/files
https://github.com/opensearch-project/anomaly-detection-dashboards-plugin/pull/274/files
https://github.com/opensearch-project/index-management-dashboards-plugin/pull/201/files
https://github.com/opensearch-project/dashboards-visualizations/pull/86/files

If noticed there are more than one place (not only just package.json) where the version has to be updated, some generic solution like a gradle task, a feasible one, generic for all dashboard repos should be integrated to auto bump the version. FOr gradle projects there is already a solution for this . Any suggestions @ashwin-pc ?

@bbarani

[ashwin-pc]

Got it, yeah i'm aligned with that :) I'd suggest just using a simple node script like the ones we have in our scripts folder for simple automation tasks and using a custom plugin with our existing bundler dependency webpack for more complex transformations. This should work for all build automation tasks. @kavilla what's your take on this solution?

[AMoo-Miki]

@prudhvigodithi, could you please clarify which one of these the ask is? We had a discussion internally and there was some confusion here.

• Simply to add a config file which the build system will use to make calls to bump or set versions and produce the artifacts for a release.

  • The code can expose commands like yarn bump-version or yarn set-version <version> and yarn package-for-release to perform those tasks.
  • I see no harm in having this.

• Change the build mechanism.

  • Dashboards has a build mechanism that simply cannot be rewritten in any other way without monstrous and unnatural effort.
  • I don't think the proposal is asking for this since it is unreasonable.

[prudhvigodithi]

Hey @AMoo-Miki

The cli's and packages that support version increment but targets only semver, but we use something like 2.0.0.0, the extra .0, and some cli's fail to update this in package.json, so right now I see a regex mechanism to increment/update the version.
I dont want to change any build mechanism, but if possible add a new process for automations that is generic for all dashboard plugins. Managing scripts per plugin will be soon be difficult to handle and my intake is not only just for the version increment, if in future if we have to support any other automations, build process we need a generic way that should be compatible for all dashboard plugins, example for gradle projects we can onboard new task/custom-plugins per requirement and its generic for all gradle projects.
Thoughts?

[kavilla]

Hello @prudhvigodithi,

It appears that the current proposal is for all future build scripts is to move to Gradle, which would call for refactoring to existing mechanism. The reason to refactor would be that future maintainers will be confused why half the project is build one way but another is built another way if all new mechanisms are added in Gradle.

If we don't do a major refactor and live in a hybrid state, I see a negative impact to this repo while adding positive value to another repo. But if do execute a major refactor it will be outside of priorities and scope as well.

However, if the proposal is we introduce Gradle that calls a bunch of Node scripts then I can maybe get behind this (as proposed by @AMoo-Miki). From the build repo's perspective, it already does this somewhat with .sh files that it calls during the release and testing process.

[bbarani]

@prudhvigodithi @AMoo-Miki @kavilla @seanneumann @dblock Having automated the version increment process for OpenSearch, I would like to focus on automating the same for OpenSearch Dashboards as well. I am totally fine with using a different solution for Dashboards but at the same time dont want to add too much overhead on the build and release process. Do you have any specific recommendations?

[prudhvigodithi]

Hey all, definitely this process has to be changed, for OpenSearch the workflow action already in place which [AUTO] raises the version increment PR's, but whereas for dashboard its still manual and a person has to spend hours of time to do this.
Having a .sh bash script per dashboard plugin is not a good idea, it will add lot of scripts to the list of maintained scripts, but if this is the choice, I'm ok to start with and keep this GH issue open until we have process of incrementing the version via a node based solution (from the discussion above, I have suggested few).
Thank you
@AMoo-Miki @kavilla @seanneumann @dblock @bbarani

[ashwin-pc]

@prudhvigodithi do you have list of build automations you intend on introducing? version increment is too simple an automation to disscuss the pros and cons of different approaches since almost any approach should be able to do that.

[ashwin-pc]

One option we have is to introduce a package in ./packages or adapt the osd-pm package to add additional build commands that we need to support like version bump. osd-pm is a cli tool that is currently responsible for similar functions like installing the dependencies, cleaning the the repo etc

e.g.

yarn osd bootstrap

workflows can use this for the different automations it needs without a lot of rewiring.

[prudhvigodithi]

Hey @ashwin-pc thanks, with list of build automations, its starts with version increment and it could extend more in future, one other scenario I see integrating a testing framework, this can also be done using a build tool, that can call a task to run the tests and many more ;), majority of the cli's I see goes with strict semVer, can you add more about this yarn osd bootstrap, how we can wire additional files apart from package json files that require version increment ? Sometimes its just more that the JSON files the version has to be updated example https://github.com/opensearch-project/security-dashboards-plugin/pull/1005/files.

[prudhvigodithi]

One other way I see is, probably better for node projects
Add custom scripts to the package.json that can do the version increment
A high level example:

{
    "scripts": {
        "hello-world": "echo \"Hello World\""
    }
}

Now call npm run hello-world
https://docs.npmjs.com/cli/v8/commands/npm-run-script

[ashwin-pc]

Yep, we can definitely do that. yarn <command> does exactly that where <command> is a custom script defined in the package.json. Thats why asked what exaclty the features we need to support are. The current build system is quite flexible and we have a lot of ways to add these functionalities.

osd-pm just takes this a step further and acts as a cli tool that also has access to the setup and build pipeline. It is also run using an npm run script like yarn osd <OSD_PM_COMMAND>. but for simple use cases, the npm run script should be more than enough.

[bbarani]

@seanneumann @seraphjiang Can you please help with this effort? We would need your help to understand the feasibility of the using npm script based solution.

[seraphjiang]

@bbarani @prudhvigodithi

@ashwin-pc proposed yarn <command> seems promising approach.

[prudhvigodithi]

Hey @ashwin-pc, is this related to npm-run-script or something different with yarn <command>, if possible can you show us a POC?
Thank you

[ashwin-pc]

@prudhvigodithi npm run <script_name> and yarn run <script_name> are the same thing. We use yarn in OSD and the run command in yarn can be skipped, hence the yarn <script_name> syntax

[bbarani]

@ashwin-pc Can you work with @prudhvigodithi on a POC to use scripts for incrementing the version for core OR any 1 plugin?

[AMoo-Miki]

As far as Dashboards is concerned, this should be just bumping the version in package.json. If we find that there are places with hardcoded version numbers, we should change them to pull in the version from the package.json file.

• To set the version to something specific: yarn version --new-version <version>
• To bump the major version and reset minor and patch to zeros: yarn version --major
• To bump the minor version and reset patch to zero: yarn version --minor
• To bump the patch version: yarn version --patch

Ref: https://classic.yarnpkg.com/lang/en/docs/cli/version/

Plugins will be very different since they don't follow semantic versioning.

[AMoo-Miki]

• I have introduced a version command to the plugin-helpers in Extends plugin-helpers to be used for automating version changes #2398; this can be used to update plugins before building them.
• For OSD version updates, please use the commands in my previous comment.

[prudhvigodithi]

Running yarn plugin-helpers version --sync legacy for a repo like opensearch-dashboards-functional-test throws an error Has it been migrated properly? Does it have a opensearch_dashboards.json file?, the plugin helpers version does not work without opensearch_dashboards.json file, yet the repo requires version increment opensearch-project/opensearch-dashboards-functional-test#476, any suggestions please on how to handle this scenario?
@AMoo-Miki @kavilla

[prudhvigodithi]

@AMoo-Miki @kavilla do we have any solution for the issue I have mentioned above, recently noticed the version increment is failing opensearch-dashboards-functional-test repo, looks like some change with jq cli, I have raised a PR to fix this.
opensearch-project/opensearch-build#4497, but I want to use plugin-helpers for opensearch-dashboards-functional-test repo, so that its generic across all OSD repos.
@bbarani @Divyaasm
Thank you