Auto-generate next version manifest(s)

.github/workflows/versions.yml

@@ -0,0 +1,42 @@
+name: versions
+
+on:
+  push:
+  schedule:
+    - cron: "0 0 * * *"
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+    env:
+      PYTHON_VERSION: 3.7
+      JDK_VERSION: 14
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}    
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set Up JDK ${{ env.JDK_VERSION }}
+        uses: actions/setup-java@v1
+        with:
+          java-version: ${{ env.JDK_VERSION }}
+      - name: Set up Python ${{ env.PYTHON_VERSION }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+      - name: Install Pipenv and Dependencies
+        run: |
+          python -m pip install --upgrade pipenv wheel
+      - name: Update OpenSearch Manifests
+        run: |
+          ./bundle-workflow/manifests.sh update
+      - name: Create Pull Request
+        uses: peter-evans/create-pull-request@v3
+        with:
+          commit-message: Updated manifests.
+          delete-branch: true
+          title: 'Updated manifests.'
+          body: |
+            I've noticed that a repo has incremented a version and have made some updates to build manifests.
+      - name: Check outputs
+        run: |
+          echo "Pull Request Number - ${{ steps.cpr.outputs.pull-request-number }}"
+          echo "Pull Request URL - ${{ steps.cpr.outputs.pull-request-url }}"

README.md

@@ -4,19 +4,30 @@
 [![manifests](https://github.com/opensearch-project/opensearch-build/actions/workflows/manifests.yml/badge.svg)](https://github.com/opensearch-project/opensearch-build/actions/workflows/manifests.yml)
 [![codecov](https://codecov.io/gh/opensearch-project/opensearch-build/branch/main/graph/badge.svg?token=03S5XZ80UI)](https://codecov.io/gh/opensearch-project/opensearch-build)
 
-- [OpenSearch Build](#opensearch-build)
+- [Releasing OpenSearch](#releasing-opensearch)
+  - [Creating a New Version](#creating-a-new-version)
+  - [Building and Testing an OpenSearch Distribution](#building-and-testing-an-opensearch-distribution)
+  - [Making a Release](#making-a-release)
 - [Contributing](#contributing)
 - [Getting Help](#getting-help)
 - [Code of Conduct](#code-of-conduct)
 - [Security](#security)
 - [License](#license)
 - [Copyright](#copyright)
 
-## OpenSearch Build
+## Releasing OpenSearch
 
-This repository contains the scripts for building OpenSearch and OpenSearch Dashboards distributions.
+### Creating a New Version
 
-* [OpenSearch Bundle Workflow](bundle-workflow/README.md)
+OpenSearch and OpenSearch Dashboards are distributed as bundles that include both core engines and plugins. Each new OpenSearch release process starts when any one component increments a version, typically on the `main` branch. For example, [OpenSearch#1192](https://github.com/opensearch-project/OpenSearch/pull/1192) incremented the version to 2.0. The [automation process in this repo](.github/workflows/manifests.yml) will notice this change and create a new manifest in [manifests](./manifests). A job is then added to start building this new version.  
+
+### Building and Testing an OpenSearch Distribution 
+
+OpenSearch and its components are built from source, assembled, signed and tested using the [bundle workflow](bundle-workflow/README.md).
+
+### Making a Release
+
+TODO
 
 ## Contributing
 

bundle-workflow/Pipfile

@@ -18,6 +18,7 @@ botocore = "1.21.33"
 coverage = "~=4.5.4"
 pytest-cov = "~=2.10.0"
 jproperties = "~=2.1.1"
+sortedcontainers = "*"
 
 [dev-packages]
 

bundle-workflow/README.md

@@ -3,16 +3,17 @@
     - [Custom Build Scripts](#custom-build-scripts)
   - [Assemble the Bundle](#assemble-the-bundle)
     - [Custom Install Scripts](#custom-install-scripts)
-  - [Signing Artifacts](#signing-artifacts)
+  - [Sign Artifacts](#sign-artifacts)
   - [Test the Bundle](#test-the-bundle)
     - [Integration Tests](#integration-tests)
     - [Backwards Compatibility Tests](#backwards-compatibility-tests)
     - [Performance Tests](#performance-tests)
   - [Sanity Check the Bundle](#sanity-check-the-bundle)
+  - [Auto-Generate Manifests](#auto-generate-manifests)
 
 ## OpenSearch Bundle Workflow
 
-This workflow builds a complete OpenSearch bundle from source. You can currently build 1.0, 1.1 and 1.1-SNAPSHOT.
+The bundle workflow builds a complete OpenSearch distribution from source. You can currently build 1.0, 1.1 and 1.1-SNAPSHOT.
 
 ### Build from Source
 
@@ -76,7 +77,7 @@ Artifacts will be updated as follows.
 
 You can perform additional plugin install steps by adding an `install.sh` script. By default the tool will look for a script in [scripts/bundle-build/components](scripts/bundle-build/components), then default to a noop version implemented in [scripts/default/install.sh](scripts/default/install.sh).
 
-### Signing Artifacts
+### Sign Artifacts
 
 The signing step (optional) takes the manifest file created from the build step and signs all its component artifacts using a tool called `opensearch-signer-client` (in progress of being open-sourced). The input requires a path to the build manifest and is expected to be inside the artifacts directory with the same directories mentioned in the build step. 
 
@@ -165,3 +166,26 @@ The following options are available.
 | --component [name] | Test a single component by name, e.g. `--component common-utils`.       |
 | --keep             | Do not delete the temporary working directory on both success or error. |
 | -v, --verbose      | Show more verbose output.                                               |
+
+### Auto-Generate Manifests
+
+The [manifests workflow](src/manifests) reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is identified, it creates a new input manifest in [manifests](../manifests) and [opens a pull request](../.github/workflows/manifests.yml).
+
+Show information about existing manifests. 
+
+```bash
+./bundle-workflow/manifests.sh list
+```
+
+Check for updates and create any new manifests. 
+
+```bash
+./bundle-workflow/manifests.sh update
+```
+
+The following options are available.
+
+| name               | description                                                             |
+|--------------------|-------------------------------------------------------------------------|
+| --keep             | Do not delete the temporary working directory on both success or error. |
+| -v, --verbose      | Show more verbose output.                                               |

bundle-workflow/manifests.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+set -e
+
+DIR="$(dirname "$0")"
+"$DIR/run.sh" "$DIR/src/manifests.py" $@

bundle-workflow/src/git/git_repository.py

@@ -29,8 +29,11 @@ def __init__(self, url, ref, directory=None, working_subdirectory=None):
             self.dir = directory
             os.makedirs(self.dir, exist_ok=False)
 
-        # Check out the repository
         self.working_subdirectory = working_subdirectory
+        self.__checkout__()
+
+    def __checkout__(self):
+        # Check out the repository
         self.execute_silent("git init", self.dir)
         self.execute_silent(f"git remote add origin {self.url}", self.dir)
         self.execute_silent(f"git fetch --depth 1 origin {self.ref}", self.dir)

bundle-workflow/src/manifests.py

@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+import logging
+
+from manifests_workflow.input_manifests import InputManifests
+from manifests_workflow.manifests_args import ManifestsArgs
+from system import console
+
+args = ManifestsArgs()
+console.configure(level=args.logging_level)
+manifests = InputManifests()
+
+if args.action == "list":
+    for manifest in manifests.values():
+        logging.info(f"{manifest.build.name} {manifest.build.version}")
+elif args.action == "update":
+    manifests.update(keep=args.keep)
+
+logging.info("Done.")

bundle-workflow/src/manifests/input_manifests.py

@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+import glob
+import os
+import re
+
+from manifests.input_manifest import InputManifest
+from manifests.manifests import Manifests
+
+
+class InputManifests(Manifests):
+    def __init__(self):
+        files = glob.glob(os.path.join(self.manifests_path, "opensearch-*.yml"))
+        # there's an opensearch-1.0.0-maven.yml that we want to skip
+        files = [f for f in files if re.search(r"/opensearch-([0-9.]*)\.yml$", f)]
+        super().__init__(InputManifest, files)

bundle-workflow/src/manifests/manifests.py

@@ -0,0 +1,45 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+import os
+import re
+
+from sortedcontainers import SortedDict  # type: ignore
+
+
+class Manifests(SortedDict):
+    def __init__(self, klass, files):
+        super(Manifests, self).__init__()
+        self.klass = klass
+        self.__append__(files)
+
+    def __append__(self, files):
+        for filename in files:
+            basename = os.path.basename(filename)
+            match = re.search(r"-([0-9.]*).yml$", basename)
+            if not match:
+                raise ValueError(f"Invalid file: {basename}")
+
+            version = match.group(1)
+            manifest = self.klass.from_path(filename)
+            self.__setitem__(version, manifest)
+
+    @property
+    def manifests_path(self):
+        return os.path.realpath(
+            os.path.join(os.path.dirname(__file__), "../../../manifests")
+        )
+
+    @property
+    def versions(self):
+        return list(map(lambda manifest: manifest.build.version, self.values()))
+
+    @property
+    def latest(self):
+        if len(self) == 0:
+            raise RuntimeError("No manifests found")
+
+        return self.values()[-1]

bundle-workflow/src/manifests_workflow/__init__.py

@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+#
+# This page intentionally left blank.