[POC] Universal Release Prototype for Data Prepper #1
Conversation
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
Signed-off-by: Peter Nied <[email protected]>
|
Closing as this will not be merged, but comments can still be created for discussion and tracking purposes. |
| ] | ||
|
|
||
| standardReleasePipeline(config) { | ||
| def distManifest = new FakeDistManifest(fakeYmlFromUrl(params.DistManifestUrl)) |
There was a problem hiding this comment.
It appears that the distribution manifest is the major defining factor for the source project.
How will the manifest work with different versions? Will it be parameterized?
Right now, for Data Prepper, I'm expecting two possible parameters:
- The version of Data Prepper
- The build which we are promoting.
There was a problem hiding this comment.
Excellent question, I hand waved over this for the POC.
Little background, OpenSearch project will be built inside of many different build systems gradle, node or python. These all have different processes to create a distributable component, by using a common format to describe different kinds of artifacts, no matter how binaries or docker images or the-next-big-thing is created these workflows can still interact with them.
Here is a snippet of 1.1 version with a OpenSearch distribution:
---
schema-version: '1.1'
build:
name: OpenSearch
version: 1.3.0
platform: linux
architecture: x64
location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/813/linux/x64/dist/opensearch/opensearch-1.3.0-linux-x64.tar.gz
id: '813'
components:
- name: OpenSearch
repository: https://github.com/opensearch-project/OpenSearch.git
ref: 1.x
commit_id: 02ca00fc7289cda1b87b34992a82b2b849a4b1a4
location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/813/linux/x64/builds/opensearch/dist/opensearch-min-1.3.0-linux-x64.tar.gz
- name: job-scheduler
repository: https://github.com/opensearch-project/job-scheduler.git
ref: main
commit_id: a67e734f5760af7225553bee17707f057c2af740
location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/813/linux/x64/builds/opensearch/plugins/opensearch-job-scheduler-1.3.0.0.zip
...How will the manifest work with different versions? Will it be parameterized?
The distribution manifest is the 'trigger' to start a release, Jenkins access might not be required to release a new version, or the url to the manifest is all that is needed. The intent is that it is trivial to generate via any build system, including but not limited to a GitHub action with no other external systems.
- Sidebar: We haven't settled on the format once available it would trigger a release, between checking in a file called data-prepper-1.2.0-distribution-manifest.yml or making a GitHub release with annotations in the body and artifacts included, tracking with #1506.
The manifest contains the URIs of artifacts and container to publish, the different functions in the workflow will prepare/publish them as desired. The manifest should be pointing to static assets that will be transformed during a release operation. Allows for N kinds of assets split across platforms, architectures or what-not. So there would be a distribution manifest for each release with no parameterization.
All that said we can build the generation for this quickly to make sure we've got the correct level of fidelity and clarity for everyone.
There was a problem hiding this comment.
Added a prototype of how the 2.0 dist manifest could include the information we need to track and how to generate it
---
schema-version: '2.0'
build:
- name: Data-Prepper
version: ${{ VERSION }}
artifacts:
- location: https://${{ secrets.S3_BUCKET }}/data-prepper-core-${{ VERSION }}.tar.gz
containers:
- type: docker
tag: data-prepper-staging/data-prepper:${{ VERSION }}| autoPublish: true | ||
| ) | ||
|
|
||
| fakePublishToArtifactRepositry( |
There was a problem hiding this comment.
Yes - they are described in the manifest that I hand-waved over
| description: 'The url to distribution manifest for this release')]) | ||
| ]) | ||
|
|
||
| lib = library( |
There was a problem hiding this comment.
Please consider making this part of the pipeline instead.
In this way, a team creating a stage can configure their pipeline with the desired source without having to change their stage file.
There was a problem hiding this comment.
This is a good question towards what the 'minimum' started file is, for the moment I think I'd rather error on the side of caution and include these sections. We can always streamline later if we come up with a good way to synchronize them
| ) | ||
|
|
||
| fakeCopyDockerImage( | ||
| sourceImagePath: distManifest.artifacts.containerReference, |
There was a problem hiding this comment.
The Data Prepper release build will create a Docker image and upload it to our staging ECR registry. Data Prepper will not have a staging DockerHub registry.
I want to be sure that this copyDockerImage function will be able to download from ECR and upload to either DockerHub or ECR.
There was a problem hiding this comment.
This signature was based on existing functionality. We can make sure that it would work as either docker or ecr first sources
| cleanup: { -> echo "Totally unneed clean for demo purposes"} | ||
| ] | ||
|
|
||
| standardReleasePipeline(config) { |
There was a problem hiding this comment.
The Data Prepper team needs to create this stage. It is not maintained by opensearch-build. The Jenkinsfile for it will exist in the Data Prepper repository.
| @@ -0,0 +1,104 @@ | |||
| properties([ | |||
| parameters([ | |||
There was a problem hiding this comment.
What does it mean for a Jenkins stage to define parameters? I thought they had to be defined at the job/pipeline level.
There was a problem hiding this comment.
The 'minimum' addressable unit for a release seems to be a distribution manifest. However this might not be a one-size-fits-all solution so I wanted to be sure that parameters could be added for other project types.
I believe its useful to see the parameter defined here as a property so its clear within the standardReleasePipeline {...} body code of this file that it can be accessed via params.DistManifestUrl
Build up a github action to be triggered manually that could create a distribution manifest. This will not work as is, but is a possible forward path Signed-off-by: Peter Nied <[email protected]>
|
Open/Closing so that the other prototype changes can be picked up |
Description
This is a proof of concept for how much effort an OpenSearch Project would need to undertake to add signing, artifacts, maven, docker. There are a number of
fake*methods used, this was to make testing inside of a one-off jenkins environment easier, and they can be largely replaced for the real versions (or they need to be written).