Update build workflow to build based on distribution (OpenSearch) #1619

peterzhuamazon · 2022-02-12T00:05:46Z

Signed-off-by: Peter Zhu [email protected]

Description

Update build workflow to build based on distribution (OpenSearch).
You can now use --distribution to build tar/zip/rpm for OpenSearch-min.
Not completely bound to platform anymore.

This would also setup the stage for bundled RPM creation in assemble_workflow (#2073)

Issues Resolved

Part of #1543

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peter Zhu <[email protected]>

dblock · 2022-02-14T14:36:23Z

This is going to require updating all build.sh's in all the active branches across all components to support -d. I think it's a lot of work, maybe only pass -d if specified, and default it to None?

peterzhuamazon · 2022-02-14T18:32:46Z

This is going to require updating all build.sh's in all the active branches across all components to support -d. I think it's a lot of work, maybe only pass -d if specified, and default it to None?

My initial thinking process is to only do this for OpenSearch core component, as that is the only thing needed for this.
The idea is to build the core in deb or rpm form, then take the files either from deb/rpm-extracted folder, or decompress from the .deb / .rpm pkg during the assemble workflow, similar to untar, then install all the plugin zips then repack.

Will try to make it to none so it will only be consumed by OpenSearch core.

Thanks.

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon · 2022-02-15T20:59:02Z

Hi @dblock I have implemented the changes you suggest, as well as tweaking the scripts to take None as default.
The problem right now is the build_recorder does not record distribution: something into the manifest.yml after builds.

Could you take a look at the code again?
Thanks.

codecov-commenter · 2022-02-15T20:59:06Z

Codecov Report

Merging #1619 (4b10b30) into main (bddbc06) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1619      +/-   ##
============================================
+ Coverage     94.67%   94.68%   +0.01%     
  Complexity       13       13              
============================================
  Files           163      163              
  Lines          3437     3446       +9     
  Branches         21       21              
============================================
+ Hits           3254     3263       +9     
  Misses          180      180              
  Partials          3        3

Impacted Files	Coverage Δ
src/run_build.py	`91.11% <ø> (ø)`
src/build_workflow/build_args.py	`100.00% <100.00%> (ø)`
src/build_workflow/build_recorder.py	`100.00% <100.00%> (ø)`
src/build_workflow/build_target.py	`100.00% <100.00%> (ø)`
src/build_workflow/builder_from_source.py	`100.00% <100.00%> (ø)`
src/manifests/build_manifest.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bddbc06...4b10b30. Read the comment docs.

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon · 2022-02-15T21:32:58Z

I added the code to record to manifest.yml now. Thanks.

dblock

It's on the right track, but should be simpler and have fewer changes if you consider that you can never have distribution: None.

scripts/components/OpenSearch/build.sh

dblock · 2022-02-15T22:53:04Z

src/build_workflow/build_recorder.py

@@ -60,6 +60,7 @@ def __init__(self, target: BuildTarget):
            self.data["build"]["version"] = target.opensearch_version
            self.data["build"]["platform"] = target.platform
            self.data["build"]["architecture"] = target.architecture
+            self.data["build"]["distribution"] = target.distribution if target.distribution else "None"


Leave this as target.distribution, it should always have a value here coming from the caller.

@dblock but if I leave as is, if no one call --distribution then it will default to None, which then will error out during recording.

ValueError: Invalid manifest schema: {'build': [{'distribution': ['null value not allowed']}]}

Set to tar if None for now.

src/build_workflow/builder_from_source.py

dblock · 2022-02-15T22:56:25Z

tests/data/opensearch-build-1.1.0.yml

@@ -2,6 +2,7 @@
 build:
  platform: linux
  architecture: x64
+  distribution: None


Version 1.1.0 never had a distribution field, remove.

dblock · 2022-02-15T22:56:39Z

tests/data/opensearch-dashboards-build-1.2.0.yml

@@ -1,6 +1,7 @@
 ---
 build:
  architecture: x64
+  distribution: None


Same as for 1.1.0, remove here and everywhere, except for 1.3.0 build output that should now write a value of distribution.

So to be clear this is to update a new test for 1.3.0 the release, not the schema?

dblock · 2022-02-15T22:58:11Z

...ests_test_workflow/test_integ_workflow/integ_test/data/build_manifest_missing_components.yml

@@ -2,6 +2,7 @@
 build:
  platform: linux
  architecture: x64
+  distribution: None


Add a 1.3.0 manifest that has a value for distribution. Possibly a new test.

distribution: None is not a thing, I don't expect to ever find this in a build file, when we don't pass distribution it defaults to something (tar?) doesn't it?

I was following the suggestion to have None as default, while previously my commit I use tar as default.
I am bit confused here what exactly should I use?
Thanks.

Signed-off-by: dblock <[email protected]>

…dle build (opensearch-project#1378) * Promote to release candidate after every bundle build Signed-off-by: Yilin Zhang <[email protected]> Signed-off-by: Sayali Gaikawad <[email protected]> Co-authored-by: Sayali Gaikawad <[email protected]>

* Added a unit test for the Data Prepper build. Signed-off-by: David Venable <[email protected]>

Signed-off-by: dblock <[email protected]>

… files at the same time (opensearch-project#1458) Signed-off-by: Peter Zhu <[email protected]>

* Add more repos to offline scan Signed-off-by: Zelin Hao <[email protected]> * Simply the way we generate temp settings gradle Signed-off-by: Zelin Hao <[email protected]>

…pensearch-project#1462) Signed-off-by: dblock <[email protected]>

…ect#1581) * Adding checksum for latest snapshots artifacts Signed-off-by: Peter Zhu <[email protected]> * Change to snapshots dir name in path Signed-off-by: Peter Zhu <[email protected]> * Update test stack with new location Signed-off-by: Peter Zhu <[email protected]> * Simplify the naming in artifacts variables Signed-off-by: Peter Zhu <[email protected]> * Rename internal filename of sha512 so checksum pass Signed-off-by: Peter Zhu <[email protected]> * Update tests Signed-off-by: Peter Zhu <[email protected]> * Remove id number Signed-off-by: Peter Zhu <[email protected]> * Move sh block outside of withAWS block Signed-off-by: Peter Zhu <[email protected]>

…i_workflow (opensearch-project#1582) * Add python typechecking for ci-workflo Signed-off-by: Zelin Hao <[email protected]>

…rch-project#1517) * Hook up integ tests to distribution-build Signed-off-by: Tyler Ohlsen <[email protected]> * yamlfix test manifest file Signed-off-by: Tyler Ohlsen <[email protected]> * Remove unnecessary BuildManifest fns Signed-off-by: Tyler Ohlsen <[email protected]> * Remove runIntegTests(); refactor buildDockerImage vars as params Signed-off-by: Tyler Ohlsen <[email protected]> * Comment out notification block on dashboards dist build job Signed-off-by: Tyler Ohlsen <[email protected]> * Test out stabler integ test job Signed-off-by: Tyler Ohlsen <[email protected]> * Add test pipeline for pulling nested build info Signed-off-by: Tyler Ohlsen <[email protected]> * Remove test pipeline Signed-off-by: Tyler Ohlsen <[email protected]> * Add back notifications in OSD dist build Signed-off-by: Tyler Ohlsen <[email protected]> * Remove few extra test output lines Signed-off-by: Tyler Ohlsen <[email protected]> * Clean up buildDockerImage tests Signed-off-by: Tyler Ohlsen <[email protected]> * Update params in check-for-build cron; add icon in notifications Signed-off-by: Tyler Ohlsen <[email protected]> * Simplify notification Signed-off-by: Tyler Ohlsen <[email protected]> * Change references back to prod jenkins jobs Signed-off-by: Tyler Ohlsen <[email protected]>

…oject#1585) Signed-off-by: Tyler Ohlsen <[email protected]>

…nsearch-project#1590) * Add invalid job case in check-for-build * Remove catch block; pass TEST_MANIFEST for all jobs Signed-off-by: Tyler Ohlsen <[email protected]>

Signed-off-by: Abhinav Gupta <[email protected]>

Signed-off-by: dblock <[email protected]> Co-authored-by: dblock <[email protected]>

* Align sign off with DCO expectation Signed-off-by: Peter Nied <[email protected]> * Allow manually running of the workflow and set the author Signed-off-by: Peter Nied <[email protected]> * Revert dbock signoff, back to ci-bot Signed-off-by: Peter Nied <[email protected]> * Updating documentation with how to manually trigger Signed-off-by: Peter Nied <[email protected]>

…tests_manifests_workflow (opensearch-project#1599) * Add python type checking for manifest workflow Signed-off-by: Zelin Hao <[email protected]>

Signed-off-by: aviau <[email protected]>

* BWC test workflow Adding backwards compatability workflow for OpenSearch and OpenSearch Dashboards. Issue: opensearch-project#705 Signed-off-by: Kawika Avilla <[email protected]> * [BWC] Update docs and return cmd Signed-off-by: Kawika Avilla <[email protected]>

Signed-off-by: Abhinav Gupta <[email protected]>

Add additional documentation for the copyContainer function to make it clear what source/destinations can be used. Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Tyler Ohlsen <[email protected]>

* Migrate the maven sign release script to build repo Signed-off-by: Zelin Hao <[email protected]>

* Adding Release owner section to maintainers Describing the intention behind being a release owner and approaches to different situations that occur during the a release. Signed-off-by: Peter Nied <[email protected]>

…#1614) * Adding dashboards-security-plugin to 1.3 manifest Signed-off-by: Dave Lago <[email protected]>

…-project#1616) * adding Anomaly Detection to 1.3 Dashboard manifest Signed-off-by: Amit Galitzky <[email protected]> * linting issue Signed-off-by: Amit Galitzky <[email protected]> * fixing linting issue Signed-off-by: Amit Galitzky <[email protected]> * linting issue fix after merging main Signed-off-by: Amit Galitzky <[email protected]> Co-authored-by: Peter Nied <[email protected]>

Signed-off-by: Eugene Lee <[email protected]>

* Update maintainers list Signed-off-by: Peter Nied <[email protected]>

Signed-off-by: Tianle Huang <[email protected]>

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon · 2022-02-16T00:47:02Z

I somehow messed up this branch and could not recover during a rebase, will open a new PR soon.

peterzhuamazon · 2022-02-16T01:03:55Z

Reopen PR: #1629

Update build workflow to build based on distribution

ce1bdf2

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon added cicd rpm labels Feb 12, 2022

peterzhuamazon requested a review from dblock February 12, 2022 00:05

peterzhuamazon self-assigned this Feb 12, 2022

peterzhuamazon changed the title ~~Update build workflow to build based on distribution~~ [WIP] Update build workflow to build based on distribution Feb 12, 2022

remove rpmbuilder placeholder

d21194c

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon mentioned this pull request Feb 12, 2022

[RPM M1] The build workflow should generate the contents of the RPM packages as pre-requisite #1543

Closed

Make distribution default to None so no need to edit all scripts

65defa6

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon added 2 commits February 15, 2022 20:59

Update description

3604567

Signed-off-by: Peter Zhu <[email protected]>

Add distribution to build manifest.yml now

31598db

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon marked this pull request as ready for review February 15, 2022 22:40

peterzhuamazon requested a review from a team as a code owner February 15, 2022 22:40

peterzhuamazon changed the title ~~[WIP] Update build workflow to build based on distribution~~ Update build workflow to build based on distribution Feb 15, 2022

dblock requested changes Feb 15, 2022

View reviewed changes

peterzhuamazon requested a review from a team February 15, 2022 23:16

peterzhuamazon changed the title ~~Update build workflow to build based on distribution~~ Update build workflow to build based on distribution (OpenSearch-Core) Feb 15, 2022

peterzhuamazon changed the title ~~Update build workflow to build based on distribution (OpenSearch-Core)~~ Update build workflow to build based on distribution (OpenSearch) Feb 15, 2022

dblock and others added 7 commits February 16, 2022 00:40

Added missing components to 1.2.4. (opensearch-project#1448)

82eb8e9

Signed-off-by: dblock <[email protected]>

Data Prepper Unit Test & Test Refactoring (opensearch-project#1451)

77e4bea

* Added a unit test for the Data Prepper build. Signed-off-by: David Venable <[email protected]>

Added remaining components to 1.3.0 manifest. (opensearch-project#1449)

799ec9d

Signed-off-by: dblock <[email protected]>

Resolve opensearch-project#1130 where sed and tee is trying to access…

ee45344

… files at the same time (opensearch-project#1458) Signed-off-by: Peter Zhu <[email protected]>

Add more repos to offline whitesource scan (opensearch-project#1438)

d8f1e3e

* Add more repos to offline scan Signed-off-by: Zelin Hao <[email protected]> * Simply the way we generate temp settings gradle Signed-off-by: Zelin Hao <[email protected]>

Remove AD from manifest to get job-scheduler SPI uploaded to maven. (o…

fce0fe7

…pensearch-project#1462) Signed-off-by: dblock <[email protected]>

peterzhuamazon and others added 24 commits February 16, 2022 00:40

Add python typechecking for mypy on src/ci_workflow and tests/tests_c…

5ca9ada

…i_workflow (opensearch-project#1582) * Add python typechecking for ci-workflo Signed-off-by: Zelin Hao <[email protected]>

Add TEST_MANIFEST param to check-for-build jenkins job (opensearch-pr…

e72e8ed

…oject#1585) Signed-off-by: Tyler Ohlsen <[email protected]>

Add invalid job case & improve error handling in check-for-build (ope…

e910bf5

…nsearch-project#1590) * Add invalid job case in check-for-build * Remove catch block; pass TEST_MANIFEST for all jobs Signed-off-by: Tyler Ohlsen <[email protected]>

remove 1.2.4 from autobuild (opensearch-project#1596)

4c8b148

Signed-off-by: Abhinav Gupta <[email protected]>

Updated manifests. (opensearch-project#1595)

087253c

Signed-off-by: dblock <[email protected]> Co-authored-by: dblock <[email protected]>

Add python typechecking for mypy on src/manifests_workflow and tests/…

1af51aa

…tests_manifests_workflow (opensearch-project#1599) * Add python type checking for manifest workflow Signed-off-by: Zelin Hao <[email protected]>

1.0.0 changelog: mention no breaking changes (opensearch-project#1598)

e4bfc5e

Signed-off-by: aviau <[email protected]>

added support for crate files for crate.io (opensearch-project#1605)

1a33c77

Signed-off-by: Abhinav Gupta <[email protected]>

CopyContainer documentation (opensearch-project#1609)

eacfe2e

Add additional documentation for the copyContainer function to make it clear what source/destinations can be used. Signed-off-by: Peter Nied <[email protected]>

Remove 1.1.1 jobs in check-for-build (opensearch-project#1610)

436c74c

Signed-off-by: Tyler Ohlsen <[email protected]>

Migrating Maven sign and release jenkins jobs (opensearch-project#1520)

a706dc2

* Migrate the maven sign release script to build repo Signed-off-by: Zelin Hao <[email protected]>

Adding dashboards-security-plugin to 1.3 manifest (opensearch-project…

ece5923

…#1614) * Adding dashboards-security-plugin to 1.3 manifest Signed-off-by: Dave Lago <[email protected]>

Add observability and workbench to 1.3 (opensearch-project#1615)

727938c

Signed-off-by: Eugene Lee <[email protected]>

Add ganttCharts and reports (opensearch-project#1617)

297046d

Signed-off-by: Eugene Lee <[email protected]>

Update maintainers list (opensearch-project#1618)

d727a02

* Update maintainers list Signed-off-by: Peter Nied <[email protected]>

add permissions for index.json (opensearch-project#1620)

a72a4ff

Signed-off-by: Tianle Huang <[email protected]>

Add all tests

0c40e51

Signed-off-by: Peter Zhu <[email protected]>

Revert tests and also add changes based on comments

923c161

Signed-off-by: Peter Zhu <[email protected]>

peterzhuamazon force-pushed the opensearch-build-distribution-2 branch from 365068a to 923c161 Compare February 16, 2022 00:42

peterzhuamazon closed this Feb 16, 2022

peterzhuamazon mentioned this pull request Feb 16, 2022

[RPM M1] (Re-open) Update build workflow to build based on distribution (OpenSearch) #1629

Merged

1 task

peterzhuamazon deleted the opensearch-build-distribution-2 branch February 17, 2022 22:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update build workflow to build based on distribution (OpenSearch) #1619

Update build workflow to build based on distribution (OpenSearch) #1619

peterzhuamazon commented Feb 12, 2022 •

edited

Loading

dblock commented Feb 14, 2022

peterzhuamazon commented Feb 14, 2022

peterzhuamazon commented Feb 15, 2022

codecov-commenter commented Feb 15, 2022 •

edited

Loading

peterzhuamazon commented Feb 15, 2022

dblock left a comment

dblock Feb 15, 2022

peterzhuamazon Feb 15, 2022

peterzhuamazon Feb 16, 2022

peterzhuamazon Feb 16, 2022

dblock Feb 15, 2022

dblock Feb 15, 2022

peterzhuamazon Feb 15, 2022

dblock Feb 15, 2022

peterzhuamazon Feb 15, 2022

peterzhuamazon commented Feb 16, 2022

peterzhuamazon commented Feb 16, 2022

Update build workflow to build based on distribution (OpenSearch) #1619

Update build workflow to build based on distribution (OpenSearch) #1619

Conversation

peterzhuamazon commented Feb 12, 2022 • edited Loading

Description

Issues Resolved

Check List

dblock commented Feb 14, 2022

peterzhuamazon commented Feb 14, 2022

peterzhuamazon commented Feb 15, 2022

codecov-commenter commented Feb 15, 2022 • edited Loading

Codecov Report

peterzhuamazon commented Feb 15, 2022

dblock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peterzhuamazon commented Feb 16, 2022

peterzhuamazon commented Feb 16, 2022

peterzhuamazon commented Feb 12, 2022 •

edited

Loading

codecov-commenter commented Feb 15, 2022 •

edited

Loading