-
Notifications
You must be signed in to change notification settings - Fork 740
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Release/CI] Github flow to build polkadot
/polkadot-parachain
rc binaries and deb package
#5963
Merged
EgorPopelyaev
merged 64 commits into
paritytech:master
from
EgorPopelyaev:ep-add-binaries-build-pipeline
Oct 9, 2024
Merged
Changes from all commits
Commits
Show all changes
64 commits
Select commit
Hold shift + click to select a range
668a114
add build release binary pipeline
EgorPopelyaev b9621b1
test checkout
EgorPopelyaev ea371fa
set bash as a shell for import gpg
EgorPopelyaev 5c216b2
use pgpkms directly
EgorPopelyaev b7b56c6
debug signing
EgorPopelyaev b05fc6f
test signing
EgorPopelyaev 20b5dbf
fix typos
EgorPopelyaev 73874df
fix typo
EgorPopelyaev 41b2b37
add upload artifacts
EgorPopelyaev 6c81ed7
test binary build
EgorPopelyaev af20353
fix signature
EgorPopelyaev b80d242
add steps to build polkadot-parachain and upload to s3
EgorPopelyaev 19afe85
change github.workspace to "${GITHUB_WORKSPACE}" due to the bug in git
EgorPopelyaev 9d6b12b
setting shell as bash explicitly for some steps
EgorPopelyaev fca6718
fix aws region
EgorPopelyaev dbbb141
fix script name
EgorPopelyaev 80fe8f2
add artifacts download path
EgorPopelyaev e104d72
chnage download path
EgorPopelyaev ebab693
test with real binaries
EgorPopelyaev 026f4af
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev 63843cb
fix package name
EgorPopelyaev 85b0691
test without polkadot-parachain
EgorPopelyaev 83fd636
move common parts to a reusable workflow
EgorPopelyaev 7f5cb2c
use fromJson
EgorPopelyaev 43c918c
add package as parameter
EgorPopelyaev 73f40bb
fix
EgorPopelyaev 8f42d25
fix quots
EgorPopelyaev f94a04f
move envs inside a build job
EgorPopelyaev 0ae01e9
define secrets
EgorPopelyaev a96bf6f
use one bucket
EgorPopelyaev 1844de7
test with real polka bin
EgorPopelyaev 81d92a0
add build polkadot deb step
EgorPopelyaev 29c8515
small fixes
EgorPopelyaev be4ccdf
make build deb as a separate job
EgorPopelyaev 8085459
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev 177d3da
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev 745ab13
Extract upload to s3 to reusable action
EgorPopelyaev d9a17f1
change rc release_tg description
EgorPopelyaev 04eeafe
roll back the checkout action version
EgorPopelyaev 54fac32
updet checkout action to 4.2.0
EgorPopelyaev 514cb53
add GITHUB_WORKSPACE to the git safe.directory
EgorPopelyaev c7792cd
put git config in the right place
EgorPopelyaev a5a7ebb
test
EgorPopelyaev 666bd52
add path to the deb build
EgorPopelyaev eab78f4
debug
EgorPopelyaev 795c586
debug
EgorPopelyaev 78e9df2
test download with merge-multiple
EgorPopelyaev dfba403
change path to artifacts in target
EgorPopelyaev 4e2bb0a
fix in build-deb.sh script
EgorPopelyaev f5b112a
adjust uplod with deb
EgorPopelyaev b99fed1
add attestaion
EgorPopelyaev 87e0a6e
download only specific artifacts for s3 upload
EgorPopelyaev e7c96cf
cleanup
EgorPopelyaev c030fab
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev 91bfeab
add input validation and environments
EgorPopelyaev f599fd2
activate check-workflow-can-run
EgorPopelyaev 91103f9
fix check sync
EgorPopelyaev 31f52de
replace tag with branch
EgorPopelyaev a7de7ba
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev 025c5a6
use cargo-deb with particular version
EgorPopelyaev eb34aec
address PR comments
EgorPopelyaev 4789c53
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev 66a89c3
fix deb version
EgorPopelyaev 3c52c45
change if
EgorPopelyaev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
#!/usr/bin/env bash | ||
set -e | ||
|
||
PRODUCT=$1 | ||
VERSION=$2 | ||
PROFILE=${PROFILE:-production} | ||
|
||
cargo install --version 2.7.0 cargo-deb --locked -q | ||
echo "Using cargo-deb v$(cargo-deb --version)" | ||
echo "Building a Debian package for '$PRODUCT' in '$PROFILE' profile" | ||
|
||
# we need to start the custom version with a didgit as requires it cargo-deb | ||
cargo deb --profile $PROFILE --no-strip --no-build -p $PRODUCT --deb-version 1-$VERSION | ||
|
||
deb=target/debian/$PRODUCT_*_amd64.deb | ||
|
||
cp $deb target/production/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
#!/usr/bin/env bash | ||
|
||
# This is used to build our binaries: | ||
# - polkadot | ||
# - polkadot-parachain | ||
# set -e | ||
|
||
BIN=$1 | ||
PACKAGE=${2:-$BIN} | ||
|
||
PROFILE=${PROFILE:-production} | ||
ARTIFACTS=/artifacts/$BIN | ||
VERSION=$(git tag -l --contains HEAD | grep -E "^v.*") | ||
|
||
echo "Artifacts will be copied into $ARTIFACTS" | ||
mkdir -p "$ARTIFACTS" | ||
|
||
git log --pretty=oneline -n 1 | ||
time cargo build --profile $PROFILE --locked --verbose --bin $BIN --package $PACKAGE | ||
|
||
echo "Artifact target: $ARTIFACTS" | ||
|
||
cp ./target/$PROFILE/$BIN "$ARTIFACTS" | ||
pushd "$ARTIFACTS" > /dev/nul | ||
sha256sum "$BIN" | tee "$BIN.sha256" | ||
|
||
EXTRATAG="$($ARTIFACTS/$BIN --version | | ||
sed -n -r 's/^'$BIN' ([0-9.]+.*-[0-9a-f]{7,13})-.*$/\1/p')" | ||
|
||
EXTRATAG="${VERSION}-${EXTRATAG}-$(cut -c 1-8 $ARTIFACTS/$BIN.sha256)" | ||
|
||
echo "$BIN version = ${VERSION} (EXTRATAG = ${EXTRATAG})" | ||
echo -n ${VERSION} > "$ARTIFACTS/VERSION" | ||
echo -n ${EXTRATAG} > "$ARTIFACTS/EXTRATAG" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,74 @@ | ||
name: Release - Build node release candidate | ||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
binary: | ||
description: Binary to be build for the release | ||
default: all | ||
type: choice | ||
options: | ||
- polkadot | ||
- polkadot-parachain | ||
- all | ||
|
||
release_tag: | ||
description: Tag matching the actual release candidate with the format stableYYMM-rcX or stableYYMM | ||
type: string | ||
|
||
jobs: | ||
check-synchronization: | ||
uses: paritytech-release/sync-workflows/.github/workflows/check-syncronization.yml@main | ||
|
||
validate-inputs: | ||
needs: [check-synchronization] | ||
if: needs.check-synchronization.outputs.checks_passed == 'true' | ||
runs-on: ubuntu-latest | ||
outputs: | ||
release_tag: ${{ steps.validate_inputs.outputs.release_tag }} | ||
|
||
steps: | ||
- name: Checkout sources | ||
uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0 | ||
|
||
- name: Validate inputs | ||
id: validate_inputs | ||
run: | | ||
. ./.github/scripts/common/lib.sh | ||
|
||
RELEASE_TAG=$(validate_stable_tag ${{ inputs.release_tag }}) | ||
echo "release_tag=${RELEASE_TAG}" >> $GITHUB_OUTPUT | ||
|
||
build-polkadot-binary: | ||
needs: [validate-inputs] | ||
if: ${{ inputs.binary == 'polkadot' || inputs.binary == 'all' }} | ||
uses: "./.github/workflows/release-reusable-rc-buid.yml" | ||
with: | ||
binary: '["polkadot", "polkadot-prepare-worker", "polkadot-execute-worker"]' | ||
package: polkadot | ||
release_tag: ${{ needs.validate-inputs.outputs.release_tag }} | ||
secrets: | ||
PGP_KMS_KEY: ${{ secrets.PGP_KMS_KEY }} | ||
PGP_KMS_HASH: ${{ secrets.PGP_KMS_HASH }} | ||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} | ||
AWS_RELEASE_ACCESS_KEY_ID: ${{ secrets.AWS_RELEASE_ACCESS_KEY_ID }} | ||
AWS_RELEASE_SECRET_ACCESS_KEY: ${{ secrets.AWS_RELEASE_SECRET_ACCESS_KEY }} | ||
|
||
build-polkadot-parachain-binary: | ||
needs: [validate-inputs] | ||
if: ${{ inputs.binary == 'polkadot-parachain' || inputs.binary == 'all' }} | ||
uses: "./.github/workflows/release-reusable-rc-buid.yml" | ||
with: | ||
binary: '["polkadot-parachain"]' | ||
package: "polkadot-parachain-bin" | ||
release_tag: ${{ needs.validate-inputs.outputs.release_tag }} | ||
secrets: | ||
PGP_KMS_KEY: ${{ secrets.PGP_KMS_KEY }} | ||
PGP_KMS_HASH: ${{ secrets.PGP_KMS_HASH }} | ||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} | ||
AWS_RELEASE_ACCESS_KEY_ID: ${{ secrets.AWS_RELEASE_ACCESS_KEY_ID }} | ||
AWS_RELEASE_SECRET_ACCESS_KEY: ${{ secrets.AWS_RELEASE_SECRET_ACCESS_KEY }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,191 @@ | ||
name: RC Build | ||
|
||
on: | ||
workflow_call: | ||
inputs: | ||
binary: | ||
description: Binary to be build for the release | ||
required: true | ||
default: polkadot | ||
type: string | ||
|
||
package: | ||
description: Package to be built, for now is either polkadot or polkadot-parachain-bin | ||
required: true | ||
type: string | ||
|
||
release_tag: | ||
description: Tag matching the actual release candidate with the format stableYYMM-rcX or stableYYMM | ||
required: true | ||
type: string | ||
|
||
secrets: | ||
PGP_KMS_KEY: | ||
required: true | ||
PGP_KMS_HASH: | ||
required: true | ||
AWS_ACCESS_KEY_ID: | ||
required: true | ||
AWS_SECRET_ACCESS_KEY: | ||
required: true | ||
AWS_DEFAULT_REGION: | ||
required: true | ||
AWS_RELEASE_ACCESS_KEY_ID: | ||
required: true | ||
AWS_RELEASE_SECRET_ACCESS_KEY: | ||
required: true | ||
|
||
permissions: | ||
id-token: write | ||
contents: read | ||
attestations: write | ||
|
||
jobs: | ||
|
||
set-image: | ||
# GitHub Actions allows using 'env' in a container context. | ||
# However, env variables don't work for forks: https://github.com/orgs/community/discussions/44322 | ||
# This workaround sets the container image for each job using 'set-image' job output. | ||
runs-on: ubuntu-latest | ||
outputs: | ||
IMAGE: ${{ steps.set_image.outputs.IMAGE }} | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0 | ||
|
||
- id: set_image | ||
run: cat .github/env >> $GITHUB_OUTPUT | ||
|
||
build-rc: | ||
needs: [set-image] | ||
runs-on: ubuntu-latest | ||
environment: release | ||
container: | ||
image: ${{ needs.set-image.outputs.IMAGE }} | ||
strategy: | ||
matrix: | ||
binaries: ${{ fromJSON(inputs.binary) }} | ||
env: | ||
PGP_KMS_KEY: ${{ secrets.PGP_KMS_KEY }} | ||
PGP_KMS_HASH: ${{ secrets.PGP_KMS_HASH }} | ||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} | ||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} | ||
|
||
steps: | ||
- name: Install pgpkkms | ||
run: | | ||
# Install pgpkms that is used to sign built artifacts | ||
python3 -m pip install "pgpkms @ git+https://github.com/paritytech-release/pgpkms.git@5a8f82fbb607ea102d8c178e761659de54c7af69" | ||
which pgpkms | ||
|
||
- name: Checkout sources | ||
uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0 | ||
with: | ||
ref: ${{ inputs.release_tag }} | ||
fetch-depth: 0 | ||
|
||
- name: Import gpg keys | ||
shell: bash | ||
run: | | ||
. ./.github/scripts/common/lib.sh | ||
|
||
import_gpg_keys | ||
|
||
- name: Build binary | ||
run: | | ||
git config --global --add safe.directory "${GITHUB_WORKSPACE}" #avoid "detected dubious ownership" error | ||
./.github/scripts/release/build-linux-release.sh ${{ matrix.binaries }} ${{ inputs.package }} | ||
|
||
- name: Generate artifact attestation | ||
uses: actions/attest-build-provenance@1c608d11d69870c2092266b3f9a6f3abbf17002c # v1.4.3 | ||
with: | ||
subject-path: /artifacts/${{ matrix.binaries }}/${{ matrix.binaries }} | ||
|
||
- name: Sign artifacts | ||
working-directory: /artifacts/${{ matrix.binaries }} | ||
run: | | ||
python3 -m pgpkms sign --input ${{matrix.binaries }} -o ${{ matrix.binaries }}.asc | ||
|
||
- name: Check sha256 ${{ matrix.binaries }} | ||
working-directory: /artifacts/${{ matrix.binaries }} | ||
shell: bash | ||
run: | | ||
. "${GITHUB_WORKSPACE}"/.github/scripts/common/lib.sh | ||
|
||
echo "Checking binary ${{ matrix.binaries }}" | ||
check_sha256 ${{ matrix.binaries }} | ||
|
||
- name: Check GPG ${{ matrix.binaries }} | ||
working-directory: /artifacts/${{ matrix.binaries }} | ||
shell: bash | ||
run: | | ||
. "${GITHUB_WORKSPACE}"/.github/scripts/common/lib.sh | ||
|
||
check_gpg ${{ matrix.binaries }} | ||
|
||
- name: Upload ${{ matrix.binaries }} artifacts | ||
uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1 | ||
with: | ||
name: ${{ matrix.binaries }} | ||
path: /artifacts/${{ matrix.binaries }} | ||
|
||
build-polkadot-deb-package: | ||
if: ${{ inputs.package == 'polkadot' }} | ||
needs: [build-rc] | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: Checkout sources | ||
uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0 | ||
with: | ||
ref: ${{ inputs.release_tag }} | ||
fetch-depth: 0 | ||
|
||
- name: Download artifacts | ||
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 | ||
with: | ||
path: target/production | ||
merge-multiple: true | ||
|
||
- name: Build polkadot deb package | ||
shell: bash | ||
run: | | ||
. "${GITHUB_WORKSPACE}"/.github/scripts/release/build-deb.sh ${{ inputs.package }} ${{ inputs.release_tag }} | ||
|
||
- name: Generate artifact attestation | ||
uses: actions/attest-build-provenance@1c608d11d69870c2092266b3f9a6f3abbf17002c # v1.4.3 | ||
with: | ||
subject-path: target/production/*.deb | ||
|
||
- name: Upload ${{inputs.package }} artifacts | ||
uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1 | ||
with: | ||
name: ${{ inputs.package }} | ||
path: target/production | ||
overwrite: true | ||
|
||
upload-polkadot-artifacts-to-s3: | ||
if: ${{ inputs.package == 'polkadot' }} | ||
needs: [build-polkadot-deb-package] | ||
uses: ./.github/workflows/release-reusable-s3-upload.yml | ||
with: | ||
package: ${{ inputs.package }} | ||
release_tag: ${{ inputs.release_tag }} | ||
secrets: | ||
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} | ||
AWS_RELEASE_ACCESS_KEY_ID: ${{ secrets.AWS_RELEASE_ACCESS_KEY_ID }} | ||
AWS_RELEASE_SECRET_ACCESS_KEY: ${{ secrets.AWS_RELEASE_SECRET_ACCESS_KEY }} | ||
|
||
|
||
upload-polkadot-parachain-artifacts-to-s3: | ||
if: ${{ inputs.package == 'polkadot-parachain-bin' }} | ||
needs: [build-rc] | ||
uses: ./.github/workflows/release-reusable-s3-upload.yml | ||
with: | ||
package: ${{ inputs.binary }} | ||
release_tag: ${{ inputs.release_tag }} | ||
secrets: | ||
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} | ||
AWS_RELEASE_ACCESS_KEY_ID: ${{ secrets.AWS_RELEASE_ACCESS_KEY_ID }} | ||
AWS_RELEASE_SECRET_ACCESS_KEY: ${{ secrets.AWS_RELEASE_SECRET_ACCESS_KEY }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
name: Upload to s3 | ||
|
||
on: | ||
workflow_call: | ||
inputs: | ||
package: | ||
description: Package to be built, for now is either polkadot or polkadot-parachain-bin | ||
required: true | ||
type: string | ||
|
||
release_tag: | ||
description: Tag matching the actual release candidate with the format stableYYMM-rcX or stableYYMM-rcX | ||
required: true | ||
type: string | ||
|
||
secrets: | ||
AWS_DEFAULT_REGION: | ||
required: true | ||
AWS_RELEASE_ACCESS_KEY_ID: | ||
required: true | ||
AWS_RELEASE_SECRET_ACCESS_KEY: | ||
required: true | ||
|
||
jobs: | ||
upload-artifacts-to-s3: | ||
runs-on: ubuntu-latest | ||
environment: release | ||
env: | ||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_RELEASE_ACCESS_KEY_ID }} | ||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_RELEASE_SECRET_ACCESS_KEY }} | ||
AWS_REGION: ${{ secrets.AWS_DEFAULT_REGION }} | ||
|
||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@d632683dd7b4114ad314bca15554477dd762a938 # v4.2.0 | ||
|
||
- name: Download artifacts | ||
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8 | ||
with: | ||
name: ${{ inputs.package }} | ||
path: artifacts/${{ inputs.package }} | ||
|
||
- name: Configure AWS Credentials | ||
uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502 # v4.0.2 | ||
with: | ||
aws-access-key-id: ${{ env.AWS_ACCESS_KEY_ID }} | ||
aws-secret-access-key: ${{ env.AWS_SECRET_ACCESS_KEY }} | ||
aws-region: ${{ env.AWS_REGION }} | ||
|
||
- name: Upload ${{ inputs.package }} artifacts to s3 | ||
run: | | ||
. ./.github/scripts/release/release_lib.sh | ||
upload_s3_release ${{ inputs.package }} ${{ inputs.release_tag }} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is it safe/ok that anyone in org (even accidentally) can run this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This won't be the case, cause the setup is going so that it will be possible to trigger it only from the new paritytech-release org