Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: explicitly authenticate AWS #1960

Merged
merged 7 commits into from
Jul 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/actions/docker/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,12 @@ inputs:
region:
description: S3 bucket region
required: true
aws_access_key_id:
description: AWS access key ID
required: true
aws_secret_access_key:
description: AWS secret access key
required: true
outputs:
digest:
value: ${{ steps.docker_build.outputs.digest }}
Expand Down Expand Up @@ -102,6 +108,8 @@ runs:
SCCACHE_BUCKET=${{ inputs.bucket }}
SCCACHE_REGION=${{ inputs.region }}
SCCACHE_S3_KEY_PREFIX=${{ runner.os }}/sccache
AWS_ACCESS_KEY_ID=${{ inputs.aws_access_key_id }}
AWS_SECRET_ACCESS_KEY=${{ inputs.aws_secret_access_key }}
cache-from: ${{ steps.layer_cache_settings.outputs.cache_from }}
cache-to: ${{ steps.layer_cache_settings.outputs.cache_to }}
outputs: type=image,name=${{ inputs.image_org }}/${{ inputs.image_name }},push-by-digest=${{ inputs.push_tags != 'true' }},name-canonical=true,push=true
Expand Down
15 changes: 14 additions & 1 deletion .github/actions/rust/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,13 +49,26 @@ runs:
esac

# TODO: Move to AMI and build every day
- name: Check if protoc is installed
id: check-protoc
shell: bash
run: |
if command -v protoc >/dev/null 2>&1; then
echo "protoc is already installed."
echo "protoc_installed=true" >> $GITHUB_OUTPUT
else
echo "protoc is not installed."
echo "protoc_installed=false" >> $GITHUB_OUTPUT
fi

- name: Install protoc
if: steps.check-protoc.outputs.protoc_installed == 'false'
id: deps-protoc
shell: bash
run: |
curl -Lo /tmp/protoc.zip \
"https://github.com/protocolbuffers/protobuf/releases/download/v25.2/protoc-25.2-linux-${{ steps.protoc_arch.outputs.arch }}.zip"
unzip /tmp/protoc.zip -d ${HOME}/.local
unzip -o /tmp/protoc.zip -d ${HOME}/.local
echo "PROTOC=${HOME}/.local/bin/protoc" >> $GITHUB_ENV
export PATH="${PATH}:${HOME}/.local/bin"

Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/release-docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Build and push by digest
uses: ./.github/actions/docker
Expand All @@ -56,7 +58,7 @@ jobs:
cargo_profile: release
dockerhub_username: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub_token: ${{ secrets.DOCKERHUB_TOKEN }}
region: ${{ vars.AWS_REGION }}
region: ${{ secrets.AWS_REGION }}

- name: Export digest
run: |
Expand Down
11 changes: 7 additions & 4 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Retrieve JS build artifacts
uses: strophy/actions-cache@opendal-update
Expand Down Expand Up @@ -69,7 +71,7 @@ jobs:
CARGO_BUILD_PROFILE: release
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: multi-runner-cache-x1xibo9c
SCCACHE_REGION: ${{ vars.AWS_REGION }}
SCCACHE_REGION: ${{ secrets.AWS_REGION }}
SCCACHE_S3_KEY_PREFIX: ${{ runner.os }}/sccache/wasm/wasm32
if: ${{ steps.cache.outputs.cache-hit != 'true' }}

Expand Down Expand Up @@ -193,8 +195,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME }}
aws-region: ${{ secrets.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Retrieve JS build artifacts
uses: strophy/actions-cache@opendal-update
Expand Down
12 changes: 8 additions & 4 deletions .github/workflows/tests-build-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,24 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Login to ECR
run: aws ecr get-login-password --region ${{ vars.AWS_REGION }} | docker login --username AWS --password-stdin ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com
run: aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
shell: bash

- name: Build and push by SHA
uses: ./.github/actions/docker
with:
image_name: ${{ inputs.image_name }}
image_org: ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com
image_org: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
target: ${{ inputs.target }}
platform: linux/arm64
push_tags: true
dockerhub_username: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub_token: ${{ secrets.DOCKERHUB_TOKEN }}
region: ${{ vars.AWS_REGION }}
region: ${{ secrets.AWS_REGION }}
aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
6 changes: 4 additions & 2 deletions .github/workflows/tests-build-js.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

# TODO: Use upload artifacts action instead
- name: Cache build artifacts
Expand Down Expand Up @@ -50,7 +52,7 @@ jobs:
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: multi-runner-cache-x1xibo9c
SCCACHE_REGION: ${{ vars.AWS_REGION }}
SCCACHE_REGION: ${{ secrets.AWS_REGION }}
SCCACHE_S3_KEY_PREFIX: ${{ runner.os }}/sccache/wasm/wasm32

- name: Ignore only already cached artifacts
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/tests-codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Setup Node.JS
uses: ./.github/actions/nodejs
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/tests-dashmate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Setup Node.JS
uses: ./.github/actions/nodejs
Expand All @@ -56,8 +58,8 @@ jobs:
set -x

# Login to ECR
DOCKER_HUB_ORG="${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com"
aws ecr get-login-password --region ${{ vars.AWS_REGION }} | docker login --username AWS --password-stdin $DOCKER_HUB_ORG
DOCKER_HUB_ORG="${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com"
aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin $DOCKER_HUB_ORG

SHA_TAG=sha-${{ github.sha }}

Expand Down
10 changes: 6 additions & 4 deletions .github/workflows/tests-js-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Setup Node.JS
uses: ./.github/actions/nodejs
Expand Down Expand Up @@ -64,8 +65,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME }}
aws-region: ${{ secrets.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Setup Node.JS
uses: ./.github/actions/nodejs
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/tests-packges-functional.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
timeout-minutes: 15
env:
CHROME_BIN: /usr/bin/brave-browser
ECR_HOST: ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com
ECR_HOST: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
steps:
- name: Check out repo
uses: actions/checkout@v4
Expand All @@ -24,10 +24,12 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Login to ECR
run: aws ecr get-login-password --region ${{ vars.AWS_REGION }} | docker login --username AWS --password-stdin ${{ env.ECR_HOST }}
run: aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ env.ECR_HOST }}

- name: Start local network
uses: ./.github/actions/local-network
Expand Down
26 changes: 16 additions & 10 deletions .github/workflows/tests-rs-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Setup Rust
uses: ./.github/actions/rust
Expand All @@ -47,7 +48,7 @@ jobs:
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: multi-runner-cache-x1xibo9c
SCCACHE_REGION: ${{ vars.AWS_REGION }}
SCCACHE_REGION: ${{ secrets.AWS_REGION }}
SCCACHE_S3_KEY_PREFIX: ${{ runner.os }}/sccache/${{ runner.arch }}/linux-gnu

formatting:
Expand Down Expand Up @@ -81,8 +82,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ vars.AWS_ROLE_TO_ASSUME }}
aws-region: ${{ secrets.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Setup Rust
uses: ./.github/actions/rust
Expand All @@ -98,7 +100,7 @@ jobs:
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: multi-runner-cache-x1xibo9c
SCCACHE_REGION: ${{ vars.AWS_REGION }}
SCCACHE_REGION: ${{ secrets.AWS_REGION }}
SCCACHE_S3_KEY_PREFIX: ${{ runner.os }}/sccache/${{ runner.arch }}/linux-gnu
with:
args: ${{ steps.crate_info.outputs.cargo_manifest_dir }}
Expand Down Expand Up @@ -175,7 +177,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-region: ${{ secrets.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Setup Rust
uses: ./.github/actions/rust
Expand All @@ -185,7 +189,7 @@ jobs:
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: multi-runner-cache-x1xibo9c
SCCACHE_REGION: ${{ vars.AWS_REGION }}
SCCACHE_REGION: ${{ secrets.AWS_REGION }}
SCCACHE_S3_KEY_PREFIX: ${{ runner.os }}/sccache/${{ runner.arch }}/linux-gnu

check_each_feature:
Expand All @@ -200,7 +204,9 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-region: ${{ secrets.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

- name: Setup Rust
uses: ./.github/actions/rust
Expand All @@ -215,7 +221,7 @@ jobs:
env:
RUSTC_WRAPPER: sccache
SCCACHE_BUCKET: multi-runner-cache-x1xibo9c
SCCACHE_REGION: ${{ vars.AWS_REGION }}
SCCACHE_REGION: ${{ secrets.AWS_REGION }}
SCCACHE_S3_KEY_PREFIX: ${{ runner.os }}/sccache/${{ runner.arch }}/linux-gnu
run: |
echo Verify all features disabled
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/tests-test-suite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
timeout-minutes: 15
env:
CHROME_BIN: /usr/bin/brave-browser
ECR_HOST: ${{ vars.AWS_ACCOUNT_ID }}.dkr.ecr.${{ vars.AWS_REGION }}.amazonaws.com
ECR_HOST: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
steps:
- name: Check out repo
uses: actions/checkout@v4
Expand All @@ -42,10 +42,12 @@ jobs:
- name: Configure AWS credentials and bucket region
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Login to ECR
run: aws ecr get-login-password --region ${{ vars.AWS_REGION }} | docker login --username AWS --password-stdin ${{ env.ECR_HOST }}
run: aws ecr get-login-password --region ${{ secrets.AWS_REGION }} | docker login --username AWS --password-stdin ${{ env.ECR_HOST }}

- name: Start local network
uses: ./.github/actions/local-network
Expand Down
7 changes: 5 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# The following build arguments can be provided using --build-arg:
# - CARGO_BUILD_PROFILE - set to `release` to build final binary, without debugging information
# - NODE_ENV - node.js environment name to use to build the library
# - RUSTC_WRAPPER - set to `sccache` to enable sccache support and make the following variables avaialable:
# - RUSTC_WRAPPER - set to `sccache` to enable sccache support and make the following variables available:
# - SCCACHE_GHA_ENABLED, ACTIONS_CACHE_URL, ACTIONS_RUNTIME_TOKEN - store sccache caches inside github actions
# - SCCACHE_MEMCACHED - set to memcache server URI (eg. tcp://172.17.0.1:11211) to enable sccache memcached backend
# - ALPINE_VERSION - use different version of Alpine base image; requires also rust:apline...
Expand Down Expand Up @@ -138,8 +138,11 @@ ENV SCCACHE_REGION=${SCCACHE_REGION}
ARG CARGO_INCREMENTAL=false
ENV CARGO_INCREMENTAL=${CARGO_INCREMENTAL}

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY

#
# DEPS: FULL DEPENCIES LIST
# DEPS: FULL DEPENDENCIES LIST
#
# This is separate from `deps` to use sccache for caching
FROM deps-${RUSTC_WRAPPER:-base} AS deps
Expand Down
Loading