Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: build nitro docker image #19

Merged
merged 51 commits into from
Dec 11, 2023
Merged
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
295e287
CI: build nitro docker image
sveitser Dec 4, 2023
3f8520c
CI: add nitro-node-dev docker image, cache layers
sveitser Dec 4, 2023
5eba11d
CI: cancel in progress
sveitser Dec 4, 2023
2d39093
CI: install QEMU
sveitser Dec 4, 2023
69b5c86
CI: don't cross build with qemu
sveitser Dec 4, 2023
b26465c
CI: Fix yml
sveitser Dec 4, 2023
86b0781
Fix yaml (again)
sveitser Dec 4, 2023
8d32bc9
CI: move action to separate directory
sveitser Dec 4, 2023
bd0d3b5
Fix action path
sveitser Dec 4, 2023
9d8fd2b
Move login action to workflow
sveitser Dec 4, 2023
c14dda2
CI: remove docker caching, merge images
sveitser Dec 4, 2023
b107dba
CI: specify shell in composite action
sveitser Dec 4, 2023
9b92670
CI: use dummy image for testing
sveitser Dec 5, 2023
cd3f28b
CI: disable build workflow for testing
sveitser Dec 5, 2023
a8c37a5
CI: remove tags
sveitser Dec 5, 2023
d85067c
Give build step an ID to use it later
sveitser Dec 5, 2023
1283316
Build both images
sveitser Dec 5, 2023
e1c7ba2
Remove env vars
sveitser Dec 5, 2023
14a595f
Use runner.temp instead of /tmp
sveitser Dec 5, 2023
b6e5188
Include target in digests dir
sveitser Dec 5, 2023
5b1b2e7
Restore real Dockerfile
sveitser Dec 5, 2023
309ccf0
CI: use self hosted runner for docker build
sveitser Dec 5, 2023
16ad07b
Maybe fix submodule permissions
sveitser Dec 5, 2023
771e361
Change owners of working directory
Ancient123 Dec 5, 2023
08ec142
CI: enable GHA docker cache
sveitser Dec 6, 2023
ce284c3
CI: export wavm machine
sveitser Dec 6, 2023
dab175f
CI: add back permissions fix
sveitser Dec 6, 2023
7450e76
Fix image name, disable pull to use local image
sveitser Dec 6, 2023
003f0b7
Pull dev node image by digest
sveitser Dec 6, 2023
24850d2
CI: debug, show files in directory
sveitser Dec 6, 2023
9709a7f
Debug permissions
sveitser Dec 6, 2023
6e31cfa
CI: remove all files in working directory
sveitser Dec 6, 2023
87b50cf
Only remove target directory
sveitser Dec 6, 2023
ee1aa25
CI: maybe root can remove the files
sveitser Dec 6, 2023
e254cfe
Add trailing dot to prevent creating dir on host
sveitser Dec 6, 2023
160781a
Fix module root step output
sveitser Dec 6, 2023
bacdf1d
Merge remote-tracking branch 'origin/integration' into ma/nitro-docker
sveitser Dec 11, 2023
5a288b9
Only export artifact for AMD64
sveitser Dec 11, 2023
3ba4b4b
Renable steps on PR
sveitser Dec 11, 2023
c213673
Fix yaml
sveitser Dec 11, 2023
3787dd0
Add custom Dockerfile for espresso build
sveitser Dec 11, 2023
9516515
Document how wasm machine download script works
sveitser Dec 11, 2023
9e4a61c
Remove TODOs
sveitser Dec 11, 2023
d3b1273
CI: re-enable test workflow
sveitser Dec 11, 2023
f782c27
CI: disable downloading of espresso wasm binary
sveitser Dec 11, 2023
4af8e5d
Undo unwanted changes
sveitser Dec 11, 2023
cd5216f
CI: fix triggers
sveitser Dec 11, 2023
7041720
CI: fix yaml
sveitser Dec 11, 2023
e28e7e3
Add back permission workaround
sveitser Dec 11, 2023
6134e0e
Fix yaml
sveitser Dec 11, 2023
622496a
Remove pull request triggers
sveitser Dec 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CI: re-enable test workflow
sveitser committed Dec 11, 2023

Verified

This commit was signed with the committer’s verified signature.
sveitser Mathis
commit d3b127332116ff9210a71400f2f903bb1d2f917b
384 changes: 192 additions & 192 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,192 +1,192 @@
# name: Go tests CI
# run-name: Go tests CI triggered from @${{ github.actor }} of ${{ github.head_ref }}

# on:
# workflow_dispatch:
# merge_group:
# pull_request:
# push:
# branches:
# - master
# - develop
# - integration
# tags:
# # YYYYMMDD
# - "20[0-9][0-9][0-1][0-9][0-3][0-9]*"

# concurrency:
# group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
# cancel-in-progress: true

# jobs:
# test:
# name: Go Tests
# container:
# image: ghcr.io/catthehacker/ubuntu:js-22.04
# runs-on: [self-hosted, X64]

# # Creates a redis container for redis tests
# services:
# redis:
# image: redis
# ports:
# - 6379:6379

# strategy:
# fail-fast: false
# matrix:
# test-mode: [defaults, race, challenge]

# steps:
# - name: Checkout
# uses: actions/checkout@v3
# with:
# submodules: true

# - name: Install dependencies
# run: >
# sudo apt update && sudo apt install -y wabt gotestsum
# cmake build-essential bison golang clang make wabt

# - name: Setup nodejs
# uses: actions/setup-node@v3
# with:
# node-version: '16'
# cache: 'yarn'
# cache-dependency-path: '**/yarn.lock'

# - name: Install go
# uses: actions/setup-go@v4
# with:
# go-version: 1.20.x

# - name: Install wasm-ld
# run: |
# sudo apt-get update && sudo apt-get install -y lld-14
# sudo ln -s /usr/bin/wasm-ld-14 /usr/local/bin/wasm-ld

# - name: Install rust wasm32-unknown-unknown
# uses: actions-rs/toolchain@v1
# with:
# profile: minimal
# toolchain: "stable"
# target: wasm32-unknown-unknown

# - name: Install rust wasm32-wasi
# uses: actions-rs/toolchain@v1
# with:
# profile: minimal
# toolchain: "stable"
# target: wasm32-wasi

# - name: Install rust stable
# uses: actions-rs/toolchain@v1
# id: install-rust
# with:
# profile: minimal
# toolchain: "stable"
# override: true

# - name: Install cbindgen
# run: cargo install cbindgen

# - name: Cache Build Products
# uses: actions/cache@v3
# with:
# path: |
# ~/go/pkg/mod
# ~/.cache/go-build
# key: ${{ runner.os }}-go-${{ hashFiles('go.sum') }}-${{ matrix.test-mode }}
# restore-keys: ${{ runner.os }}-go-

# - name: Cache Rust Build Products
# uses: actions/cache@v3
# with:
# path: |
# ~/.cargo/registry/
# ~/.cargo/git/
# arbitrator/target/
# arbitrator/wasm-libraries/target/
# arbitrator/wasm-libraries/soft-float/SoftFloat/build
# target/etc/initial-machine-cache/
# key: ${{ runner.os }}-cargo-${{ steps.install-rust.outputs.rustc_hash }}-min-${{ hashFiles('arbitrator/Cargo.lock') }}-${{ matrix.test-mode }}
# restore-keys: ${{ runner.os }}-cargo-${{ steps.install-rust.outputs.rustc_hash }}-

# - name: Cache cbrotli
# uses: actions/cache@v3
# id: cache-cbrotli
# with:
# path: |
# target/include/brotli/
# target/lib-wasm/
# target/lib/libbrotlicommon-static.a
# target/lib/libbrotlienc-static.a
# target/lib/libbrotlidec-static.a
# key: ${{ runner.os }}-brotli-${{ hashFiles('scripts/build-brotli.sh') }}-${{ hashFiles('.github/workflows/arbitrator-ci.yaml') }}-${{ matrix.test-mode }}
# restore-keys: ${{ runner.os }}-brotli-

# - name: Build cbrotli-local
# if: steps.cache-cbrotli.outputs.cache-hit != 'true'
# run: ./scripts/build-brotli.sh -l

# - name: Build cbrotli-wasm in docker
# if: steps.cache-cbrotli.outputs.cache-hit != 'true'
# run: ./scripts/build-brotli.sh -w -d

# - name: Build
# run: make build -j

# - name: test-go-deps
# run: make --debug test-go-deps

# - name: Build all lint dependencies
# run: make -j build-node-deps

# - name: Lint
# uses: golangci/golangci-lint-action@v2
# with:
# version: latest
# skip-go-installation: true
# skip-pkg-cache: true
# - name: Custom Lint
# run: |
# go run ./linter/koanf ./...
# go run ./linter/pointercheck ./...

# - name: Set environment variables
# run: |
# mkdir -p target/tmp/x
# echo "TMPDIR=$(pwd)/target/tmp/x" >> "$GITHUB_ENV"
# echo "GOMEMLIMIT=6GiB" >> "$GITHUB_ENV"
# echo "GOGC=80" >> "$GITHUB_ENV"

# - name: run tests without race detection
# if: matrix.test-mode == 'defaults'
# run: |
# packages=`go list ./...`
# gotestsum --format short-verbose --packages="$packages" --rerun-fails=1 -- -timeout 20m -coverprofile=coverage.txt -covermode=atomic -coverpkg=./...,./go-ethereum/...

# - name: run tests with race detection
# if: matrix.test-mode == 'race'
# run: |
# packages=`go list ./...`
# gotestsum --format short-verbose --packages="$packages" --rerun-fails=1 -- --timeout 20m -race

# - name: run redis tests
# if: matrix.test-mode == 'defaults'
# run: TEST_REDIS=redis://redis:6379/0 gotestsum --format short-verbose -- -timeout 20m -p 1 -run TestRedis ./arbnode/... ./system_tests/... -coverprofile=coverage-redis.txt -covermode=atomic -coverpkg=./...

# - name: run challenge tests
# if: matrix.test-mode == 'challenge'
# run: |
# packages=`go list ./...`
# gotestsum --format short-verbose --packages="$packages" --rerun-fails=1 -- -timeout 20m ./... -coverprofile=coverage.txt -covermode=atomic -coverpkg=./...,./go-ethereum/... -tags=challengetest -run=TestChallenge

# - name: Upload coverage to Codecov
# uses: codecov/codecov-action@v2
# if: matrix.test-mode == 'defaults'
# with:
# fail_ci_if_error: false
# files: ./coverage.txt,./coverage-redis.txt
# verbose: false
# token: ${{ secrets.CODECOV_TOKEN }}
name: Go tests CI
run-name: Go tests CI triggered from @${{ github.actor }} of ${{ github.head_ref }}

on:
workflow_dispatch:
merge_group:
pull_request:
push:
branches:
- master
- develop
- integration
tags:
# YYYYMMDD
- "20[0-9][0-9][0-1][0-9][0-3][0-9]*"

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
test:
name: Go Tests
container:
image: ghcr.io/catthehacker/ubuntu:js-22.04
runs-on: [self-hosted, X64]

# Creates a redis container for redis tests
services:
redis:
image: redis
ports:
- 6379:6379

strategy:
fail-fast: false
matrix:
test-mode: [defaults, race, challenge]

steps:
- name: Checkout
uses: actions/checkout@v3
with:
submodules: true

- name: Install dependencies
run: >
sudo apt update && sudo apt install -y wabt gotestsum
cmake build-essential bison golang clang make wabt

- name: Setup nodejs
uses: actions/setup-node@v3
with:
node-version: '16'
cache: 'yarn'
cache-dependency-path: '**/yarn.lock'

- name: Install go
uses: actions/setup-go@v4
with:
go-version: 1.20.x

- name: Install wasm-ld
run: |
sudo apt-get update && sudo apt-get install -y lld-14
sudo ln -s /usr/bin/wasm-ld-14 /usr/local/bin/wasm-ld

- name: Install rust wasm32-unknown-unknown
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: "stable"
target: wasm32-unknown-unknown

- name: Install rust wasm32-wasi
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: "stable"
target: wasm32-wasi

- name: Install rust stable
uses: actions-rs/toolchain@v1
id: install-rust
with:
profile: minimal
toolchain: "stable"
override: true

- name: Install cbindgen
run: cargo install cbindgen

- name: Cache Build Products
uses: actions/cache@v3
with:
path: |
~/go/pkg/mod
~/.cache/go-build
key: ${{ runner.os }}-go-${{ hashFiles('go.sum') }}-${{ matrix.test-mode }}
restore-keys: ${{ runner.os }}-go-

- name: Cache Rust Build Products
uses: actions/cache@v3
with:
path: |
~/.cargo/registry/
~/.cargo/git/
arbitrator/target/
arbitrator/wasm-libraries/target/
arbitrator/wasm-libraries/soft-float/SoftFloat/build
target/etc/initial-machine-cache/
key: ${{ runner.os }}-cargo-${{ steps.install-rust.outputs.rustc_hash }}-min-${{ hashFiles('arbitrator/Cargo.lock') }}-${{ matrix.test-mode }}
restore-keys: ${{ runner.os }}-cargo-${{ steps.install-rust.outputs.rustc_hash }}-

- name: Cache cbrotli
uses: actions/cache@v3
id: cache-cbrotli
with:
path: |
target/include/brotli/
target/lib-wasm/
target/lib/libbrotlicommon-static.a
target/lib/libbrotlienc-static.a
target/lib/libbrotlidec-static.a
key: ${{ runner.os }}-brotli-${{ hashFiles('scripts/build-brotli.sh') }}-${{ hashFiles('.github/workflows/arbitrator-ci.yaml') }}-${{ matrix.test-mode }}
restore-keys: ${{ runner.os }}-brotli-

- name: Build cbrotli-local
if: steps.cache-cbrotli.outputs.cache-hit != 'true'
run: ./scripts/build-brotli.sh -l

- name: Build cbrotli-wasm in docker
if: steps.cache-cbrotli.outputs.cache-hit != 'true'
run: ./scripts/build-brotli.sh -w -d

- name: Build
run: make build -j

- name: test-go-deps
run: make --debug test-go-deps

- name: Build all lint dependencies
run: make -j build-node-deps

- name: Lint
uses: golangci/golangci-lint-action@v2
with:
version: latest
skip-go-installation: true
skip-pkg-cache: true
- name: Custom Lint
run: |
go run ./linter/koanf ./...
go run ./linter/pointercheck ./...

- name: Set environment variables
run: |
mkdir -p target/tmp/x
echo "TMPDIR=$(pwd)/target/tmp/x" >> "$GITHUB_ENV"
echo "GOMEMLIMIT=6GiB" >> "$GITHUB_ENV"
echo "GOGC=80" >> "$GITHUB_ENV"

- name: run tests without race detection
if: matrix.test-mode == 'defaults'
run: |
packages=`go list ./...`
gotestsum --format short-verbose --packages="$packages" --rerun-fails=1 -- -timeout 20m -coverprofile=coverage.txt -covermode=atomic -coverpkg=./...,./go-ethereum/...

- name: run tests with race detection
if: matrix.test-mode == 'race'
run: |
packages=`go list ./...`
gotestsum --format short-verbose --packages="$packages" --rerun-fails=1 -- --timeout 20m -race

- name: run redis tests
if: matrix.test-mode == 'defaults'
run: TEST_REDIS=redis://redis:6379/0 gotestsum --format short-verbose -- -timeout 20m -p 1 -run TestRedis ./arbnode/... ./system_tests/... -coverprofile=coverage-redis.txt -covermode=atomic -coverpkg=./...

- name: run challenge tests
if: matrix.test-mode == 'challenge'
run: |
packages=`go list ./...`
gotestsum --format short-verbose --packages="$packages" --rerun-fails=1 -- -timeout 20m ./... -coverprofile=coverage.txt -covermode=atomic -coverpkg=./...,./go-ethereum/... -tags=challengetest -run=TestChallenge

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
if: matrix.test-mode == 'defaults'
with:
fail_ci_if_error: false
files: ./coverage.txt,./coverage-redis.txt
verbose: false
token: ${{ secrets.CODECOV_TOKEN }}