Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: initial Earthly CI #5069

Merged
merged 202 commits into from
Mar 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
202 commits
Select commit Hold shift + click to select a range
a03cdc6
Add noir pull
ludamad Mar 7, 2024
6d8c7ce
feat(earthly): beginnings
ludamad Mar 8, 2024
812d86d
revert
ludamad Mar 8, 2024
07ab202
cool stuff
ludamad Mar 8, 2024
4ba38e6
Try to take it live
ludamad Mar 8, 2024
ef7d5ee
yaml syntax
ludamad Mar 8, 2024
543aed5
yaml syntax
ludamad Mar 8, 2024
f0987a7
yaml syntax
ludamad Mar 8, 2024
9680a38
yaml syntax
ludamad Mar 8, 2024
5ea5e46
check it out
ludamad Mar 8, 2024
0da6e28
test
ludamad Mar 8, 2024
f790c7f
allow passing complex commands to runscript
ludamad Mar 8, 2024
a0986ed
earthly on spot
ludamad Mar 8, 2024
7aef782
earthly on spot
ludamad Mar 8, 2024
0e5d167
earthly on spot
ludamad Mar 8, 2024
9f0ab9d
fix
ludamad Mar 8, 2024
7a72211
tests
ludamad Mar 8, 2024
29f4cd1
run
ludamad Mar 8, 2024
5426698
oldest doh in the book
ludamad Mar 8, 2024
6a4d4a6
earthly fix
ludamad Mar 8, 2024
cbd9254
earthly fix
ludamad Mar 8, 2024
b1ac8b8
hijack charlies buildhost
ludamad Mar 8, 2024
8eeb7d6
command fix
ludamad Mar 8, 2024
6d0e60f
yeah thats it
ludamad Mar 8, 2024
50ac702
independence
ludamad Mar 8, 2024
efe5848
ssh test
ludamad Mar 8, 2024
26ce920
attempt working
ludamad Mar 8, 2024
ea00fe7
test workings [ci debug]
ludamad Mar 8, 2024
dfa4759
update
ludamad Mar 8, 2024
8eed54d
fix url syntax
ludamad Mar 8, 2024
1d5fbc9
sudo etc hosts
ludamad Mar 8, 2024
8c4d162
ok one last try
ludamad Mar 8, 2024
b7bed2b
cache
ludamad Mar 8, 2024
67f82d4
buildkit setting in wrapper
ludamad Mar 8, 2024
7741413
can still be docker?
ludamad Mar 8, 2024
85d8d8a
ci bench
ludamad Mar 8, 2024
126bb69
fix
ludamad Mar 8, 2024
9b66094
fixes
ludamad Mar 8, 2024
215138e
build
ludamad Mar 8, 2024
81ec27c
script
ludamad Mar 8, 2024
210918e
srs_db
ludamad Mar 8, 2024
3892ae6
fix
ludamad Mar 8, 2024
038677d
cache
ludamad Mar 8, 2024
f74b7a6
minimal srs db download
ludamad Mar 8, 2024
a1a6e15
try minimal circleci
ludamad Mar 8, 2024
9e92ce6
Woops
ludamad Mar 8, 2024
0145093
Caching fixes
ludamad Mar 8, 2024
6c89430
incremental build fixup
ludamad Mar 8, 2024
06517a7
build fix
ludamad Mar 8, 2024
1d7e6d9
New utilities
ludamad Mar 8, 2024
7f287d7
caching
ludamad Mar 8, 2024
8a3bb36
builds
ludamad Mar 8, 2024
b76eaa4
fix wasm build
ludamad Mar 8, 2024
5e05070
refactoring passes
ludamad Mar 8, 2024
904f24d
ts earthfile
ludamad Mar 8, 2024
5c819ba
source remote_initializse
ludamad Mar 8, 2024
354b172
indecision
ludamad Mar 8, 2024
4bacfe5
noir earthfilication
ludamad Mar 8, 2024
7e7c322
Noir down
ludamad Mar 9, 2024
2bca080
avm transpiler working
ludamad Mar 9, 2024
f047b71
todos
ludamad Mar 9, 2024
a462ebd
versioning
ludamad Mar 9, 2024
71aafdd
versioning
ludamad Mar 9, 2024
f43486b
flesh out yarn project
ludamad Mar 9, 2024
db36e87
benches
ludamad Mar 9, 2024
441ca22
fix yarn-project
ludamad Mar 9, 2024
65f4ce5
fix yarn-project
ludamad Mar 9, 2024
a1b5d9f
Update
ludamad Mar 9, 2024
ac5dc92
earthly build fix
ludamad Mar 9, 2024
9fb615b
fixes
ludamad Mar 9, 2024
54f9d6b
benching sequential order
ludamad Mar 9, 2024
5be25cf
wasm benches
ludamad Mar 9, 2024
5d3b708
dont use spot
ludamad Mar 9, 2024
3559c71
try inline cache
ludamad Mar 9, 2024
98d98a2
Files
ludamad Mar 11, 2024
27e99f2
more earthly
ludamad Mar 11, 2024
129f269
working build-ci
ludamad Mar 11, 2024
65e82f5
more earthly ci
ludamad Mar 11, 2024
f66acac
more earthly ci
ludamad Mar 11, 2024
324b852
test earthly
ludamad Mar 11, 2024
a77e97c
try just DOCKER_HOST
ludamad Mar 11, 2024
6b57f7a
fix docker host
ludamad Mar 11, 2024
a7f8a7b
fix docker host
ludamad Mar 11, 2024
07364c9
Fix runner
ludamad Mar 11, 2024
db5b221
checkout
ludamad Mar 11, 2024
9474cf9
Attempt to make wasm build. wip
charlielye Mar 11, 2024
e0a6b70
Registry stuff
ludamad Mar 11, 2024
cd73bdc
de-templatize
ludamad Mar 12, 2024
d5e6374
earthly helper script
ludamad Mar 12, 2024
19b6ad6
e2e sandbox example
ludamad Mar 12, 2024
7a0f755
add e2e earthfile
ludamad Mar 12, 2024
0448e45
end-to-end tests
ludamad Mar 12, 2024
9f25bc7
end-to-end tests
ludamad Mar 12, 2024
2ce4d38
e2e
ludamad Mar 12, 2024
b7af582
work on end to end and caching
ludamad Mar 13, 2024
bba6708
Maybe ditch this. stash.
ludamad Mar 13, 2024
a947c84
Attempts
ludamad Mar 13, 2024
0774fdb
progress on end-to-end, next: fixing @aztec/accounts
ludamad Mar 13, 2024
fd9fa87
earthly
ludamad Mar 13, 2024
6bb6aae
no circleci jobs
ludamad Mar 13, 2024
d21e781
add ci
ludamad Mar 13, 2024
6d570e5
Merge remote-tracking branch 'origin/master' into feat/earthly
ludamad Mar 13, 2024
e26c2bd
fix conditions
ludamad Mar 13, 2024
099b37c
try set term
ludamad Mar 13, 2024
5f039a0
Fix failure mode
ludamad Mar 13, 2024
956e46c
fix noir build
ludamad Mar 13, 2024
7bd1a08
split out earthly-timed
ludamad Mar 13, 2024
03fbb46
split out earthly-timed
ludamad Mar 13, 2024
933ceaa
earthly-timed fix
ludamad Mar 13, 2024
a2ac99b
fix
ludamad Mar 13, 2024
ea4daac
proper filtering in noir-projects
ludamad Mar 13, 2024
ca36d43
dont cache earthly
ludamad Mar 13, 2024
7be588b
Update
ludamad Mar 13, 2024
5977737
fix concurrency
ludamad Mar 13, 2024
b1af007
submodule recursive
ludamad Mar 13, 2024
70658eb
fix timed
ludamad Mar 13, 2024
d3bb94e
fix artifacts being baked in
ludamad Mar 13, 2024
e97f960
try better ignore scheme, ln to .gitignore
ludamad Mar 13, 2024
8f197ab
try better ignore scheme, ln to .gitignore
ludamad Mar 13, 2024
ab923de
git ignore removal
ludamad Mar 13, 2024
e48ccd3
fix multiplatform
ludamad Mar 13, 2024
b505019
feat: Cl/earthly host build (#5195)
charlielye Mar 13, 2024
b23e948
download-again
ludamad Mar 13, 2024
930c3ff
Merge remote-tracking branch 'origin/master' into feat/earthly
ludamad Mar 13, 2024
92281ae
fix path
ludamad Mar 13, 2024
320c525
attempt to run just 1 e2e test.
charlielye Mar 13, 2024
c153500
cached foundry
ludamad Mar 13, 2024
f6cf37b
less to build in yarn project
ludamad Mar 13, 2024
612c81c
Fix granularity
ludamad Mar 13, 2024
13a45c2
Try again at pattern
ludamad Mar 14, 2024
b6349e6
fix end to end
ludamad Mar 14, 2024
c18a22a
more intricate caching
ludamad Mar 14, 2024
41d186a
more intricate caching
ludamad Mar 14, 2024
1b09df1
Almost working
ludamad Mar 14, 2024
7ab2ce1
Test pipeline
ludamad Mar 14, 2024
c9ce994
fix build
ludamad Mar 14, 2024
0998438
try simple
ludamad Mar 14, 2024
bca1239
flow
ludamad Mar 14, 2024
cd1d122
test
ludamad Mar 14, 2024
12622ac
alt option
ludamad Mar 14, 2024
36a5b23
Update
ludamad Mar 14, 2024
5631270
fix ci
ludamad Mar 14, 2024
9584efa
env
ludamad Mar 14, 2024
d561b81
fix shell type
ludamad Mar 14, 2024
a938128
drop -ci
ludamad Mar 14, 2024
110ace1
foundry
ludamad Mar 14, 2024
dcc430a
groov
ludamad Mar 14, 2024
6e4caa3
revert
ludamad Mar 14, 2024
247eaf8
Login to dockerhub in CI
ludamad Mar 14, 2024
7e00998
fix dockerhub login
ludamad Mar 14, 2024
77b5059
fix intended ci target
ludamad Mar 14, 2024
ec644bc
fix intended ci target
ludamad Mar 14, 2024
5bb3e00
mem stats
ludamad Mar 14, 2024
58e93a1
Merge remote-tracking branch 'origin/master' into feat/earthly
ludamad Mar 14, 2024
b5da55f
reinstate ci
ludamad Mar 14, 2024
8f22d1d
conditional fix
ludamad Mar 14, 2024
99ae65b
Fix image
ludamad Mar 14, 2024
bcb300d
try fix
ludamad Mar 14, 2024
23150b1
jiggle
ludamad Mar 14, 2024
ff95655
fix versions
ludamad Mar 14, 2024
71f8a9b
fix versions
ludamad Mar 14, 2024
33375cb
Merge remote-tracking branch 'origin/master' into ad/try-fix-acir
ludamad Mar 14, 2024
c943564
Merge remote-tracking branch 'origin/ad/try-fix-acir' into feat/earthly
ludamad Mar 14, 2024
564c1c4
Update
ludamad Mar 14, 2024
6ccf4f0
compiler dependency
ludamad Mar 14, 2024
785a774
compiler dependency
ludamad Mar 14, 2024
0f30090
Merge branch 'master' into feat/earthly
ludamad Mar 14, 2024
7e53c48
fix ignition calls
ludamad Mar 15, 2024
6bade04
Merge remote-tracking branch 'origin/feat/earthly' into feat/earthly
ludamad Mar 15, 2024
17504d5
Merge remote-tracking branch 'origin/master' into feat/earthly
ludamad Mar 15, 2024
98ab25f
matrix
ludamad Mar 15, 2024
bedf8ec
exec stats not working anyway
ludamad Mar 15, 2024
2c0b824
try and prevent settings issues
ludamad Mar 15, 2024
ec6e3de
firing off ci
ludamad Mar 15, 2024
da8815c
is bootstrapping necessary?
ludamad Mar 15, 2024
97c2ea3
concurrenct ci run 1
ludamad Mar 15, 2024
09286de
concurrenct ci run 2
ludamad Mar 15, 2024
5f65543
concurrenct ci run 3
ludamad Mar 15, 2024
fc09036
concurrenct ci run 4
ludamad Mar 15, 2024
82cf2da
concurrenct ci run 5
ludamad Mar 15, 2024
52824d0
concurrenct ci run 6
ludamad Mar 15, 2024
e926af3
concurrenct ci run 7
ludamad Mar 15, 2024
280de10
concurrenct ci run 8
ludamad Mar 15, 2024
aa0c805
concurrenct ci run 9
ludamad Mar 15, 2024
4c4cd8c
concurrenct ci run 10
ludamad Mar 15, 2024
0dd5f7b
concurrenct ci run 11
ludamad Mar 15, 2024
58c6327
concurrenct ci run 12
ludamad Mar 15, 2024
f8448b0
concurrenct ci run 13
ludamad Mar 15, 2024
7da0a93
concurrenct ci run 14
ludamad Mar 15, 2024
3d9030c
concurrenct ci run 15
ludamad Mar 15, 2024
be1b16c
concurrenct ci run 16
ludamad Mar 15, 2024
0320040
fix concurrency of gate action
ludamad Mar 15, 2024
71afc70
updates
ludamad Mar 15, 2024
d42cbcf
fix exit code
ludamad Mar 15, 2024
58fc121
clean
ludamad Mar 15, 2024
c571769
shorter
ludamad Mar 15, 2024
4de72a2
fix arm runner
ludamad Mar 15, 2024
c90a91b
Maybe not
ludamad Mar 15, 2024
a52e956
push down grep v
ludamad Mar 15, 2024
49198b0
status in foundry
ludamad Mar 15, 2024
e45563d
fix spam
ludamad Mar 15, 2024
f39c2f8
try fix
ludamad Mar 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1313,7 +1313,10 @@ workflows:
- barretenberg-stdlib-tests: *bb_test
- barretenberg-stdlib-recursion-ultra-tests: *bb_test
- barretenberg-acir-tests-bb: *bb_acir_tests
- barretenberg-acir-tests-bb-sol: *bb_acir_tests
- barretenberg-acir-tests-bb-sol:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing dep

requires:
- barretenberg-x86_64-linux-clang-sol
<<: *bb_acir_tests
- barretenberg-docs: *defaults
- bb-js:
requires:
Expand Down
46 changes: 46 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Run CI with Earthly
on:
push:
branches:
- master
pull_request: {}
workflow_dispatch: {}

jobs:
ci:
runs-on: ubuntu-latest
# run ci for both x86_64 and arm64
strategy: {matrix: {environment: [x86, arm]}}
# cancel if reran on same PR if exists, otherwise if on same commit
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}-${{ matrix.environment }}
cancel-in-progress: true
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
submodules: recursive

- name: Setup
run: |
mkdir -p ~/.ssh
echo DOCKER_HOST=ssh://build-instance-${{ matrix.environment }}.aztecprotocol.com >> $GITHUB_ENV
echo ${{ secrets.DOCKERHUB_PASSWORD}} | docker login -u aztecprotocolci --password-stdin
echo ${{ secrets.BUILD_INSTANCE_SSH_KEY }} | base64 -d > ~/.ssh/build_instance_key
chmod 600 ~/.ssh/build_instance_key
cat > ~/.ssh/config <<EOF
IdentityFile ~/.ssh/build_instance_key
StrictHostKeyChecking no
User ubuntu
EOF

# Turn on if updating our custom built WASM-enabled clang (wasi-sdk), foundry or other base images
#- name: Ensure Base Images
# run: |
# scripts/earthly --push ./foundry/+build
# Uncomment the following line if needed for the arm environment
# scripts/earthly --push ./barretenberg/cpp/+build-wasi-sdk

- name: CI
run: scripts/earthly +build-ci-small
4 changes: 4 additions & 0 deletions .github/workflows/protocol-circuits-gate-diff.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ on:
pull_request:

jobs:
# cancel if reran on same PR if exists, otherwise if on same commit
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
compare_protocol_circuits_gates:
if: "!startsWith(github.head_ref, 'release-please--')"
runs-on: ubuntu-20.04
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
dest
node_modules
.cache
scripts/.earthly
.pnp.cjs
.pnp.loader.mjs
build/
Expand All @@ -16,4 +17,4 @@ cmake-build-debug
.graphite*
.DS_Store

**/*.dockerignore
**/*.dockerignore
31 changes: 31 additions & 0 deletions Earthfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
VERSION 0.8
FROM ubuntu:lunar

build-ci:
BUILD ./avm-transpiler/+build
BUILD ./barretenberg/cpp/+build-release
BUILD ./barretenberg/cpp/+preset-wasm
BUILD ./barretenberg/cpp/+build-gcc
BUILD ./barretenberg/cpp/+build-fuzzing
BUILD ./barretenberg/cpp/+build-clang-assert
BUILD ./barretenberg/cpp/+test-clang-format
BUILD ./barretenberg/cpp/+test-clang-format
BUILD ./boxes/+build
BUILD ./noir/+packages
BUILD ./noir/+nargo
BUILD ./noir-projects/+build
BUILD ./yarn-project/+build
BUILD +test-end-to-end

build-ci-small:
BUILD ./yarn-project/end-to-end/+e2e-escrow-contract

build:
# yarn-project has the entry point to Aztec
BUILD ./yarn-project/+build

test-end-to-end:
BUILD ./yarn-project/end-to-end/+test-all

bench:
RUN echo hi
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,9 @@ It is faster to debug CI failures within a persistent ssh session compared to pu

```bash
cd project
./build-system/scripts/setup_env "$(git rev-parse HEAD)" "" "" ""
./build-system/scripts/setup_env "$(git rev-parse HEAD)" "" https://github.com/AztecProtocol/aztec-packages
source /tmp/.bash_env*
set +euo
{start testing your CI commands here}
```

Expand Down Expand Up @@ -76,3 +77,18 @@ Recovering if the sync is not happening with basic pull commands:
this needs to exist in the branch we push to, and have the same content as our base. This is similar to submodules, except instead of pointing to the final state of the module, it points to the last commit we have sync'd from, for purposes of commit replay. This can be fixed to match the commit in master after merges.
- manually editing the parent variable in noir/noir-repo/.gitrepo: this is the parent of the last sync commit on aztec side. If you get errors with a commit not being found in the upstream repo, and the commit mentioned is not the commit variable above, it might indicate this is somehow incorrect. This can happen when commit content is ported without its history, e.g. squashes
- use pull --force ONLY where you would use git reset. That is, if you really want to match some upstream noir for a purpose its fine, but you'll lose local changes (if any)


## Earthly

Earthly is a reproducible build tool that aims to combine the functionality of Docker, Makefiles and BASH.
Non-build earthly targets should start with 'test', 'run', or 'bench' as a general rule (but not hard rule) while builds can be nouns or start with build-.
If something is a bundle of targets for CI, we can do e.g. build-ci, test-ci etc.
See barretenberg/cpp/Earthfile for an example of a fairly involved Earthfile that can be used for inspiration.
Earthly docs https://docs.earthly.dev/ are extensive and show the various build patterns.

In a nutshell:
- Docker-like syntax defines all builds. We lean on docker heavily for when to rebuild and cache, and how to run in a reproducible manner.
- It supports modularization of the build manifest into multiple directories that can be imported. Simple functions and conditional logic can be used.
- We provide two modes, one for CI by passing --ci and one for local with incremental builds.
- We do NOT provide a native execution story for anything but Linux and WASM currently.
23 changes: 23 additions & 0 deletions avm-transpiler/Earthfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
VERSION 0.8
IMPORT ../noir AS noir
# we rely on noir source, which this image has
FROM noir+nargo

# move noir contents to /build/noir
RUN mv /build /noir && mkdir /build && mv /noir /build
# work in avm-transpiler
WORKDIR /build/avm-transpiler

RUN apt-get update && apt-get install -y git

COPY --keep-ts --dir scripts src Cargo.lock Cargo.toml rust-toolchain.toml .

build:
RUN ./scripts/bootstrap_native.sh
SAVE ARTIFACT target/release/avm-transpiler avm-transpiler

run:
#TODO needed?
FROM ubuntu:focal
COPY +build/avm-transpiler /build/avm-transpiler
ENTRYPOINT ["sh", "-c"]
2 changes: 0 additions & 2 deletions barretenberg/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -257,8 +257,6 @@ CI will automatically run integration tests against Aztec. It is located in the

### Integration tests with Aztec in Barretenberg Standalone Repo

CI will automatically run integration tests against Aztec's circuits which live [here](https://github.com/AztecProtocol/aztec-packages/tree/master/circuits). To change which Aztec branch or commit for CI to test against, modify [`.aztec-packages-commit`](./cpp/.aztec-packages-commit).

When working on a PR, you may want to point this file to a different Aztec branch or commit, but then it should probably be pointed back to master before merging.

### Testing locally in docker
Expand Down
1 change: 0 additions & 1 deletion barretenberg/cpp/.aztec-packages-commit

This file was deleted.

11 changes: 5 additions & 6 deletions barretenberg/cpp/CMakePresets.json
Original file line number Diff line number Diff line change
Expand Up @@ -206,11 +206,11 @@
}
},
{
"name": "op-count-track",
"name": "op-count",
"displayName": "Release build with operation counts",
"description": "Build with op counting",
"inherits": "clang16",
"binaryDir": "build-op-count-track",
"binaryDir": "build-op-count",
"environment": {
"CXXFLAGS": "-DBB_USE_OP_COUNT -DBB_USE_OP_COUNT_TRACK_ONLY"
}
Expand Down Expand Up @@ -256,7 +256,7 @@
"generator": "Ninja",
"toolchainFile": "cmake/toolchains/wasm32-wasi.cmake",
"environment": {
"WASI_SDK_PREFIX": "${sourceDir}/src/wasi-sdk-20.0",
"WASI_SDK_PREFIX": "${sourceDir}/src/wasi-sdk",
"CC": "$env{WASI_SDK_PREFIX}/bin/clang",
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bundled: small changes to presets

"CXX": "$env{WASI_SDK_PREFIX}/bin/clang++",
"AR": "$env{WASI_SDK_PREFIX}/bin/llvm-ar",
Expand Down Expand Up @@ -289,7 +289,6 @@
"inherits": "wasm",
"binaryDir": "build-wasm-threads",
"environment": {
"WASI_SDK_PREFIX": "${sourceDir}/src/wasi-sdk-20.0+threads",
"CMAKE_BUILD_TYPE": "Release"
},
"cacheVariables": {
Expand Down Expand Up @@ -355,9 +354,9 @@
"configurePreset": "op-count-time"
},
{
"name": "op-count-track",
"name": "op-count",
"inherits": "default",
"configurePreset": "op-count-track"
"configurePreset": "op-count"
},
{
"name": "clang16-dbg",
Expand Down
Loading