-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache final images instead of build layers #161
Closed
Closed
Changes from 21 commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
8b8c996
Try caching images instead of build layers
MarcoPolo 93d4f71
Pass in abort controller signal
MarcoPolo b2d6350
Derive input hash from args, build_context, and dockerfile (even if d…
MarcoPolo de1553d
Fix restargs to build command
MarcoPolo 8b7bbab
Cache differently
MarcoPolo 932b496
Silent check
MarcoPolo e0ecfef
Use bash in makefile
MarcoPolo a8fcce3
Fix build.ts shebang
MarcoPolo ee27a53
Use -S in env
MarcoPolo cc1e261
Fix Dockerfile for use outside of buildx
MarcoPolo a13f2ae
Use --fail in curl
MarcoPolo 2293a30
Fix typo
MarcoPolo bf78222
Echo something in shouldUseCache
MarcoPolo c4f1f39
Use buildx for rust builds
MarcoPolo e9f44f3
Load from buildx
MarcoPolo c0b6f1d
Fix version import
MarcoPolo 1603724
Start from playwright image
MarcoPolo 3caee19
Try parallel builds
MarcoPolo 2564c1b
Normalized name for compose
MarcoPolo e326eba
Move compose name sanitization to before serializing
MarcoPolo 97d9215
Double worker count
MarcoPolo 1cd6935
Use explicit PUSH_CACHE env var
MarcoPolo 5255cea
Fix name
MarcoPolo 4e9a600
Don't use git ls-files since composite actions aren't in a git repo
MarcoPolo e057f2c
Shebang is not consistent across platforms
MarcoPolo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,27 @@ | ||
| image_name := go-v0.22 | ||
| SHELL := /usr/bin/env bash | ||
| imageName := go-v0.22 | ||
|
|
||
| cacheKey=${shell git ls-files | tr "\n" " " | xargs ../../helpers/hashFiles.sh} | ||
| shouldUseCache := ${shell IMAGE_NAME=${imageName} CACHE_KEY=${cacheKey} ../../helpers/shouldUseCache.sh} | ||
|
|
||
| all: image.json | ||
|
|
||
| ifneq (${shouldUseCache},) | ||
| # We have a cached image, let's use it | ||
| image.json: | ||
| CACHE_KEY=${cacheKey} IMAGE_NAME=${imageName} ../../helpers/tryLoadCache.sh | ||
| docker image inspect ${imageName} -f "{{.Id}}" | \ | ||
| xargs -I {} echo "{\"imageID\": \"{}\"}" > $@ | ||
| else | ||
| image.json: Dockerfile main.go go.mod go.sum | ||
| IMAGE_NAME=${image_name} ../../dockerBuildWrapper.sh . | ||
| docker image inspect ${image_name} -f "{{.Id}}" | \ | ||
| docker build -t ${imageName} . | ||
| docker image inspect ${imageName} -f "{{.Id}}" | \ | ||
| xargs -I {} echo "{\"imageID\": \"{}\"}" > $@ | ||
| # If we're in CI, save the cache | ||
| if [[ -n "$${CI:-}" ]]; then IMAGE_NAME=${imageName} CACHE_KEY=${cacheKey} ../../helpers/saveCache.sh; fi | ||
| endif | ||
|
|
||
| .PHONY: clean | ||
|
|
||
| clean: | ||
| rm image.json | ||
| rm image.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,27 @@ | ||
| image_name := go-v0.23 | ||
| SHELL := /usr/bin/env bash | ||
| imageName := go-v0.23 | ||
|
|
||
| cacheKey=${shell git ls-files | tr "\n" " " | xargs ../../helpers/hashFiles.sh} | ||
| shouldUseCache := ${shell IMAGE_NAME=${imageName} CACHE_KEY=${cacheKey} ../../helpers/shouldUseCache.sh} | ||
|
|
||
| all: image.json | ||
|
|
||
| ifneq (${shouldUseCache},) | ||
| # We have a cached image, let's use it | ||
| image.json: | ||
| CACHE_KEY=${cacheKey} IMAGE_NAME=${imageName} ../../helpers/tryLoadCache.sh | ||
| docker image inspect ${imageName} -f "{{.Id}}" | \ | ||
| xargs -I {} echo "{\"imageID\": \"{}\"}" > $@ | ||
| else | ||
| image.json: Dockerfile main.go go.mod go.sum | ||
| IMAGE_NAME=${image_name} ../../dockerBuildWrapper.sh . | ||
| docker image inspect ${image_name} -f "{{.Id}}" | \ | ||
| docker build -t ${imageName} . | ||
| docker image inspect ${imageName} -f "{{.Id}}" | \ | ||
| xargs -I {} echo "{\"imageID\": \"{}\"}" > $@ | ||
| # If we're in CI, save the cache | ||
| if [[ -n "$${CI:-}" ]]; then IMAGE_NAME=${imageName} CACHE_KEY=${cacheKey} ../../helpers/saveCache.sh; fi | ||
| endif | ||
|
|
||
| .PHONY: clean | ||
|
|
||
| clean: | ||
| rm image.json | ||
| rm image.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,27 @@ | ||
| image_name := go-v0.24 | ||
| SHELL := /usr/bin/env bash | ||
| imageName := go-v0.24 | ||
|
|
||
| cacheKey=${shell git ls-files | tr "\n" " " | xargs ../../helpers/hashFiles.sh} | ||
| shouldUseCache := ${shell IMAGE_NAME=${imageName} CACHE_KEY=${cacheKey} ../../helpers/shouldUseCache.sh} | ||
|
|
||
| all: image.json | ||
|
|
||
| ifneq (${shouldUseCache},) | ||
| # We have a cached image, let's use it | ||
| image.json: | ||
| CACHE_KEY=${cacheKey} IMAGE_NAME=${imageName} ../../helpers/tryLoadCache.sh | ||
| docker image inspect ${imageName} -f "{{.Id}}" | \ | ||
| xargs -I {} echo "{\"imageID\": \"{}\"}" > $@ | ||
| else | ||
| image.json: Dockerfile main.go go.mod go.sum | ||
| IMAGE_NAME=${image_name} ../../dockerBuildWrapper.sh . | ||
| docker image inspect ${image_name} -f "{{.Id}}" | \ | ||
| docker build -t ${imageName} . | ||
| docker image inspect ${imageName} -f "{{.Id}}" | \ | ||
| xargs -I {} echo "{\"imageID\": \"{}\"}" > $@ | ||
| # If we're in CI, save the cache | ||
| if [[ -n "$${CI:-}" ]]; then IMAGE_NAME=${imageName} CACHE_KEY=${cacheKey} ../../helpers/saveCache.sh; fi | ||
| endif | ||
|
|
||
| .PHONY: clean | ||
|
|
||
| clean: | ||
| rm image.json | ||
| rm image.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/usr/bin/env /bin/bash | ||
| set -eou pipefail | ||
| set -x | ||
|
|
||
| # Assert that IMAGE_NAME is not empty | ||
| if [ -z "$IMAGE_NAME" ]; then | ||
| echo "IMAGE_NAME is not set" | ||
| exit 1 | ||
| fi | ||
|
|
||
| ARCH=$(docker info -f "{{.Architecture}}") | ||
| BUILD_CACHE_KEY="$IMAGE_NAME-$CACHE_KEY-$ARCH" | ||
| echo $BUILD_CACHE_KEY |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/usr/bin/env /bin/bash | ||
| set -eou pipefail | ||
|
|
||
| # sort all the files passed in by name, then hash them | ||
| echo "$@" | xargs -n1 | sort | xargs sha256sum | sha256sum | sed 's/.$//' | xargs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| #!/usr/bin/env /bin/bash | ||
| set -eou pipefail | ||
| set -x | ||
|
|
||
| script_dir=$(dirname "$0") | ||
| BUILD_CACHE_KEY=$($script_dir/buildCacheKey.sh) | ||
| AWS_BUCKET=${AWS_BUCKET:-libp2p-by-tf-aws-bootstrap} | ||
|
|
||
| docker image save $IMAGE_NAME | gzip | aws s3 cp - s3://$AWS_BUCKET/imageCache/$BUILD_CACHE_KEY.tar.gz |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| #!/usr/bin/env /bin/bash | ||
| set -eou pipefail | ||
| set -x | ||
|
|
||
| # shouldUseCache.sh - Returns exit code 0 if we don't have the image locally and we have a cache hit. | ||
|
|
||
| AWS_BUCKET=${AWS_BUCKET:-libp2p-by-tf-aws-bootstrap} | ||
|
|
||
| script_dir=$(dirname "$0") | ||
| BUILD_CACHE_KEY=$($script_dir/buildCacheKey.sh) | ||
|
|
||
| # If we already have this image name in docker, lets not use the cache, since a | ||
| # small change will probaby be faster than refetch the cache | ||
| if docker image inspect $IMAGE_NAME -f "{{.Id}}" &> /dev/null; then | ||
| exit 1; | ||
| fi | ||
|
|
||
| curl --fail --head https://s3.amazonaws.com/$AWS_BUCKET/imageCache/$BUILD_CACHE_KEY.tar.gz &> /dev/null | ||
| # We need to echo something so that Make sees this as a target | ||
| echo $BUILD_CACHE_KEY |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/usr/bin/env /bin/bash | ||
| set -eou pipefail | ||
| set -x | ||
|
|
||
| script_dir=$(dirname "$0") | ||
| BUILD_CACHE_KEY=$($script_dir/buildCacheKey.sh) | ||
| AWS_BUCKET=${AWS_BUCKET:-libp2p-by-tf-aws-bootstrap} | ||
|
|
||
|
|
||
| curl https://s3.amazonaws.com/$AWS_BUCKET/imageCache/$BUILD_CACHE_KEY.tar.gz \ | ||
| | docker image load && \ | ||
| exit 0 \ | ||
| || exit 1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| *-image.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,22 +1,6 @@ | ||
| # syntax=docker/dockerfile:1 | ||
|
|
||
| FROM mcr.microsoft.com/playwright | ||
| ARG BASE_IMAGE | ||
| FROM $BASE_IMAGE | ||
|
|
||
| WORKDIR /app | ||
|
|
||
|
|
||
| COPY package*.json . | ||
|
|
||
| RUN npm ci | ||
|
|
||
| # Install browsers | ||
| RUN ./node_modules/.bin/playwright install | ||
|
|
||
| COPY tsconfig.json . | ||
| COPY .aegir.js . | ||
| COPY test ./test | ||
| COPY src ./src | ||
|
|
||
| RUN npm run build | ||
|
|
||
| ENTRYPOINT [ "npm", "test", "--", "--build", "false", "--types", "false", "-t", "browser" ] | ||
| ENTRYPOINT [ "npm", "test", "--", "--build", "false", "--types", "false", "-t", "browser" ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,17 +1,21 @@ | ||
| # syntax=docker/dockerfile:1 | ||
| FROM node:18 | ||
| # Using playwright so that we have the same base across NodeJS + Browser tests | ||
| FROM mcr.microsoft.com/playwright | ||
|
|
||
| WORKDIR /app | ||
|
|
||
| COPY package*.json . | ||
| COPY package*.json ./ | ||
|
|
||
| RUN npm ci | ||
|
|
||
| # Install browsers, Needed for the browser tests, but we do it here so we have the same base | ||
| RUN ./node_modules/.bin/playwright install | ||
|
|
||
| COPY tsconfig.json . | ||
| COPY .aegir.js . | ||
| COPY test ./test | ||
| COPY src ./src | ||
|
|
||
| RUN npm run build | ||
|
|
||
| ENTRYPOINT [ "npm", "test", "--", "--build", "false", "--types", "false", "-t", "node" ] | ||
| ENTRYPOINT [ "npm", "test", "--", "--build", "false", "--types", "false", "-t", "node" ] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason to still use make here? We could as well just use typescript for this step.
Going even further, we could make it a "proper" action that is published to the GitHub marketplace. That would allow us to move away from a
compositeone to either Docker (and allow us to write it in go or Rust) or JavaScript.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not opposed to it. Make gives us a lot of stuff out of the box, like this
-jflag for paralleization. I might punt on that for now though.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To add another point, this is the root directory of these tests. Make here does nothing but call make in the implementation folders. I wouldn't want to remove this make. It's a good interface point such that implementations know they have to provide a Makefile that generates a image.json. The rest can be done however they want.