Use Docker BuildKit for the CI and docker_script #1316
Conversation
cloudbuild.yaml
Outdated
| @@ -20,6 +20,11 @@ steps: | |||
| '--tag=gcr.io/oak-ci/oak:latest', | |||
| '.', | |||
| ] | |||
| env: | |||
| [ | |||
| "DOCKER_BUILDKIT=1", | |||
There was a problem hiding this comment.
(nit) we seem to be using single-quoted string elsewhere, so perhaps worth sticking with that convention (it may even be enforced by some linters already, not sure)
cloudbuild.yaml
Outdated
| @@ -20,6 +20,11 @@ steps: | |||
| '--tag=gcr.io/oak-ci/oak:latest', | |||
| '.', | |||
| ] | |||
| env: | |||
| [ | |||
| "DOCKER_BUILDKIT=1", | |||
There was a problem hiding this comment.
Could you also add a short comment here to explain what this is for, and / or a reference to some online page.
Dockerfile
Outdated
| ARG debian_snapshot=buster-20200327 | ||
| FROM debian/snapshot:${debian_snapshot} | ||
| ARG debian_snapshot=buster-20200720-slim | ||
| FROM debian:${debian_snapshot} |
There was a problem hiding this comment.
note that this is not a snapshot image, in particular AFAICT apt-get packages are not guaranteed to be reproducibly installable. Is there no way to make it work with a (perhaps more recent) snapshot image?
There was a problem hiding this comment.
Hmm - By 'snapshot' in this case, I'm guessing that you meant that the base image has the apt sources that are preconfigured to point at the snapshot.debian.org locations? I'll try and tidy this up to be consistent - and find a later snapshot that has the apropriate image in Docker.io so that the cache can be populated.
Dockerfile
Outdated
| @@ -1,6 +1,6 @@ | |||
| # Use fixed snapshot of Debian to create a deterministic environment. | |||
| # Snapshot tags can be found at https://hub.docker.com/r/debian/snapshot/tags | |||
| ARG debian_snapshot=buster-20200327 | |||
| ARG debian_snapshot=buster-20200720 | |||
There was a problem hiding this comment.
looking at it again, it seems that the build step is rebuilding the entire image, presumably because the base image was changed here. Does it not work by keeping it to the old value?
There was a problem hiding this comment.
After some research: It does rebuild the whole image if the docker back end is changed (e.g. to BuildKit), but the pull of the original image fails, partially because it doesnt exist any more (so it seems), so I changed that parameter to the currently available image from Docker.io. It looks like the gcb infrastructure hasnt got that image available either, and therfore it fails. BuildKit builds do seem to work (I have it enabled by default on my x86 and the packet machine), but for some reason the CI here has a kink in it...
ebeasant-arm
force-pushed
the
main
branch
2 times, most recently
from
bf63855
to
46424c7
Compare
|
Looks to be blocked by a BuildKit problem: If the gcr default mirror can be pre-populated with the Oak snapshot, that would enable this to PR to proceede, or an updated Docker + Buildkit version for gcr (seems unlikely) Related bugfix: moby/buildkit#1397 |
Docker BuildKit has a number of advanages over the tradiditonal Docker back end - it's potentially quicker as it can make use of parallel jobs when multi-stage Dockerfiles are used, and it also enables the use of automatically set target/host ARGs, such as TARGETARCH. This is especially useful when working with multiple target architectures. This change modifies the main docker image name to indicate it is using BuildKit. This is a WIP testing change.
|
Replaced by #2552 |
Docker BuildKit has a number of advanages over the tradiditonal
Docker back end - it's potentially quicker as it can make use of
parallel jobs when multi-stage Dockerfiles are used, and it also
enables the use of automatically set target/host ARGs, such as
TARGETARCH. This is especially useful when working with multiple
target architectures.
(This PR is primarily created to check the functionality of integration with GCP)
Checklist
Cloudbuild
cover any TODOs and/or unfinished work.
construction.