Implement build cache based on history array #26839

tonistiigi · 2016-09-22T22:11:55Z

Adds capability to specify images used as a cache source on build. These images do not need to have local parent chain and can be pulled from other registries. User needs to make sure to only use trusted images as sources.

Usage:

docker pull myimage:v1.0
docker build --cache-from myimage:v1.0 -t myimage:v1.1 .

graingert · 2016-09-22T22:27:46Z

Is this not going to be plugin based like #24711 (comment) ?

tonistiigi · 2016-09-22T22:32:04Z

@graingert If you look at the contents of #24711 it is not about the plugin method. This was changed if favor of #26065 proposal a long time.

tonistiigi · 2016-09-22T23:53:20Z

cc @stevvooe @aaronlehmann @dmcgowan @KJTsanaktsidis

icecrime · 2016-09-23T00:08:38Z

I think design was already approved, so I'm tentatively moving to code review.

Thanks for the carry @tonistiigi!

stevvooe · 2016-09-22T23:58:10Z

api/server/router/build/build_routes.go

+	var cacheFrom = []string{}
+	cacheFromJSON := r.FormValue("cachefrom")
+	if cacheFromJSON != "" {
+		if err := json.NewDecoder(strings.NewReader(cacheFromJSON)).Decode(&cacheFrom); err != nil {


json.Unmarshal(cacheFromJSON, &cacheFrom)

stevvooe · 2016-09-23T00:40:54Z

LGTM after the small nit.

aaronlehmann · 2016-09-23T00:41:40Z

daemon/cache.go

+		return false
+	}
+	if parent == nil || len(parent.History) == 0 && len(parent.RootFS.DiffIDs) == 0 {
+		return true


nil is considered a valid parent?

nil means FROM scratch so we ignore the parent and only check that the configuration matches

aaronlehmann · 2016-09-23T00:43:00Z

daemon/cache.go

+		OSVersion:  target.OSVersion,
+	})
+	if err != nil {
+		return "", errors.Wrapf(err, "failed to marshal image config")


errors.Wrap

aaronlehmann · 2016-09-23T00:43:09Z

daemon/cache.go

+
+	imgID, err := ic.daemon.imageStore.Create(config)
+	if err != nil {
+		return "", errors.Wrapf(err, "failed to create cache image")


errors.Wrap

aaronlehmann · 2016-09-23T01:06:42Z

builder/dockerfile/internals.go

@@ -447,11 +447,11 @@ func (b *Builder) processImageFrom(img builder.Image) error {
 // If no image is found, it returns `(false, nil)`.
 // If there is any error, it returns `(false, err)`.
 func (b *Builder) probeCache() (bool, error) {


Comment should be updated. Right now it says "probeCache checks if b.docker implements builder.ImageCache and image-caching is enabled"

aaronlehmann · 2016-09-23T01:27:47Z

LGTM after nits

Based on work by KJ Tsanaktsidis Signed-off-by: Tonis Tiigi <[email protected]> Signed-off-by: KJ Tsanaktsidis <[email protected]>

Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi · 2016-09-23T19:44:30Z

@stevvooe @aaronlehmann updated

stevvooe · 2016-09-23T21:53:51Z

LGTM

aaronlehmann

LGTM

This adds the cache-from build option (moby/moby#26839) and fixes #1382.

kachkaev · 2017-02-03T09:52:41Z

Hi guys,

Is docker pull myimage:v1.0 strictly necessary? Won't just docker build --cache-from myimage:v1.0 -t myimage:v1.1 . do pull as well? https://docs.docker.com/engine/reference/builder/ or CLI help do not explain this unfortunately.

I also struggled to find an explanation of what happens if there are multiple --cache-from. I've got a GitLab CI script that builds an image and I'd like it to reuse previous builds to save time. The idea is to grab a previously built image with the current branch tag, then if this fails pull master and if both pulls fail, just build from scratch.

Should the following work smoothly for all possible states of the "cache"? Or is anything here missing or redundant here?

docker pull $CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME || docker pull $CI_REGISTRY_IMAGE:master || true
docker build --tag=$CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME --cache-from=$CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME --cache-from=$CI_REGISTRY_IMAGE:master .
docker push $CI_REGISTRY_IMAGE:$CI_BUILD_REF_NAME

For my-branch this resolves to:

docker pull registry.example.com/group/repo:my-branch || docker pull registry.example.com/group/repo:master || true
docker build --tag=registry.example.com/group/repo:my-branch --cache-from=registry.example.com/group/repo:my-branch --cache-from=registry.example.com/group/repo:master .
docker push registry.example.com/group/repo:my-branch

And for master the script looks like this:

docker pull registry.example.com/group/repo:master || docker pull registry.example.com/group/repo:master || true
docker build --tag=registry.example.com/group/repo:master --cache-from=registry.example.com/group/repo:my-branch --cache-from=registry.example.com/group/repo:master .
docker push registry.example.com/group/repo:master

gajus · 2017-02-03T10:55:57Z

As far as my testing goes, docker build --cache-from is sufficient.

However, I have written a blog post about using --cache-from in my GitLab EE CI setup (https://medium.com/@gajus/making-docker-in-docker-builds-x2-faster-using-docker-cache-from-option-c01febd8ef84#.z3rl5bahm) and someone commented that docker pull is missing.

I am not sure whats the relevance of docker pull when using --cache-from should automatically attempt to pull the image. Is that not the case?

kachkaev · 2017-02-03T11:06:22Z

Yeah, that's what is unclear to me too. If --cache-from does pull an image, what will happen if it does not exist or if there is a problem with logging in? If I use multiple --cache-from and the first image exists, will docker build attempt to pull the second and the third image and reuse as much layers as possible? If at least one of the images fails to pull, with the whole build command fail? Or will it warn? Or will it just silently build from scratch?

I know that these questions can be answered after a few tests, but it'd be great if they were documented so that people could write robust CI scripts without too much iteration.

graingert · 2017-02-03T11:08:11Z

I'd like to know so I can add --cache-from to shipwright.

tonistiigi · 2017-02-03T22:46:31Z

--cache-from does not pull if an image is not found. There have been discussions about it but the way we would like to enable it is to not pull full images but only pull data that is needed to determine the cache hit and from there decide to actually pull layers.

When using multiple --cache-from they are checked for a cache hit in the order that user specified. If one of the images produces a cache hit for a command only that image is used for the rest of the build.

With docker 1.13 a new build flag was added: --cache-from This flag deprecated the buildcache tool that we used with docker 1.12 Details: moby/moby#26839

Implement build cache based on history array

With docker 1.13 a new build flag was added: --cache-from This flag deprecated the buildcache tool that we used with docker 1.12 Details: moby/moby#26839

jkp · 2019-01-28T15:16:14Z

--cache-from does not pull if an image is not found. There have been discussions about it but the way we would like to enable it is to not pull full images but only pull data that is needed to determine the cache hit and from there decide to actually pull layers.

When using multiple --cache-from they are checked for a cache hit in the order that user specified. If one of the images produces a cache hit for a command only that image is used for the rest of the build.

I just ran into this issue and only by digging deep through history to find this PR did I find the answer to explain the behaviour of the flag - it is not documented this way. Suggestion: could the documentation be updated to include this detail?

garrett-hopper · 2020-01-22T19:09:09Z

--cache-from does not pull if an image is not found. There have been discussions about it but the way we would like to enable it is to not pull full images but only pull data that is needed to determine the cache hit and from there decide to actually pull layers.

When using multiple --cache-from they are checked for a cache hit in the order that user specified. If one of the images produces a cache hit for a command only that image is used for the rest of the build.

Was this behavior ever implemented? It looks like --cache-from still doesn't work at all unless the full image is pulled first. (I'd like to avoid having to pull the entire image and only have layers which are useful be pulled.)

tonistiigi · 2020-01-22T19:30:41Z

@Jumblemuddle Yes, if you use buildkit #34715 (comment)

Documented usage of cache-from allows for users to specify multiple images to search through by passing the cache-from argument more than once. Current implementation of cache-from as a string does not enable this behavior. See: moby/moby#26839 (comment) Signed-off-by: Ryan Drew <[email protected]>

It is unclear how docker handles multiple caches under the hood. I did find an older comment¹ that if multiple `--cache-from` sources are provided, it will use the first cache hit for the whole run. This is my attempt to prioritize the ordering of the sources to use the tagged images from the ghcr.io registry over the latest images from docker.io. ¹ moby/moby#26839 (comment)

GordonTheTurtle added the status/0-triage label Sep 22, 2016

This was referenced Sep 22, 2016

Proposal: using registry images for build caching #24711

Closed

Proposal: control cache sources during build with --cache-from #26065

Closed

thaJeztah added status/1-design-review and removed status/0-triage labels Sep 22, 2016

tonistiigi force-pushed the build-cache branch 2 times, most recently from ba1feda to d3222ea Compare September 22, 2016 23:48

tonistiigi changed the title ~~wip: Implement build cache based on history array~~ Implement build cache based on history array Sep 22, 2016

icecrime added status/2-code-review and removed status/1-design-review labels Sep 23, 2016

stevvooe requested changes Sep 23, 2016

View reviewed changes

tonistiigi force-pushed the build-cache branch from d3222ea to 0a43b1d Compare September 23, 2016 00:35

aaronlehmann reviewed Sep 23, 2016

View reviewed changes

tonistiigi force-pushed the build-cache branch from 0a43b1d to fe741fc Compare September 23, 2016 00:48

aaronlehmann reviewed Sep 23, 2016

View reviewed changes

tonistiigi added 2 commits September 23, 2016 11:30

Implement build cache based on history array

690882c

Based on work by KJ Tsanaktsidis Signed-off-by: Tonis Tiigi <[email protected]> Signed-off-by: KJ Tsanaktsidis <[email protected]>

Add integration test for build —cache-from

4b8e680

Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi force-pushed the build-cache branch from fe741fc to 4b8e680 Compare September 23, 2016 18:30

stevvooe approved these changes Sep 23, 2016

View reviewed changes

aaronlehmann approved these changes Sep 23, 2016

View reviewed changes

thomaschaaf mentioned this pull request Jan 9, 2017

Add --cache-from docker/compose#4309

Closed

tuxity mentioned this pull request Jan 19, 2017

[WIP] Docker 1.13 drone-plugins/drone-docker#100

Merged

5 tasks

shin- added a commit to docker/docker-py that referenced this pull request Jan 26, 2017

Merge pull request #1417 from graingert/cachefrom

432730a

This adds the cache-from build option (moby/moby#26839) and fixes #1382.

databus23 mentioned this pull request Feb 13, 2017

Implement cache using docker's --cache-from flag concourse/docker-image-resource#99

Merged

thaJeztah mentioned this pull request Mar 7, 2017

bind /var/lib/docker:/var/lib/docker on dind:17.03-dind cause dockerd starting failed #31600

Closed

dnephin pushed a commit to dnephin/docker that referenced this pull request Apr 17, 2017

Merge pull request moby#26839 from tonistiigi/build-cache

dc14c5a

Implement build cache based on history array

dnephin pushed a commit to dnephin/docker that referenced this pull request Apr 17, 2017

Merge pull request moby#26839 from tonistiigi/build-cache

18f54cb

Implement build cache based on history array

dwrensha referenced this pull request in rust-lang/rust-playground May 18, 2017

Combine the compiler Dockerfiles and use the Docker registry

0083434

ghost mentioned this pull request Jun 21, 2017

Allow --cache-from on push command for faster builds. psykube/psykube#43

Closed

shouze mentioned this pull request Jul 15, 2017

Docker build cache does not ignore uid/gid on ADD or COPY #32816

Closed

friism mentioned this pull request Sep 20, 2017

Add multi-stage build support docker-library/official-images#3383

Closed

sjezewski mentioned this pull request Jan 22, 2018

Shrink build images and cache on CI pachyderm/pachyderm#2632

Closed

cosmicexplorer mentioned this pull request Jan 13, 2019

consider caching travis_ci/Dockerfile builds in CI pantsbuild/pants#7072

Closed

danking mentioned this pull request Mar 19, 2019

[docker, scorecard] add docker base image, use in scorecard hail-is/hail#5623

Merged

gustavovalverde mentioned this pull request Sep 27, 2022

doc(ci): Fix incorrect docs about Docker build cache order ZcashFoundation/zebra#5190

Merged

1 task

SoftwareApe mentioned this pull request Jan 11, 2023

docker build multiple "--cache-from" docs clarification needed docker/docs#8531

Closed

joverlee521 mentioned this pull request May 16, 2023

Revert "build: Use different cache and output registries" nextstrain/docker-base#155

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement build cache based on history array #26839

Implement build cache based on history array #26839

tonistiigi commented Sep 22, 2016 •

edited

Loading

graingert commented Sep 22, 2016

tonistiigi commented Sep 22, 2016

tonistiigi commented Sep 22, 2016

icecrime commented Sep 23, 2016

stevvooe Sep 22, 2016

stevvooe commented Sep 23, 2016

aaronlehmann Sep 23, 2016

tonistiigi Sep 23, 2016

aaronlehmann Sep 23, 2016

aaronlehmann Sep 23, 2016

aaronlehmann Sep 23, 2016

aaronlehmann commented Sep 23, 2016

tonistiigi commented Sep 23, 2016

stevvooe commented Sep 23, 2016

aaronlehmann left a comment

kachkaev commented Feb 3, 2017 •

edited

Loading

gajus commented Feb 3, 2017

kachkaev commented Feb 3, 2017 •

edited

Loading

graingert commented Feb 3, 2017

tonistiigi commented Feb 3, 2017

jkp commented Jan 28, 2019

garrett-hopper commented Jan 22, 2020

tonistiigi commented Jan 22, 2020

Implement build cache based on history array #26839

Implement build cache based on history array #26839

Conversation

tonistiigi commented Sep 22, 2016 • edited Loading

graingert commented Sep 22, 2016

tonistiigi commented Sep 22, 2016

tonistiigi commented Sep 22, 2016

icecrime commented Sep 23, 2016

stevvooe Sep 22, 2016

Choose a reason for hiding this comment

stevvooe commented Sep 23, 2016

aaronlehmann Sep 23, 2016

Choose a reason for hiding this comment

tonistiigi Sep 23, 2016

Choose a reason for hiding this comment

aaronlehmann Sep 23, 2016

Choose a reason for hiding this comment

aaronlehmann Sep 23, 2016

Choose a reason for hiding this comment

aaronlehmann Sep 23, 2016

Choose a reason for hiding this comment

aaronlehmann commented Sep 23, 2016

tonistiigi commented Sep 23, 2016

stevvooe commented Sep 23, 2016

aaronlehmann left a comment

Choose a reason for hiding this comment

kachkaev commented Feb 3, 2017 • edited Loading

gajus commented Feb 3, 2017

kachkaev commented Feb 3, 2017 • edited Loading

graingert commented Feb 3, 2017

tonistiigi commented Feb 3, 2017

jkp commented Jan 28, 2019

garrett-hopper commented Jan 22, 2020

tonistiigi commented Jan 22, 2020

tonistiigi commented Sep 22, 2016 •

edited

Loading

kachkaev commented Feb 3, 2017 •

edited

Loading

kachkaev commented Feb 3, 2017 •

edited

Loading