Build fails when FROM image isn't cached

[howardjohn]

My dockerfile has a first line like:

FROM golang:1.13.4 as binary_tools_context

When building this, I get the following error:

$ docker buildx build .
[+] Building 1.8s (27/176)
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 19.49kB                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:1.13.4                                                             0.3s
 => [internal] load metadata for docker.io/library/ubuntu:bionic                                                                   0.0s
 => CACHED [clang_context 1/6] FROM docker.io/library/ubuntu:bionic                                                                0.0s
 => [internal] load build context                                                                                                  0.1s
 => => transferring context: 4.77kB                                                                                                0.0s
 => https://nodejs.org/download/release/v12.8.0/node-v12.8.0-headers.tar.gz                                                        1.0s
 => CANCELED https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-265.0.0-linux-x86_64.tar.gz               1.0s
 => CANCELED https://nodejs.org/download/release/v12.8.0/node-v12.8.0-linux-x64.tar.gz                                             1.0s
 => CANCELED https://github.com/ncopa/su-exec/archive/v0.2.tar.gz                                                                  0.1s
 => CANCELED https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz                                                            0.8s
 => CANCELED https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_am  0.1s
 => CANCELED [ruby_tools_context 2/14] RUN apt-get update && apt-get install -y --no-install-recommends     build-essential     c  1.0s
 => CANCELED https://storage.googleapis.com/kubernetes-release/release/v1.15.0/bin/linux/amd64/kubectl                             0.1s
 => CANCELED https://github.com/mikefarah/yq/releases/download/2.4.0/yq_linux_amd64                                                0.1s
 => CANCELED https://download.docker.com/linux/ubuntu/gpg                                                                          0.1s
 => CANCELED https://storage.googleapis.com/kubernetes-helm/helm-v2.16.1-linux-amd64.tar.gz                                        0.1s
 => CANCELED [base_os_context 2/19] RUN apt-get update && apt-get install -y --no-install-recommends     apt-transport-https       0.8s
 => CANCELED [python_context 2/8] RUN apt-get update && apt-get install -y --no-install-recommends     ca-certificates     curl    1.1s
 => CANCELED https://github.com/gohugoio/hugo/releases/download/v0.58.2/hugo_0.58.2_Linux-64bit.tar.gz                             0.1s
 => CANCELED https://github.com/hadolint/hadolint/releases/download/v1.17.2/hadolint-Linux-x86_64                                  0.1s
 => CANCELED https://storage.googleapis.com/shellcheck/shellcheck-v0.7.0.linux.x86_64.tar.xz                                       0.1s
 => CANCELED https://github.com/upx/upx/releases/download/v3.95/upx-3.95-amd64_linux.tar.xz                                        0.1s
 => CANCELED [bazel_context 2/5] RUN apt-get update && apt-get install -y --no-install-recommends     wget     ca-certificates  &  1.2s
 => CANCELED [clang_context 2/6] RUN apt-get update && apt-get install -y --no-install-recommends     xz-utils     wget     ca-ce  1.4s
 => CANCELED https://github.com/protocolbuffers/protobuf/releases/download/v3.9.2/protoc-3.9.2-linux-x86_64.zip                    0.1s
 => ERROR [binary_tools_context 1/77] FROM docker.io/library/golang:1.13.4                                                         0.1s
 => => resolve docker.io/library/golang:1.13.4                                                                                     0.1s
------
 > [internal] load metadata for docker.io/library/golang:1.13.4:
------
------
 > [binary_tools_context 1/77] FROM docker.io/library/golang:1.13.4:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: docker.io/library/golang:1.13.4 not found

If I docker pull docker.io/library/golang:1.13.4 everything works

Any suggestions? Is this user error? I assume not since it works with standard docker build but maybe I missed some docs around this

[tonistiigi]

Is this error persistent or did you just get it once?

[howardjohn]

@tonistiigi

Here is what I did just now:

docker buildx build . -> works (I have the image cached)
docker rmi golang:1.13.4
docker buildx build . -> fails
docker buildx build . -> fails
docker buildx build . -> fails
docker pull golang:1.13.4
docker buildx build . -> passes

So seems to be persistent

[tonistiigi]

Please post docker info docker version and dockerd daemon logs (enable debug if needed before).

[howardjohn]

I made a minimal docker file now. Adding more info

$ cat Dockerfile
FROM golang:1.13.4
$ docker buildx build .
[+] Building 0.4s (4/4) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                           0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:1.13.4                                                                                                                                         0.2s
 => ERROR [1/1] FROM docker.io/library/golang:1.13.4                                                                                                                                                           0.2s
 => => resolve docker.io/library/golang:1.13.4                                                                                                                                                                 0.2s
------
 > [internal] load metadata for docker.io/library/golang:1.13.4:
------
------
 > [1/1] FROM docker.io/library/golang:1.13.4:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: docker.io/library/golang:1.13.4 not found
$ docker version
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf583a
 Built:             Fri Oct 18 15:52:34 2019
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf583a
  Built:            Fri Oct 18 15:51:05 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 41
  Running: 1
  Paused: 0
  Stopped: 40
 Images: 1172
 Server Version: 19.03.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.67-2rodete2-amd64
 Operating System: Debian GNU/Linux bullseye/sid
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 62.64GiB
 Name: howardjohn.svl.corp.google.com
 ID: A757:6JXI:57IH:OP4C:LWX7:R2OK:ZYYG:GIQK:ARZ4:Q34S:7FUG:A4O5
 Docker Root Dir: /usr/local/google/docker
 Debug Mode: true
  File Descriptors: 50
  Goroutines: 81
  System Time: 2019-11-25T10:30:46.199097216-08:00
  EventsListeners: 0
 Username: howardjohn
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false

WARNING: No swap limit support

Might have something to do with

 Registry Mirrors:
  https://mirror.gcr.io/

?

Will look into getting the logs

[tonistiigi]

This seems to be caused by the mirror being configured while the mirror doesn't actually contain the image you are trying to use. Note that this is different from the behavior of the mirror mode in the upstream registry where it proxies to the origin registry if no local data is found. The capability to request from other registries if mirrors return errors was only added to containerd very recently and buildkit/docker has not been updated to the new logic yet. If you would like to help, the change would be to move from using Host callback to Hosts in https://github.com/containerd/containerd/blob/394db03f1531bfaa7018ecc470f29930aa30590f/remotes/docker/resolver.go#L90-L112

[mathieudevos]

For those still having issues with this, we configured a global env var: DOCKER_BUILDKIT=0 as temporary workaround. Note that this does disable buildkit as a whole, but this was pretty much the only thing which worked for us.

[caleb15]

I'm still having issues with this as well, I had to disable buildkit for one of my docker builds. Maybe this should be reopened?

#1142 (comment)

[Lakshpiya]

use docker login command and then provide user login account detail once login success then try again docker build command
It will work.

[tonistiigi]

This should be fixed in Buildkit v0.7+ / Docker 20.10 and iirc from some trace I saw it was also fixed on gcr side. If you still see this open a new issue with reproduction steps.

[black-snow]

@Lakshpiya Creating an account / signing in does not help.

I experience the same issue. I have to run an initial build with DOCKER_BUILDKIT=0. Follow-up runs work fine with DOCKER_BUILDKIT=1. Seems to fail downloading the image with buildkit.

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)
Server:

Containers: 0

Running: 0

Paused: 0

Stopped: 0

Images: 22

Server Version: 20.10.2

Storage Driver: overlay2

Backing Filesystem: extfs

Supports d_type: true

Native Overlay Diff: true

Logging Driver: json-file

Cgroup Driver: cgroupfs

Cgroup Version: 1

Plugins:

Volume: local

Network: bridge host ipvlan macvlan null overlay

Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog

Swarm: inactive

Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2

Default Runtime: runc

Init Binary: docker-init

containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b

runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff

init version: de40ad0

Security Options:

seccomp

Profile: default

Kernel Version: 4.19.128-microsoft-standard

Operating System: Docker Desktop

OSType: linux

Architecture: x86_64

CPUs: 8

Total Memory: 12.35GiB

Name: docker-desktop

ID: xxxx

Docker Root Dir: /var/lib/docker

Debug Mode: true

File Descriptors: 43

Goroutines: 50

System Time: 2021-01-08T16:46:21.9770826Z

EventsListeners: 4

Registry: https://index.docker.io/v1/

Labels:

Experimental: false

Insecure Registries:

127.0.0.0/8

Live Restore Enabled: false

Product License: Community Engine

[moltar]

Same here. I got this issue after pruning and was unable to recover. Only DOCKER_BUILDKIT=0 docker build works.

[felipecrs]

This issue seems to be returned. Same here.

[StefanoMantero]

Same here.
Docker version 20.10.5, build 55c4c88 and had to use the DOCKER_BUILDKIT=0.

[felipecrs]

I'm using the latest version of buildx, but I don't know which version of buildkit is being used. Is there any way to check the version of buildkit being used?

[foresthoffman]

@felipecrs Running docker info provides the version number for BuildKit for Docker in the current environment. Look for the line containing buildx.

[felipecrs]

Oh nice. I tried only docker version.

[HSPDev]

Same issue here:
Docker 20.10.6 on Ubuntu 20.4
BuildX v0.5.1-docker

I had to ENABLE DOCKER_BUILDKIT because --mount=type=cache,target=/tmp/cache started requiring BUILDKIT and making our builds fail on Github CI (We probably forgot to pin our builder base image, but okay).

Anyway, I thought "What a good reminder to move ahead and enable Buildkit".

Well, now I get this exact error as well. Gonna just pin our build image to a previous release, as I should have, from the start.

[thomas-bauer]

Same here:
macOS M1:
Docker 20.10.5
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

DOCKER_BUILDKIT=0 solves the problem (or docker pull the image first also works).

[prophet60091]

I too have started seeing this last week:
Windows 10 enterprise 17763
Docker 20.10.15
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

[rgeraads]

I ran into this same issue.

Building rmq
[+] Building 1.1s (4/6)                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile                                                                                                                                                      0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/rabbitmq:3.5.7-management                                                                                                                              0.8s
 => ERROR [1/3] FROM docker.io/library/rabbitmq:3.5.7-management@sha256:ceee535cb3ca8f24914d43583fd559887f01eaa9a6ef698791b47cc4097d0876                                                                  0.0s
 => => resolve docker.io/library/rabbitmq:3.5.7-management@sha256:ceee535cb3ca8f24914d43583fd559887f01eaa9a6ef698791b47cc4097d0876                                                                        0.0s
------
 > [1/3] FROM docker.io/library/rabbitmq:3.5.7-management@sha256:ceee535cb3ca8f24914d43583fd559887f01eaa9a6ef698791b47cc4097d0876:
------
failed to load cache key: invalid empty config file resolved for docker.io/library/rabbitmq:3.5.7-management@sha256:ceee535cb3ca8f24914d43583fd559887f01eaa9a6ef698791b47cc4097d0876

Solved it by running docker pull docker.io/library/rabbitmq:3.5.7-management@sha256:ceee535cb3ca8f24914d43583fd559887f01eaa9a6ef698791b47cc4097d0876 and then running build again.

[alansikora]

@rgeraads solution worked like a charm

[sheldonldev]

I got this problem because I have a typo in my Dockerfile FROM.

[salykin]

Our organization faced with the same issue only after moving to the new office building. We still have the same network configuration except network bandwidth - it was increased from 100Mb/s to 1Gb/s. It's very odd. 😳 We can't research this issue quickly to proof or refute the bandwidth impact.

We use a local Docker repository (Nexus) as a mirror (hosted and proxy repos). Some images pull from this repository, some images pull directly from Docker Hub. The issue happens for both kind of images randomly. It seems not to be an internet provider issue.

Because of random nature of the issue, it isn't reproducible for us.

The issue happens on Docker 19.03 with BuildKit, without buildx.

Today we did the upgrade of the Docker to 20.10 with buildx 0.5.1. Will keep watching.

Upd 1 [16.06.2021]: In our case the issue still happens after upgrading Docker version.

Upd 2 [16.06.2021]: In our case it seems to be a problem with pull rate limit on Nexus side when it tries to pull images from Docker Hub.

[harishsp]

Pulling the base image and then running docker build resolved the issue. No change to DOCKER_BUILDKIT=1

[hobin-kim]

I just modified the "credsStore" of docker.config as below and solved the problem.

{
"auths": {
"xxxxxxxxxxxxx": {
"auth": "xxxxxxxxxx"
}
},
"credsStore": ""
}

[painhardcore]

I just wanna set this thing on fire after 3 hours of debugging. DOCKER_BUILDKIT=0 solved this issue

Version:          20.10.8
OS/Arch:        darwin/arm64

[adjenks]

My recent experience:
The thing I'm building requires buildkit. Disabling buildkit helped it to download some of the required images. Then it failed because it needed buildkit. Updating to v20.10.17 did not resolve the issue, it failed again after update.
After manually calling docker pull for the rest of the images and turning buildkit back on it succeeded.