docker build --pull does not respect insecure registries configuration

[petr-ujezdsky]

Description

It seems like the docker build --pull flag does not respect insecure registries configuration as the docker pull does.

Standard docker pull <url> works fine. But building from Dockerfile

FROM <url>
...

using docker build -t my-image --pull . will fail. It is using https instead of http:

[+] Building 0.2s (3/3) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 37B
 => [internal] load .dockerignore
 => => transferring context: 2B
 => ERROR [internal] load metadata for <url>
------
 > [internal] load metadata for <url>:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head https://<url>: http: server gave HTTP response to HTTPS client

Steps to reproduce the issue:

1. use docker build command with --pull flag on Dockerfile with base image located at insecure registry

Describe the results you received:
Error - docker tried the url with https

Describe the results you expected:
Docker should fetch the image using http

Additional information you deem important (e.g. issue happens only occasionally):
This always happens. But it must have been broken some time ago, at some point this worked (like 2 years ago).

Output of docker version:

docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:22 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 17
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.64GiB
 Name: docker-desktop
 ID: 643S:D3I5:HOSB:I4OC:XR3J:HGTH:ZPQP:ZLYE:VWII:EBEM:MG5F:65RL
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  <url-domain-with-port>
  127.0.0.0/8
 Live Restore Enabled: false

Migrated from distribution/distribution#3454.

[dy21335]

did u solve your problem? i have the same problem and don't know how to fix it...

[petr-ujezdsky]

Its bug on docker side, you can not solve it directly. The only solution is to use secured registries with https, certificates etc.

[guai]

docker pull works for me. and then I can docker build using this image

[thaJeztah]

It's an issue with BuildKit, but may depend on the driver that's used for the builder (container driver, docker driver, etc.). I think various fixes have been implemented in BuildKit, but @crazy-max or @tonistiigi may have a better view on possibly remaining situations where the daemon configuration is not used for the builder.

[liyuntao]

It's an annoying manner. Break my gitlab-runner for a long time.

A quick fix:
add export DOCKER_BUILDKIT=0 to /etc/profile.

[sergeykad]

@liyuntao It's more of a workaround and you cannot use any modern Docker build features if you disable BuildKit.

[liyuntao]

@liyuntao It's more of a workaround and you cannot use any modern Docker build features if you disable BuildKit.

There's no way out of this for me. At least in the company. The inner image repository is not in my control.

Personally, I would like to bump version or using modern features all the time.

[thaJeztah]

@liyuntao what version of docker are you using? I'm curious if this is still an issue with current versions of the docker engine (which contain updated versions of BuildKit). Note that docker no longer ships updated versions of 20.10 (we only package the current (24.0 at time of writing) version). Docker 20.10 is still packaged/maintained by Mirantis for their products, but will reach full EOL next month.

[liyuntao]

Hi @thaJeztah thanks for your warm concern.

We are using Ubuntu 22.04 as CI build server, and a self-hostd harbor served on 80 port. insecure-registries configured for the docker daemon.

> docker version

Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:44 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:44 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

When a base image does not exist locally:

> docker build -t test-whatever .

[+] Building 0.2s (3/3) FINISHED                                                                                                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                    0.1s
 => => transferring dockerfile: 627B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for harbor.xx.xxx/common/openjdk17:alpine                                                                                                                       0.0s
------
 > [internal] load metadata for harbor.xx.xxx/common/openjdk17:alpine :
------
Dockerfile:1
--------------------
   1 | >>> FROM harbor.xx.xxx/common/openjdk17:alpine 
   2 |
   3 |     WORKDIR application
--------------------
ERROR: failed to solve: harbor.xx.xxx/common/openjdk17:alpine: failed to do request: Head "https://harbor.xx.xxx/v2/common/openjdk17/manifests/alpine": dial tcp 192.168.21.108:443: connect: connection refused

So I had to export this env, and then everything went normal after a warning message.

> export DOCKER_BUILDKIT=0

> docker build -t test-whatever .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  21.18MB
Step 1/8 : FROM harbor.xx.xxx/common/openjdk17:alpine
...

[thaJeztah]

Thanks!

I think this ticket may be a duplicate of / duplicated by docker/buildx#1642 (comment)

And I recall there was a fix for this that's merged and will be part of the upcoming v25.0 release of docker engine;

• daemon: use the registry service config for getting registry hosts moby/moby#45992

/cc @jedevc could you check if that's indeed the same?

[jedevc]

@thaJeztah sorry for the delay - just caching up on a pile of emails now.

This looks exactly like the symptoms around moby/moby#45992 - so would be fixed in the upcoming release.

Even if it's somehow subtly different (potentially possible, the code around this is insanely fiddly), then it would still likely be affected by moby/moby#45992 in some way, since the logic has now been centralized.

I think this is safe to close as a duplicate.

[thaJeztah]

Thanks! Let me go ahead and close this as duplicate. The patch above will be in the v25.0 release (beta's are available, and we're planning to do an rc + final release soon)