20.10.18 invalid memory address or nil pointer dereference

[Antiarchitect]

Description

time="2022-09-13T16:09:31.115294233Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :
time="2022-09-13T16:09:31.134434213Z" level=warning msg="image exporter: unknown option push"
time="2022-09-13T16:09:31.155217321Z" level=warning msg="image exporter: unknown option push"
time="2022-09-13T16:09:31.158718108Z" level=info msg="parsed scheme: \"\"" module=grpc
time="2022-09-13T16:09:31.158788839Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
time="2022-09-13T16:09:31.158822171Z" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" mod
time="2022-09-13T16:09:31.158844282Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-09-13T16:09:31.166961115Z" level=warning msg="image exporter: unknown option push"
time="2022-09-13T16:09:31.181344167Z" level=info msg="parsed scheme: \"\"" module=grpc
time="2022-09-13T16:09:31.181411502Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
time="2022-09-13T16:09:31.181440977Z" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" mod
time="2022-09-13T16:09:31.181454392Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-09-13T16:09:31.188971802Z" level=warning msg="image exporter: unknown option push"
time="2022-09-13T16:09:31.196852177Z" level=info msg="parsed scheme: \"\"" module=grpc
time="2022-09-13T16:09:31.196932846Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
time="2022-09-13T16:09:31.197006363Z" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" mod
time="2022-09-13T16:09:31.197029586Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-09-13T16:09:31.211864536Z" level=info msg="parsed scheme: \"\"" module=grpc
time="2022-09-13T16:09:31.212043528Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
time="2022-09-13T16:09:31.212104712Z" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" mod
time="2022-09-13T16:09:31.212128015Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-09-13T16:09:39.323666417Z" level=warning msg="failed to match any cache with layers"
time="2022-09-13T16:09:39.323738651Z" level=warning msg="failed to match any cache with layers"
time="2022-09-13T16:09:39.323854836Z" level=warning msg="failed to match any cache with layers"
time="2022-09-13T16:09:39.333863530Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :
time="2022-09-13T16:09:39.338772180Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :
time="2022-09-13T16:09:39.339128501Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :
time="2022-09-13T16:10:11.490516561Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1432b6d]

goroutine 61877 [running]:
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*sharedOp).CalcSlowCache.func2({0x23e19d0, 0xc002bee100})
    /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/jobs.go:650 +0x32d
github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).run(0xc002bf0000)
    /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:121 +0x5e
sync.(*Once).doSlow(0x441685?, 0xc0051ad620?)
    /usr/local/go/src/sync/once.go:68 +0xc2
sync.(*Once).Do(0xc0051ad620?, 0x0?)
    /usr/local/go/src/sync/once.go:59 +0x1f
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol.(*call).wait
    /go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:156 +0x3be

Reproduce

docker buildx bake with rather complex docker-bake.hcl something like:

variable "APP_VERSION" {}
variable "CACHE_IMAGE" {}
variable "IMAGE" {}
variable "BUILD_CONTEXT" {}
variable "DOCKERFILE_INLINE" {}
variable "CI_COMMIT_REF_SLUG" { default = "local" }
variable "CI_PIPELINE_ID" { default = "" }
variable "CI_COMMIT_TAG" { default = "" }

group "group--main" {
  targets = ["ruby", "gemfiles", "bundled", "assets", "main"]
}

group "group--cache-new" {
  targets = ["cache-new"]
}

target "default" {
  context = "${BUILD_CONTEXT}"
  dockerfile-inline = "${DOCKERFILE_INLINE}"
  args = {
    APP_VERSION = "${APP_VERSION}",
    CACHE_IMAGE = "${CACHE_IMAGE}",
  }
  cache-to = ["type=inline"]
}

target "ruby" {
  inherits = ["default"]
  target = "ruby"
  tags = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--ruby"
  ]
  cache-from = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--ruby",
    "${IMAGE}:master--ruby",
  ]
}

target "gemfiles" {
  inherits = ["default"]
  target = "gemfiles"
  tags = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--gemfiles",
  ]
  cache-from = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--gemfiles",
    "${IMAGE}:master--gemfiles",
  ]
}

target "bundled" {
  inherits = ["default"]
  target = "bundled"
  tags = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--bundled",
  ]
  cache-from = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--bundled",
    "${IMAGE}:master--bundled",
  ]
}

target "assets" {
  inherits = ["default"]
  target = "assets"
  tags = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--assets",
  ]
  cache-from = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--assets",
    "${IMAGE}:master--assets",
  ]
}

target "main" {
  inherits = ["default"]
  target = "main"
  tags = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}",
    equal("master",CI_COMMIT_REF_SLUG) ? "${IMAGE}:latest" : "",
    notequal("",CI_PIPELINE_ID) ? "${IMAGE}:${CI_COMMIT_REF_SLUG}-${CI_PIPELINE_ID}" : "",
    notequal("",CI_COMMIT_TAG) ? "${IMAGE}:${CI_COMMIT_TAG}" : "",
  ]
  cache-from = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}",
    "${IMAGE}:master",
  ]
}

target "cache-new" {
  inherits = ["default"]
  target = "cache-new"
  tags = [
    "${IMAGE}:${CI_COMMIT_REF_SLUG}--cache",
  ]
}

Expected behavior

Build success. Docker daemon does not crash.

docker version

Client:
 Version:           20.10.18
 API version:       1.41
 Go version:        go1.18.6
 Git commit:        b40c2f6
 Built:             Thu Sep  8 23:05:51 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       e42327a
  Built:            Thu Sep  8 23:11:23 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d1
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.10.2)
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.18
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: v1.1.4-0-g5fd4c4d1
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.19.7-1.el7.elrepo.x86_64
 Operating System: Alpine Linux v3.16 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.6GiB
 Name: app-gitlab-runners-docker-6
 ID: L6R5:4OMC:OFIA:FWNS:5DOZ:WSWN:OFNO:4S57:GYKH:7IKI:PG7J:AWDM
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://dockerhub.xxx.xxx/
 Live Restore Enabled: false
 Product License: Community Engine
 Default Address Pools:
   Base: 10.10.0.0/16, Size: 24
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/

Additional Info

Baremetal K8s
CentOS 7
Kernel: 5.19.7-1.el7.elrepo.x86_64

[LaurensRietveld]

We encounter this as well occasionally. What happens:

• There is a bug in buildkit 0.8.1
• It's fixed in this PR
• Released as buildkit 0.8.3
• But.... the docker CLI still builds with 0.8.1 (which is about two years old now)

[Antiarchitect]

Aha, I've expected something like this. @thaJeztah Is there anything possible to do not building custom dind image? Is there any info on the first 22.06.x release?

[thaJeztah]

We encounter this as well occasionally. What happens:

• There is a bug in buildkit 0.8.1
• It's fixed in this PR
• Released as buildkit 0.8.3
• But.... the docker CLI still builds with 0.8.1 (which is about two years old now)

That link looks to be to the docker cli, so only the client side of buildkit; the version used in the daemon is here, https://github.com/moby/moby/blob/v20.10.18/vendor.conf#L36

Which is some commits ahead of v0.8.3, so should contain the patch you linked to 🤔

[LaurensRietveld]

I'm not familiar with the internals. Does all the buildkit processing take place in the daemon we're connected to? I.e., if we're using an old docker cli (with buildkit 0.8.1) to connect to a new daemon via the socket, then we'll use the 0.8.3 version of buildkit for the actual building?

If that's the case, then I'm confused indeed, and there may be some other bug in play here

[bsousaa]

Are you still able to reproduce this on 23.0?

[Antiarchitect]

@bsousaa seems not this one. Now we're facing moby/buildkit#3635 on 23.0.1

[sam-thibault]

I will close this issue as it appears to be fixed in 23.0, and we can follow the issue in the buildkit issue you linked. (moby/buildkit#3635)