skaffold build: failed to connecto to docker socket

[broady]

Expected behavior

App gets built

Actual behavior

Error message:

===> ANALYZING
[analyzer] ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: connection refused
couldn't build "gcr.io/k8s-skaffold/skaffold-buildpacks": executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 1

Information

• Skaffold version: v1.13.2
• Operating system: Ubuntu running inside WSL2 on Windows 10
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta5
kind: Config
build:
  artifacts:
  - image: gcr.io/k8s-skaffold/skaffold-buildpacks
    buildpacks:
      builder: "gcr.io/buildpacks/builder:v1"

The pack command seems to get further:

$ go run github.com/buildpacks/pack/cmd/pack build charlie/flask --builder gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/builder
Digest: sha256:7811ff307f0b99c76d8f97a2166954d011e447529ae5c6688c042f3119052c58
Status: Image is up to date for gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/gcp/run
Digest: sha256:de0fd481b8156bba03e80a4544fca2d6cbcdf6bcac3c1e9aeb6fc364ccfae54a
Status: Image is up to date for gcr.io/buildpacks/gcp/run:v1
===> DETECTING
google.python.runtime            0.9.0
google.python.missing-entrypoint 0.9.0
google.utils.label               0.0.1
===> ANALYZING
Previous image with name "index.docker.io/charlie/flask:latest" not found
===> RESTORING
===> BUILDING
=== Python - Runtime ([email protected]) ===
Using latest runtime version: 3.8.2
Installing Python v3.8.2
Upgrading pip to the latest version and installing build tools
--------------------------------------------------------------------------------
Running "/layers/google.python.runtime/python/bin/python3 -m pip install --upgrade pip setuptools wheel"
Collecting pip
  Downloading https://files.pythonhosted.org/packages/5a/4a/39400ff9b36e719bdf8f31c99fe1fa7842a42fa77432e584f707a5080063/pip-20.2.2-py2.py3-none-any.whl (1.5MB)
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/6d/ed/52e05469314a266f68d9f251a8c1ab7a21a03327b1724056e3eea654bfd1/setuptools-50.0.3-py3-none-any.whl (784kB)
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/a7/00/3df031b3ecd5444d572141321537080b40c1c25e1caa3d86cdd12e5e919c/wheel-0.35.1-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed pip-20.2.2 setuptools-50.0.3 wheel-0.35.1
Done "/layers/google.python.runtime/python/bin/python3 -m pip inst..." (2.8032094s)
=== Python - pip ([email protected]) ===
Failure: (ID: 194879d1) Failed to run /bin/build: for Python, an entrypoint must be manually set, either with "GOOGLE_ENTRYPOINT" env var or by creating a "Procfile" file
--------------------------------------------------------------------------------
Sorry your project couldn't be built.
Our documentation explains ways to configure Buildpacks to better recognise your project:
 -> https://github.com/GoogleCloudPlatform/buildpacks/blob/main/README.md
If you think you've found an issue, please report it:
 -> https://github.com/GoogleCloudPlatform/buildpacks/issues/new
--------------------------------------------------------------------------------
ERROR: failed to build: exit status 1
ERROR: failed to build: failed with status code: 145
exit status 1

Docker works fine:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$ docker context ls
NAME                DESCRIPTION                               DOCKER ENDPOINT               KUBERNETES ENDPOINT   ORCHESTRATOR
default *           Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                         swarm
$ echo $DOCKER_HOST

[briandealwis]

@gsquared94 I think you have a Windows box with WSL2 available? Do you see this with examples/buildpacks?

[gsquared94]

@briandealwis I don't have access to it right now unfortunately.

[broady]

Oh and to clarify, I'm using Docker Desktop's WSL2 backend

[gsquared94]

I'll get to this as soon as I can get a device setup.

[gsquared94]

sorry it took a while to get to this. I was successful in running examples/buildpacks on a GCP windows VM on a minikube cluster. However it didn't have Docker Desktop's WSL2 backend and was using the default HyperV.

@broady is the skaffold.yaml that you posted the entire file? Can you try to clone the examples/buildpacks project and try running that?

[broady]

@gsquared94

examples/buildpacks master $ skaffold run
Generating tags...
 - skaffold-buildpacks -> skaffold-buildpacks:v1.14.0-35-gd9cf47da6
Checking cache...
 - skaffold-buildpacks: Not found. Building
Building [skaffold-buildpacks]...
v1: Pulling from buildpacks/builder
ab0f59294661: Already exists
c2ae8b336f77: Already exists
3c2cba919283: Already exists
ecbf9124d083: Pull complete
40c7a2450407: Pull complete
dc8468fab7ae: Pull complete
f6a53edd0edd: Pull complete
c6a578d0591f: Pull complete
efd5a1de0e17: Pull complete
3dcc8a7ccfe2: Pull complete
2d5913e2b6f3: Pull complete
357fefdf9bc9: Pull complete
8a1df4ac84fa: Pull complete
33d464a1bccb: Pull complete
0ca41938accd: Pull complete
95edf20b060f: Pull complete
44e6934c1fff: Pull complete
cf695b4a504a: Pull complete
f80c30c20159: Pull complete
413a1d9efb9e: Pull complete
85ddbdf094d1: Pull complete
8a0d4b53d3e4: Pull complete
4fa939729406: Pull complete
223aed41c40f: Pull complete
a811727540f3: Pull complete
eda73169f415: Pull complete
d4ccf8504827: Pull complete
7afe5741f91a: Pull complete
02e69e6af7a4: Pull complete
87a1da5c9339: Pull complete
e28266e69ddc: Pull complete
3868e87d6aa6: Pull complete
3df107be08aa: Pull complete
9fc358be80e9: Pull complete
0f6bdda4fd48: Pull complete
e4d7e3cfdfef: Pull complete
d3d2c50cb9ae: Pull complete
66675da0196f: Pull complete
4ff11d1898ed: Pull complete
53f7877c3d60: Pull complete
4f4fb700ef54: Pull complete
3e35a4588c45: Pull complete
5b5b5cb60f58: Pull complete
Digest: sha256:96aa400762f095134649dc967892ac123fd4f7107fb5f2916386411774589e7b
Status: Downloaded newer image for gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/gcp/run
ab0f59294661: Already exists
c2ae8b336f77: Already exists
3c2cba919283: Already exists
ecbf9124d083: Already exists
40c7a2450407: Already exists
dc8468fab7ae: Already exists
Digest: sha256:388e1efa517d495ced62633f7d86e7b85c5afb1cec62b97c1f099cbd763d0a94
Status: Downloaded newer image for gcr.io/buildpacks/gcp/run:v1
0.9.1: Pulling from buildpacksio/lifecycle
Digest: sha256:53bf0e18a734e0c4071aa39b950ed8841f82936e53fb2a0df56c6aa07f9c5023
Status: Image is up to date for buildpacksio/lifecycle:0.9.1
===> DETECTING
[detector] 3 of 6 buildpacks participating
[detector] google.go.runtime  0.9.0
[detector] google.go.build    0.9.0
[detector] google.utils.label 0.0.1
===> ANALYZING
[analyzer] ERROR: failed to initialize docker client: failed to connect to docker socket: dial unix /var/run/docker.sock: connect: connection refused
failed to build: couldn't build "skaffold-buildpacks": executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 1
Exit code 1
examples/buildpacks master $ skaffold version
v1.14.0

[broady]

FWIW a Windows machine running in a VM is going to be set up very different than Docker Desktop with WSL2 backend.

[broady]

Any way to remove the "priority/awaiting-more-evidence" tag? I don't think anything's waiting on me?

[broady]

I updated my Docker Desktop version and now get this error instead:

$ skaffold run
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-buildpacks -> gcr.io/k8s-skaffold/skaffold-buildpacks:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-buildpacks: Not found. Building
Building [gcr.io/k8s-skaffold/skaffold-buildpacks]...
v1: Pulling from buildpacks/builder
Digest: sha256:d7a9613e0a7a5cb3b3ca747ae8315330d160606226b6f089636a13d17127cc03
Status: Image is up to date for gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/gcp/run
Digest: sha256:3c96cb8a1a4c32a563147ffd8968244e515b48518e20b775d0d7fc64c521f17a
Status: Image is up to date for gcr.io/buildpacks/gcp/run:v1
0.9.1: Pulling from buildpacksio/lifecycle
Digest: sha256:53bf0e18a734e0c4071aa39b950ed8841f82936e53fb2a0df56c6aa07f9c5023
Status: Image is up to date for buildpacksio/lifecycle:0.9.1
===> DETECTING
[detector] 3 of 6 buildpacks participating
[detector] google.go.runtime  0.9.1
[detector] google.go.build    0.9.0
[detector] google.utils.label 0.0.1
===> ANALYZING
failed to build: couldn't build "gcr.io/k8s-skaffold/skaffold-buildpacks": executing lifecycle. This may be the result of using an untrusted builder: failed to create 'analyzer' container: Error response from daemon: not a directory
Exit code 1

I added some log statements:

===> ANALYZING
2020/10/13 23:47:25 sendRequest/checkResponseErr POST http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create err Error response from daemon: not a directory
2020/10/13 23:47:25 post create container: Error response from daemon: not a directory
2020/10/13 23:47:25 sendRequest/checkResponseErr DELETE http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers?force=1 err Error response from daemon: page not found
2020/10/13 23:47:25 sendRequest/checkResponseErr DELETE http://%2Fvar%2Frun%2Fdocker.sock/v1.40/volumes/pack-layers-zzrrjtizob?force=1 err <nil>
2020/10/13 23:47:25 sendRequest/checkResponseErr DELETE http://%2Fvar%2Frun%2Fdocker.sock/v1.40/volumes/pack-app-kdohiuvfxo?force=1 err <nil>
2020/10/13 23:47:25 sendRequest/checkResponseErr DELETE http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/pack.local/builder/6f67667376646b736f79:latest?force=1&noprune=1 err <nil>
failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed to create 'analyzer' container: Error response from daemon: not a directory

[IsaacPD]

I have Docker Desktop with WSL2 Backend on my Windows Machine and I was not able to reproduce the errors above with examples/buildpacks. Everything seemed to work fine when i ran "skaffold run" the only reason the build didn't complete on my end was cause "Build Failed. No push access to specified image repository. Trying running with --default-repo flag." which is expected.

I tried with
skaffold version v1.14.0 and v1.15.0
Docker Version 19.03.13

[briandealwis]

Hmm, this works for me too (Win 10 version 2004, OS Build 19041.450, Skaffold v1.15.0).

[chanseokoh]

@broady what do docker version and docker info say?

[broady]

Users/cbro $ docker version
Client: Docker Engine - Community
 Cloud integration  0.1.18
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:36 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:07:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
Users/cbro $ docker info
Client:
 Debug Mode: false
 Plugins:
  ecs: Docker ECS (Docker Inc., v1.0.0-beta.5)
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-tp-docker)

Server:
 Containers: 33
  Running: 2
  Paused: 0
  Stopped: 31
 Images: 744
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.104-microsoft-standard
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.36GiB
 Name: docker-desktop
 ID: 7EGP:XRAW:XEFK:C3H6:QYAO:4CNY:YLMC:YIAS:W4AI:552V:IW4F:NEVV
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 58
  Goroutines: 63
  System Time: 2020-10-14T15:48:17.8232853Z
  EventsListeners: 3
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[chanseokoh]

Error response from daemon: not a directory
Error response from daemon: page not found

From these errors, at least it seems like you can connect to the docker socket from Skaffold after you upgraded Docker.

I speculated that the "page not found" error response from the daemon might be because of API version mismatch between the client and the server (e.g., hitting a wrong endpoint, hence "page not found"). But obviously, your server and client are using the same version, so I guess it's not the cause. It may just be logging that there's no containers to force-delete.

For the "not a directory" response, I just have a wild speculation that some internal Docker daemon state is corrupted or to that effect. For example,

https://stackoverflow.com/a/45104143/1701388
https://stackoverflow.com/a/56090070/1701388

Maybe try cleaning up everything (running/stopped containers, all the images, but especially all the volumes that are both being used and unused, ...). But at least you are getting a response from the daemon, so I think your original issue that you cannot connect to the daemon is resolved somehow.

But I'm not a Docker expert, and all I said may just be wrong.

[broady]

I stopped all my running containers, ran docker system prune --all -force, docker volume prune, restarted the Docker daemon,
and get the same error.

I also just tried moving the directory away from the network-mounted filesystem (/mnt/c/...) to the Ubuntu/WSL homedir, same error.

[broady]

But now, strangely, pack is failing. So it's no longer only failing under Skaffold:

$ run github.com/buildpacks/pack/cmd/pack build charlie/flask --builder gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/builder
Digest: sha256:d7a9613e0a7a5cb3b3ca747ae8315330d160606226b6f089636a13d17127cc03
Status: Image is up to date for gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/gcp/run
Digest: sha256:3c96cb8a1a4c32a563147ffd8968244e515b48518e20b775d0d7fc64c521f17a
Status: Image is up to date for gcr.io/buildpacks/gcp/run:v1
ERROR: failed to build: executing lifecycle: failed to create 'creator' container: Error response from daemon: not a directory

[briandealwis]

@broady did Clean / Purge data or Reset to factory defaults make a difference?

[broady]

it did not. should I close this bug out? Looks like if there is a bug, it's probably in cmd/pack, not skaffold.

[briandealwis]

Yeah, let's close it given that several of us tried and were able to build successfully. You might try recreating your WSL2 instance, or uninstalling and reinstalling Docker. But at this point I'm just gesticulating wildly while trying to come up with a useful suggestion.