❗ Docker >=v20.10.5 (runc >= v1.0.0-rc93) is required for k3d v5.x ❗

[manumartin]

❗ TL;DR: Docker >=v20.10.5 is required for k3d v5.x ❗

What did you do

I tried to create a k3d cluster with k3d 5.x but it fails. Its working with 4.x

• How was the cluster created?

  • sudo k3d cluster create MYCLUSTER --trace --verbose

• What did you do afterwards?
  It seems like some kind of k3d/docker integration issue. I have reproduced the error message by using this docker-compose.yml:

test-container:
    image: rancher/k3d-tools:5.0.1
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    tmpfs:
      - /var/run

It seems you can't mount the docker sock volume inside the /var/run tmpfs mount and k3d is failing when trying to do the same with the k3d-tools container.

What did you expect to happen

I expected the cluster to start normally

Screenshots or terminal output

❯ sudo k3d cluster create MYCLUSTER --trace --verbose

...

INFO[0000] Starting new tools node...                   
TRAC[0000] Creating node from spec
&{Name:k3d-MYCLUSTER-tools Role:noRole Image:docker.io/rancher/k3d-tools:5.0.0 Volumes:[k3d-MYCLUSTER-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: RuntimeLabels:map[app:k3d k3d.cluster:MYCLUSTER k3d.version:v5.0.0] K3sNodeLabels:map[] Networks:[k3d-MYCLUSTER] ExtraHosts:[] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-MYCLUSTER-tools Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[noop] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-tools:5.0.0 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:MYCLUSTER k3d.role:noRole k3d.version:v5.0.0] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-MYCLUSTER-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc00020630f} NetworkingConfig:{EndpointsConfig:map[k3d-MYCLUSTER:0xc00024e000]}} 
DEBU[0000] Created container k3d-MYCLUSTER-tools (ID: 6aed150173f55ae288a9334302ccb6684dda8e9bdc1bc77af3454e4ab8743c64) 
DEBU[0000] Node k3d-MYCLUSTER-tools Start Time: 2021-10-18 17:15:55.925990393 +0200 CEST m=+0.122850186 
TRAC[0000] Starting node 'k3d-MYCLUSTER-tools'               
INFO[0000] Starting Node 'k3d-MYCLUSTER-tools'               
ERRO[0000] Failed to run tools container for cluster 'MYCLUSTER' 

...

INFO[0001] Starting existing tools node k3d-devo-tools... 
INFO[0001] Starting Node 'k3d-devo-tools'               
ERRO[0001] failed to gather environment information used for cluster creation: error starting existing tools node k3d-devo-tools: docker failed to start container for node 'k3d-devo-tools': Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/var/run/docker.sock" to rootfs at "/var/run/docker.sock" caused: stat /var/run/docker.sock: no such file or directory: unknown 
ERRO[0001] Failed to create cluster >>> Rolling Back 

Which OS & Architecture

• Linux x86

Which version of k3d

k3d version v5.0.1
k3s version v1.21.5-k3s1 (default)`

Which version of docker

Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:32 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

[manumartin]

It doesn't happen anymore after upgrading from Docker 2.10.2 and containerd 1.4.3 to Docker 2.10.9 and containerd 1.4.11

[franpog859]

Hey! I'm facing similar issue on Mac:

$ k3d cluster create mycluster

INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster'
INFO[0000] Created volume 'k3d-mycluster-images'
INFO[0000] Starting new tools node...
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0002] Pulling image 'docker.io/rancher/k3d-tools:5.0.1'
INFO[0003] Pulling image 'docker.io/rancher/k3s:latest'
INFO[0006] Starting Node 'k3d-mycluster-tools'
ERRO[0007] Failed to run tools container for cluster 'mycluster'
INFO[0031] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0033] Pulling image 'docker.io/rancher/k3d-proxy:5.0.1'
INFO[0041] Using the k3d-tools node to gather environment information
INFO[0041] Starting existing tools node k3d-mycluster-tools...
INFO[0041] Starting Node 'k3d-mycluster-tools'
ERRO[0041] failed to gather environment information used for cluster creation: error starting existing tools node k3d-mycluster-tools: docker failed to start container for node 'k3d-mycluster-tools': Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/run/host-services/docker.proxy.sock\\\" to rootfs \\\"/var/lib/docker/overlay2/7fd7682bb903b7768c61cc827cb4638f13568661f1c5b3ab007b9e289ebca6eb/merged\\\" at \\\"/var/run/docker.sock\\\" caused \\\"stat /run/host-services/docker.proxy.sock: no such file or directory\\\"\"": unknown
ERRO[0041] Failed to create cluster >>> Rolling Back
INFO[0041] Deleting cluster 'mycluster'
INFO[0041] Deleted k3d-mycluster-serverlb
INFO[0041] Deleted k3d-mycluster-server-0
INFO[0041] Deleted k3d-mycluster-tools
INFO[0041] Deleting cluster network 'k3d-mycluster'
INFO[0041] Deleting image volume 'k3d-mycluster-images'
FATA[0041] Cluster creation FAILED, all changes have been rolled back!

I have these installed and just upgraded with brew:

k3d version v5.0.1
docker engine version 19.03.13
docker containerd version v1.3.7
docker runc version 1.0.0-rc10
docker docker-init version 0.18.0

Do you maybe have a suggestion on what to do?

[manumartin]

At first I downgraded k3d to 4.x and that fixed the problem but k3d 5.x with the newest docker/containerd versions also fixed it for me.

[franpog859]

Thanks, @manumartin ! Upgrading it with brew didn't work for me. Either something is wrong with linking the binary or there is not the latest version of Docker available here. Anyway, upgrading it just via GUI (sic!) worked fine :D

[tiswanso]

Hit the same issue on Ubuntu 18.0.4 with the same docker, containerd, and runc versions @franpog859 mentioned above.

Upgraded docker via apt upgrade docker-ce to resolve it. The k3d prereqs should mention version compatibility or min version requirements.

The resulting versions that worked for me:

Client: Docker Engine - Community
 Version:           20.10.10
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        b485636
 Built:             Mon Oct 25 07:42:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.10
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       e2f740d
  Built:            Mon Oct 25 07:41:06 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[mjpowersjr]

I had the same exact problem, 4.x worked, upgrade to 5.x broke. As @tiswanso and others suggest, upgrading Docker fixed the issue. In case it helps - this is the version of Docker I was using on Ubuntu when in a broken state:

Docker version 20.10.5, build 55c4c88

[iwilltry42]

Hi @manumartin , thanks for opening this issue and thanks to the others for providing additional input and a "fix"/workaround.
The last update related to Docker in the tools package was a dependency update from docker v20.10.5 to v20.10.8 which already landed in k3d v4.4.8 in August 🤔 (#679) (similar for k3d itself)
I'm not sure if we can fix forward here, but I'll investigate today 👍

[iwilltry42]

Tested some versions of Docker now with k3d v5.1.0 and I can confirm that it only starts working with Docker v20.10.4+ 🤔

[tiswanso]

@iwilltry42 -- in my case just documenting the version dependencies would have spared me a decent amount of effort. I'm just using k3d in an environment I completely control so updating dependencies is not much of a problem. Ideally, the k3d doc has a version compatibility matrix.

[iwilltry42]

@tiswanso , I'm definitely going to add a requirement for Docker >=v20.10.5.
Unfortunately, there are just 5 million things to consider in compatibility matrices: OS (Linux/Mac/Windows) x Docker Variant (plain docker, Docker Machine, Docker for Mac, Docker for Windows, Docker for Windows with WSL2 backend) x Docker Version (what we face now) x k3d version (including dependency versions) x K3s version x Kernel version.

k3d x K3s I already had in CI/CD, which broke CI due to taking way to long to go through the E2E Tests with all K3s versions.
I guess Docker versions we can cover with a simple test case that should be run before merging a dependency update though ✔️

Update: After some investigation, I feel like it's caused by runc (seems like the update v1.0.0-rc92...v1.0.0-rc93 fixed it)

Update 2: Updated requirements on k3d.io and in the Readme: https://k3d.io/v5.1.0/#requirements

[schnatterer]

Thanks for this issue, it saved me from a lot of debugging!

I can report an occurrence of this issue on docker 20.10.5 running on Ubuntu 20.04.2 LTS, which seems to have been shipped with runc 1.0.0-rc92 😬

@iwilltry42 wouldn't it make more sense to clarify the requirement to "a docker version that uses runc >= v1.0.0-rc93?

$ docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:20 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:15 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

[iwilltry42]

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest.
I'll update the notes accordingly to reflect this.
Update: e82810a

[schnatterer]

@iwilltry42 yeah, I was wondering about this myself. Thanks for updating.

[regunakyle]

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest. I'll update the notes accordingly to reflect this. Update: e82810a

Sorry for bumping.

Do I need to check the version of both Docker and runc, or just runc?
My Synology NAS has:

• Docker 20.10.3
• runc v1.0.0-rc93
• containerd v1.4.3

Not sure if it can run K3d.

[iwilltry42]

@schnatterer I didn't expect docker to ship the same exact version with a different version of runc to be honest. I'll update the notes accordingly to reflect this. Update: e82810a

Sorry for bumping.

Do I need to check the version of both Docker and runc, or just runc?
My Synology NAS has:

• Docker 20.10.3
• runc v1.0.0-rc93
• containerd v1.4.3

Not sure if it can run K3d.

Hmm.. as far as I could tell, runc was the issue here, so that should be enough.
I'm not certain though, so better just try 🤔