[BUG] Cannot import images on NixOS 20.09

[jlavelle]

What did you do

• How was the cluster created?

  • k3d cluster create test-cluster

• What did you do afterwards?

  docker pull hello-world:latest
  k3d image import hello-world:latest -c test-cluster --verbose

What did you expect to happen

Expected the image to be successfully imported into the cluster.

Screenshots or terminal output

~ $ k3d cluster create test-cluster
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-test-cluster' (86c878ebd0a4addd60d53f6b3e59219fcfc29971bcd83b313dbf2be720c6f50b) 
INFO[0000] Created volume 'k3d-test-cluster-images'     
INFO[0001] Creating node 'k3d-test-cluster-server-0'    
INFO[0001] Creating LoadBalancer 'k3d-test-cluster-serverlb' 
INFO[0001] Starting cluster 'test-cluster'              
INFO[0001] Starting servers...                          
INFO[0001] Starting Node 'k3d-test-cluster-server-0'    
INFO[0006] Starting agents...                           
INFO[0006] Starting helpers...                          
INFO[0006] Starting Node 'k3d-test-cluster-serverlb'    
INFO[0006] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access 
INFO[0009] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap 
INFO[0009] Cluster 'test-cluster' created successfully! 
INFO[0009] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false 
INFO[0009] You can now use it like this:                
kubectl config use-context k3d-test-cluster
kubectl cluster-info
~ $ docker pull hello-world:latest
latest: Pulling from library/hello-world
Digest: sha256:df5f5184104426b65967e016ff2ac0bfcd44ad7899ca3bbcf8e44e4461491a9e
Status: Image is up to date for hello-world:latest
docker.io/library/hello-world:latest
~ $ k3d image import hello-world:latest -c test-cluster --verbose
DEBU[0000] Runtime Info:
&{Name:docker Endpoint:/var/run/docker.sock Version:19.03.12 OSType:linux OS:NixOS 20.09 (Nightingale) Arch:x86_64 CgroupVersion: CgroupDriver:cgroupfs Filesystem:extfs} 
DEBU[0000] Importing image(s) [[hello-world:latest]] from runtime [{}] into cluster(s) [[{Name:test-cluster Network:{Name: ID: External:false IPAM:{IPPrefix:invalid IP prefix IPsUsed:[] Managed:false}} Token: Nodes:[] InitNode:<nil> ExternalDatastore:<nil> KubeAPI:<nil> ServerLoadBalancer:<nil> ImageVolume:}]]... 
INFO[0000] Importing image(s) into cluster 'test-cluster' 
DEBU[0000] Selected image 'hello-world:latest' (found as 'hello-world:latest') in runtime 
DEBU[0000] Attaching to cluster's image volume 'k3d-test-cluster-images' 
INFO[0000] Starting new tools node...                   
DEBU[0000] Failed to parse cgroupVersion: strconv.Atoi: parsing "": invalid syntax 
DEBU[0000] Created container k3d-test-cluster-tools (ID: a733f491a4ef69e5d155f617ab9b62f3cd03578c4e110c2748fa9b57ef2d49d4) 
DEBU[0000] Node k3d-test-cluster-tools Start Time: 2021-07-14 16:57:06.320319131 -0400 EDT m=+0.084225184 
INFO[0000] Starting Node 'k3d-test-cluster-tools'       
DEBU[0000] Truncated 2021-07-14 20:57:06.611099721 +0000 UTC to 2021-07-14 20:57:06 +0000 UTC 
INFO[0000] Saving 1 image(s) from runtime...            
DEBU[0000] Executing command '[./k3d-tools save-image -d /k3d/images/k3d-test-cluster-images-20210714165706.tar hello-world:latest]' in node 'k3d-test-cluster-tools' 
ERRO[0001] Failed to save image(s) in tools container for cluster 'test-cluster' 
ERRO[0001] Failed to import image(s) into cluster 'test-cluster': Exec process in node 'k3d-test-cluster-tools' failed with exit code '1' 
WARN[0001] At least one error occured while trying to import the image(s) into the selected cluster(s)

Which OS & Architecture

NixOS 20.09 (Nightingale)

Which version of k3d

k3d version v4.4.7
k3s version v1.20.6-k3s1 (default)

Which version of docker

Client:
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.15.8
 Git commit:        v19.03.12
 Built:             Wed Mar 10 05:46:22 2021
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.15.8
  Git commit:       v19.03.12
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc10
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:       

[iwilltry42]

Hi @jlavelle , thanks for opening this issue!
Does running the command with the --trace flag give some more information?
Can you see via docker ps during the process if there's something wrong with the tools container? 🤔

[kannamr]

@iwilltry42 I am facing the same issue.
I tried to docker exec -d k3d-osdk-test-tools ./k3d-tools save-image -d /k3d/images/k3d-test-images-20210716144727.tar hello-world:latest which gave an error Error response from daemon: client version 1.41 is too new. Maximum supported API version is 1.40

BTW, I have the same docker version as @jlavelle

[iwilltry42]

Hey there 👋
Can you try with this release please and give some feedback?
https://github.com/rancher/k3d/releases/tag/v4.4.8-rc.0

[fkn-iteratec]

Hi, I am facing the same issue and I tried the release v4.4.8-rc.0. But this dosen't fix the problem. As @kannamr reported by calling the docker exec command the error ist: Error response from daemon: client version 1.41 is too new. Maximum supported API version is 1.40

[iwilltry42]

Releasing v4.4.8-rc.1 with verified fix right now. Please re-open if the issue is still not fixed for your with that new version 👍
May take up to 30m depending on the release server utilization 👍

UPDATE: https://github.com/rancher/k3d/releases/tag/v4.4.8-rc.1
UPDATE 2: release pipeline was broken and didn't push the required manifests to dockerhub..

[iwilltry42]

Releasing v4.4.8-rc.1 with verified fix right now. Please re-open if the issue is still not fixed for your with that new version +1
May take up to 30m depending on the release server utilization +1

UPDATE: v4.4.8-rc.1 (release)
UPDATE 2: release pipeline was broken and didn't push the required manifests to dockerhub..

Okay, so now that the release pipeline is fixed, please give https://github.com/rancher/k3d/releases/tag/v4.4.8-rc.2 a try 👍

[iwilltry42]

@jlavelle , @kannamr & @fkn-iteratec can anyone of you confirm (or deny) the fix in https://github.com/rancher/k3d/releases/tag/v4.4.8-rc.2 please?

[fkn-iteratec]

There is a new problem with the given release above. So I cannot test it. During the cluster create k3d tries to pull the image rancher/k3d-proxy:v4.4.8-rc.2. The problem is the version tag. The letter 'v' must be omitted: rancher/k3d-proxy:4.4.8-rc.2.

My test log:

➜  k3d ./k3d-linux-amd64 cluster create shorttest
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-shorttest' (305a18fa96a15ebad6df74d49f2f214d30cef9854b89e4f77659042c5df947a7) 
INFO[0000] Created volume 'k3d-shorttest-images'        
INFO[0001] Creating node 'k3d-shorttest-server-0'       
INFO[0002] Pulling image 'docker.io/rancher/k3s:v1.21.3-k3s1' 
INFO[0014] Creating LoadBalancer 'k3d-shorttest-serverlb' 
ERRO[0015] Failed to pull image 'docker.io/rancher/k3d-proxy:v4.4.8-rc.2' 
ERRO[0015] Failed to create container 'k3d-shorttest-serverlb' 
ERRO[0015] Failed to create node 'k3d-shorttest-serverlb' 
ERRO[0015] Failed to create loadbalancer                
ERRO[0015] Failed Cluster Creation: Error response from daemon: manifest for rancher/k3d-proxy:v4.4.8-rc.2 not found: manifest unknown: manifest unknown 
ERRO[0015] Failed to create cluster >>> Rolling Back    
INFO[0015] Deleting cluster 'shorttest'                 
INFO[0015] Deleted k3d-shorttest-server-0               
ERRO[0015] Failed to delete container 'k3d-shorttest-serverlb' 
ERRO[0015] Error: No such container: k3d-shorttest-serverlb 
INFO[0015] Deleting cluster network 'k3d-shorttest'     
INFO[0016] Deleting image volume 'k3d-shorttest-images' 
FATA[0016] Cluster creation FAILED, all changes have been rolled back! 

[iwilltry42]

Ah crap, that's an issue with the new release pipeline (auto server tagging).
You can still test by setting the environment variable (K3D_HELPER_IMAGE_TAG=4.4.8-rc.2).
I will fix the reference in the code 👍

[fkn-iteratec]

Seems to work. I tried only to import the image without starting pods.

@iwilltry42: Thanks for your support !