[BUG] k3d v5.5.1 still not works with Podman rootless

[tbshrst]

Still experiencing the same issues with rootless podman as described in Issue 1052.
According to one answer the problem was fixed in k3d v5.4.2.

What did you do

• How was the cluster created?

  export DOCKER_SOCK=/var/run/docker.sock
  export DOCKER_HOST=unix:///var/run/docker.sock
  k3d cluster create test
  

What did you expect to happen

The cluster should be created.

Screenshots or terminal output

INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-test'                   
INFO[0000] Created image volume k3d-test-images         
INFO[0000] Starting new tools node...                   
ERRO[0000] Failed to run tools container for cluster 'test' 
INFO[0001] Creating node 'k3d-test-server-0'            
INFO[0001] Creating LoadBalancer 'k3d-test-serverlb'    
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] Starting new tools node...                   
ERRO[0001] Failed to run tools container for cluster 'test' 
ERRO[0001] failed to gather environment information used for cluster creation: failed to run k3d-tools node for cluster 'test': failed to create node 'k3d-test-tools': runtime failed to create node 'k3d-test-tools': failed to create container for node 'k3d-test-tools': docker failed to create container 'k3d-test-tools': Error response from daemon: make cli opts(): making volume mountpoint for volume /var/run/docker.sock: mkdir /var/run/docker.sock: permission denied 
ERRO[0001] Failed to create cluster >>> Rolling Back    
INFO[0001] Deleting cluster 'test'                      
INFO[0002] Deleting cluster network 'k3d-test'          
INFO[0002] Deleting 1 attached volumes...               
FATA[0002] Cluster creation FAILED, all changes have been rolled back! 

Which OS & Architecture

k3d runtime-info
arch: amd64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: hostname
name: docker
os: fedora
ostype: linux
version: 4.5.0 

Which version of k3d

k3d version
k3d version v5.5.1
k3s version v1.26.4-k3s1 (default)

Which version of docker podman

podman version
Client:       Podman Engine
Version:      4.5.0
API Version:  4.5.0
Go Version:   go1.19.7
OS/Arch:      linux/amd64

[Gianluca755]

Stupid question, did you use the official instructions? https://k3d.io/v5.5.2/usage/advanced/podman/?h=podman#using-rootless-podman

[iwilltry42]

Hey @tbshrst !
The error there is making volume mountpoint for volume /var/run/docker.sock: mkdir /var/run/docker.sock: permission denied, which means that basically docker run -v /var/run/docker.sock:/var/run/docker.sock doesn't work which is required to start the k3d-tools container to setup the environment for k3d.
That's nothing that k3d can do itself if it doesn't have permissions.

[tcassaert]

I don't think you can say that rootless Podman works, if you still rely on a docker run -v /var/run/docker.sock:/var/run/docker.sock, so to me this does look like something k3d should solve.

Can't this docker run... be replaced by a podman run... which mounts the podman socket?

EDIT
I just saw that OP didn't really follow the guide for rootless podman. I did follow the steps in the docs, but I see the same result.

$ systemctl --user enable --now podman.socket
$ systemctl --user status podman.socket                                                                                     1 ↵
● podman.socket - Podman API Socket
     Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; preset: enabled)
     Active: active (listening) since Mon 2023-09-25 08:49:22 CEST; 2 days ago
   Triggers: ● podman.service
       Docs: man:podman-system-service(1)
     Listen: /run/user/1000/podman/podman.sock (Stream)
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/podman.socket

Sep 25 08:49:22 workbox systemd[1360]: Listening on Podman API Socket.

$ XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
$ export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
$ echo $DOCKER_HOST
unix:///run/user/1000/podman/podman.sock
$ k3d cluster create test
WARN[0001] Failed to get random free port: failed to create tcp listener: listen tcp 192.168.130.12:0: bind: cannot assign requested address
WARN[0001] Falling back to internal port 6443 (may be blocked though)...
INFO[0001] Prep: Network
INFO[0001] Created network 'k3d-test'
INFO[0001] Created image volume k3d-test-images
INFO[0001] Starting new tools node...
INFO[0002] Creating node 'k3d-test-server-0'
ERRO[0002] Failed to run tools container for cluster 'test'
INFO[0002] Creating LoadBalancer 'k3d-test-serverlb'
INFO[0003] Using the k3d-tools node to gather environment information
INFO[0003] Starting new tools node...
ERRO[0004] Failed to run tools container for cluster 'test'
ERRO[0004] failed to gather environment information used for cluster creation: failed to run k3d-tools node for cluster 'test': failed to create node 'k3d-test-tools': runtime failed to create node 'k3d-test-tools': failed to create container for node 'k3d-test-tools': docker failed to create container 'k3d-test-tools': Error response from daemon: make cli opts(): making volume mountpoint for volume /var/run/docker.sock: mkdir /var/run/docker.sock: permission denied
ERRO[0004] Failed to create cluster >>> Rolling Back
INFO[0004] Deleting cluster 'test'
INFO[0004] Deleting cluster network 'k3d-test'
INFO[0004] Deleting 1 attached volumes...
FATA[0004] Cluster creation FAILED, all changes have been rolled back!

[tcassaert]

When doing a
sudo ln -s $XDG_RUNTIME_DIR/podman/podman.sock /var/run/docker.sock, it gets a step further. I guess this step should be added to the docs for rootless Podman?

But then it errors on

ERRO[0005] failed to gather environment information used for cluster creation: error starting existing tools node k3d-test-tools: docker failed to start container for node 'k3d-test-tools': Error response from daemon: crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied

Which I tracked down to a problem with crun, that can be solved with containers/podman#19930 (comment).

[tcassaert]

I gave it another shot today and got it to work. The most important part that I was missing and that seems to be missing in the docs is that we need to inform the Kubelet to run in userspace:

---
apiVersion: k3d.io/v1alpha5
kind: Simple
options:
  k3s:
    extraArgs:
      - arg: "--kubelet-arg=feature-gates=KubeletInUserNamespace=true"
        nodeFilters:
          - server:*
          - agent:*