EPERM when opening /dev/net/tun in 1.8.0-beta.0 (tested in QEMU provisioner cluster)

[jfroy]

Bug Report

Description

In a brand new test cluster created with the qemu provisioner with talos v1.8.0-beta.0, in a test pod that should be able to open /dev/net/tun, it will fail with EPERM.

If you repeat the same steps with v1.7.6, you will be able to open /dev/net/tun.

This is likely a regression or change with containerd v2 or the kubelet.

I can reproduce on a bigger baremetal cluster as well. This is not limited to the qemu provisioner.

Steps

mkdir _out
curl https://github.com/siderolabs/talos/releases/download/v1.8.0-beta.0/vmlinuz-amd64 -L -o _out/vmlinuz-amd64
curl https://github.com/siderolabs/talos/releases/download/v1.8.0-beta.0/initramfs-amd64.xz -L -o _out/initramfs-amd64.xz
curl https://github.com/siderolabs/talos/releases/download/v1.8.0-beta.0/talosctl-linux-amd64 -L -o talosctl
chmod +x talosctl
sudo --preserve-env=HOME ./talosctl cluster create --provisioner qemu
kubectl apply -f ./tun-test.yaml
kubectl exec -ti -n kube-system tun-test -- sh

In the pod shell:

apk add -U python3
python3 -c 'open("/dev/net/tun")'

tun-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: tun-test
  namespace: kube-system
spec:
  containers:
  - image: alpine
    name: test
    command:
      - sleep
      - "10000000"
    resources: {}
    volumeMounts:
      - name: tun
        mountPath: /dev/net
    securityContext:
      capabilities:
        drop:
          - ALL
        add:
          - NET_ADMIN
  volumes:
    - name: tun
      hostPath:
        path: /dev/net
  dnsPolicy: ClusterFirst
  restartPolicy: Always

Logs

/ # python3 -c 'open("/dev/net/tun")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
PermissionError: [Errno 1] Operation not permitted: '/dev/net/tun'

Environment

• Talos version:

  Client:
      Tag:         v1.8.0-beta.0
      SHA:         51b91d64
      Built:       
      Go version:  go1.22.7
      OS/Arch:     linux/amd64
  

[jfroy]

This is likely caused by this runc change. I am not sure if the pod spec can be changed or if containerd or kubelet need to be patched.

[jfroy]

containerd used to include tun/tap in its default allow list, but it also has removed it (2 years ago!) in this PR.

[jfroy]

One workaround (or arguably correct approach) is to use a device plugin, such as https://github.com/squat/generic-device-plugin or https://gitlab.com/arm-research/smarter/smarter-device-manager, and expose net/tun that way.

I've tested with generic-device-plugin with this manifest and it works:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: generic-device-plugin
  namespace: kube-system
  labels:
    app.kubernetes.io/name: generic-device-plugin
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: generic-device-plugin
  template:
    metadata:
      labels:
        app.kubernetes.io/name: generic-device-plugin
    spec:
      priorityClassName: system-node-critical
      tolerations:
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
      containers:
      - image: squat/generic-device-plugin
        args:
        - --device
        - |
          name: tun
          groups:
            - count: 1000
              paths:
                - path: /dev/net/tun
        name: generic-device-plugin
        resources:
          requests:
            cpu: 50m
            memory: 10Mi
          limits:
            cpu: 50m
            memory: 20Mi
        ports:
        - containerPort: 8080
          name: http
        securityContext:
          privileged: true
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
        - name: dev
          mountPath: /dev
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins
      - name: dev
        hostPath:
          path: /dev
  updateStrategy:
    type: RollingUpdate

[jfroy]

I don't think that Talos can (should) do anything about this, but it probably merits a call out in the release notes. This is going to cause some pain, especially since runc 1.2x is not widely deployed yet.

[smira]

Yes, thanks, I guess it's worth an additional release notes item. But it was some grey area before - do you need to be privileged or not, e.g. loop devices require additional context.

[smira]

@jfroy thanks a lot for the detailed issue, I reworked your comments into #9330, putting you as a co-author, if you don't mind.