Support for user namespaces for stateful pods

[ikwork]

Hi

K8s 1.28 with crio 1.28 fully support user namespaces for stateful pods.

More info here:
https://kubernetes.io/blog/2023/09/13/userns-alpha/

This got GA'd in K8S in 1.28 and we've been using this with CRIO 1.28 successfully.

When is this going to be available in OpenShift/OKD?

[vrutkovs]

In 4.15. Feel free to try nightlies (make sure you mirror them as they are pruned often) - https://amd64.origin.releases.ci.openshift.org/#4.15.0-0.okd

[vrutkovs]

See https://docs.okd.io/latest/nodes/clusters/nodes-cluster-enabling-features.html and customNoUpgrade in https://docs.okd.io/4.14/rest_api/config_apis/featuregate-config-openshift-io-v1.html#spec

[ikwork]

Thanks @vrutkovs , that worked, got the feature gate enabled on apiserver as well as kubelet.

But it seems like the usernamespacessupport feature wouldn't work yet in 4.15?

I created my single node ocp with version 4.15.0-ec.2, and I can see it uses crio 1.28 and k8s 1.28, but the kernel is too old, it is 5.14

oc get node -o wide
NAME   STATUS   ROLES                         AGE   VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION
      CONTAINER-RUNTIME
node   Ready    control-plane,master,worker   88m   v1.28.3+4cbdd29   192.168.123.5   <none>        Red Hat Enterprise Linux CoreOS 415.92.202311061558-0 (Plow)   5.14.0-284.40.1.el9_2.x86_64   cri-o://1.28.2-2.rhaos4.15.gite7be4e1.el9

According to the k8s documentation on UserNamespacesSupport, the kernel version needs to be a at least 6.3 and above, for the idmap mount support to be available

Node Requirements for UserNamespacesSupport

I guess I'm facing the below issue because of the same:

# cat deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      hostUsers: false
      containers:
      - image: nginx
        name: nginx
        command:
        - "/bin/bash"
        - "-c"
        - "tail -f /dev/null"

# oc get po test-6f5bc89f5b-mblk2
NAME                    READY   STATUS                 RESTARTS   AGE
test-6f5bc89f5b-mblk2   0/1     CreateContainerError   0          30m

# oc describe po test-6f5bc89f5b-mblk2
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       29m                   default-scheduler  Successfully assigned default/test-6f5bc89f5b-mblk2 to node
  Normal   AddedInterface  28m                   multus             Add eth0 [10.128.0.81/23] from ovn-kubernetes
  Normal   Pulled          28m                   kubelet            Successfully pulled image "nginx" in 2.786s (2.786s including waiting)
  Warning  Failed          13m (x43 over 28m)    kubelet            Error: idmap mounts specified but OCI runtime does not support them. Perhaps the OCI runtime is too old
  Normal   Pulled          8m46s (x47 over 25m)  kubelet            (combined from similar events): Successfully pulled image "nginx" in 2.66s (2.66s including waiting)
  Normal   Pulling         3m41s (x71 over 28m)  kubelet            Pulling image "nginx"

As OCP and OKD use Red Hat Enterprise Linux CoreOS for the node's OS, I can see the latest versions of RHEL 9.3 still uses 5.14 https://access.redhat.com/articles/3078

What does this mean regarding the availability of kernel version 6.3+ in OCP/OKD as it is required for UserNamespacesSupport ?

Thanks

[vrutkovs]

OKD FCOS is using Fedora kernel, which is 6.5+

[ikwork]

Thanks @vrutkovs for all the help!

I want to try out OKD 4.15 with FCOS, so I was following your instructions from https://hackmd.io/tDnHM2BoQru0VgqMGI_Q5g as it relates the most to the scenario I'm trying it in (single node okd with fcos on kvm)

But I can't find the RELEASE_IMAGE for 4.15 on quay https://quay.io/repository/openshift/okd?tab=tags

Is there any other repository to pull it from? or if I can setup my repository if I can get the image.

Thanks!

EDIT :
Just found that the below ones exist
registry.ci.openshift.org/origin/release:4.15
registry.ci.openshift.org/origin/release:4.15.0-0.okd-2023-12-04-004851

trying those right now, will update here

[vrutkovs]

See https://amd64.origin.releases.ci.openshift.org/#4.15.0-0.okd for nightly builds and their status. Don't forget to mirror them as those are regularly pruned