Running rootless Podman inside OpenShift container #6667
Comments
openshift-ci-robot
added
the
kind/bug
|
If you are pulling an image with multiple UIDs then you need to run the container with multiple UIDs Usually you can do this as root. If you are running in openshift without being root, then this could cause issues. You could setup user namespace within the container, but it would still require CAP_SETUID and CAP_SETGID in order to start the user namespace within the container. |
|
Yes, I am pulling the fedora image. I do not mind CAP_SETUID/GID as much on their own. But I think running under the 'privileged' SCC would be a deal breaker. I will give it a shot though, just to confirm that I can get it running. Do I understand the last sentence correctly? Is it possible to setup an additional user namespace inside the container, even though OpenShift only appears to provide it an uidmap size of 1? Ta! |
|
You should be able to get this to work. This is the way the buildah and podman images are configured in quay.io/buildah/stable and quay.io/podman/stable https://github.com/containers/libpod/blob/master/contrib/podmanimage/stable/Dockerfile The idea is to setup a user within the container and then to launch the container with that user. |
|
Just a progress update: I have tried using the podman:stable image as base for my OpenShift container, and it does bring me past the user id problem. So I can now pull images. Thanks! But running anything in those pulled images still fails (due to missing fuse module, it seems). I will explore further on Monday. |
|
Waiting for the rain to end, so I collected a little more info. Running an image fails with: The kernel is 4.18.0-147.8.1.el8_1.x86_64 (which should be new enough). But according to https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/ the container needs to be provided with access to And that matches the "setup" arguments: Presumably this is a deal breaker in context of OpenShift? |
|
You need to add /dev/fuse to the container, you can do this with CRI-O via the crio.conf for every container, which is what I am recommending we do by default. I am no Kubernetes expert, but I believe their is now a way to add devices via Kubernetes, which should also work. BTW Great that you are working on this. I have plans to try this out, but am tied up in releasing podman2.0. |
|
I got /dev/fuse added (via hostPath mapping in the Pod), but access to it fails: |
|
Adding to this, if I run the container privileged, it actually does work. But that just wets the appetite for running it without high privileges. Any suggestions appreciated! |
|
Run podman as root not rootless. |
|
BTW, this works fine for me. When running rootless do you have a /dev/fuse on your host? $ ls /dev/fuse -l |
|
Hm, run as root. I assume you mean something like the OpenShift anyuid SCC? If I do that, it fails on just the info command - because podman no longer uses the rootless configuration entries? I see in containers/buildah#867 that you suggest fixing this by mapping in some other drive at But when running as rootless, this was not a problem, and it was using the same drive (at another path, /home/.local/share/containers/storage). The mount map looks like this: We only have NFS-based PVCs on the platform, and I have found earlier that overlay fails on NFS. |
|
Oh, the warning in the output shows it reads another config file when running as root. |
|
I have tried mounting a folder at I am out of ideas of tweaking stuff now :( |
|
Your question about Same as when I run as root. |
|
Regarding missing UIDs and GIDs in the user namespace. You can set Regarding fuse: although we need to find a way to address the issue, a quick workaround could be using the VFS storage driver. That will have some considerable performance impacts though. |
|
I will give VFS a shot. It would be nice with image building on OpenShift that is secure. Performance is not a super critical parameter right now (we are still bringing up the platform). Of course, I do not know how slow it is yet :) The alternative we had discussed was building images on a dedicated box. But we would obviously prefer to keep all workloads on the OpenShift platform. Thanks! |
|
It also fails with VFS. (full log in https://github.com/jskovjyskebankdk/openshift-podman/blob/master/README.md) |
|
@jskovjyskebankdk how did you manage to make /dev/fuse work inside of the container? and: |
|
@danielkucera I am not sure I only saw it working/failing in context of podman execution. And podman was only happy when I ran in a container with privileged: true So I think you see the same problem as I do; it appears /dev/fuse cannot be used in a non-privileged container. (or that is my theory - hopefully Daniel has something to add) |
|
Ok I am trying to run a buildah container within a privileged container non root and I am failing. I am not sure if you start inside of a user namespace that this is possible At this point I modified the /etc/subuild and /etc/subgid inside of the container to use different UID mappings, since my host account was only able to use 65k UIDS. Now I will switch to the build user and attempt to pull an image So far so good, this means this used fuse-overlayfs to pull to a user namespace. Now I will attempt to run a container on it. Using --isolation=chroot I can enter the user namespace and make sure everything is set, then mount the image, and attempt to mount the proc file system. I do not know what is causing permission denied here. Basically I am blocked from mounting a proc file system from inside of the user namespace. |
|
I am unable to even open the file when not privileged:
|
|
@jskovjyskebankdk I was trying to find out how builds triggered by BuildConfig are executed and it turns out that they run a pod with serviceaccount So I presume there is no way how to avoid this when even OpenShift native build mechanism runs as privileged.... |
|
We are under full discussion of this right now. We see two choices, one involves a container running with CAP_SETUID, CAP_SETGID, CAP_SYS_CHROOT and we can get that to run buildah inside of a user namespace as a non root user. Second choice is to get OpenShift/Kubernetes/CRI-O to launch the builder container inside of a user namespace which would be fully locked down from a UID point of view, but might have issues dealing with volumes and secrets, since kubernetes would have to set any content it creates for the container to match the "root" of the container. |
|
I am happy with the CAP_SYS_CHROOT route (easiest for me to work with, and I will need volumes). But I fail to make anything work as suggested. I probably need a little more specific guidance :) What I have done is run the podman-stable image (installing buildah) with SCC anyuid, and it shows CAP_SYS_CHROOT: Seems to be the same problem as with podman, so I am probably missing some (hopefully not too obvious) magic. |
|
This is the current minimal configuration working for me: command: |
|
Yes! Obviously I should not have used fuse but tried VFS again. It does indeed work! Thank you all! I will make a PR with an OpenShift-specific howto for how to set it up. |
|
We believe that you should be able to specify the /dev/fuse device in kubernetes but currently the kublet is only passing in block devices. @nalind is looking into a fix. |
|
I have written this: https://github.com/jskovjyskebankdk/buildah/blob/rootlessBudOpenShift/docs/tutorials/05-openshift-rootless-bud.md @ashokponkumar would you have a look? @rhatdan I have written a tutorial for the buildah project since that is what works right now. |
|
No that sounds good. |
|
I created containers/buildah#2453
Do you have an issue for this so that I can track it? Thanks! |
|
In a pod spec, one possibility would be to mount a kubernetes/kubernetes#79925 attempts to modify the kubelet to add devices to the device cgroup for not- |
|
OK, thanks @nalind |
|
A friendly reminder that this issue had no activity for 30 days. |
|
I can make builds now (using Buildah). But we only have NFS-based storage which makes it really hard to get performant builds (because VFS cannot use NFS). So I am still very much hoping that there will be some way in the future to make use of /dev/fuse. |
|
@jskovjyskebankdk did you try https://github.com/flavio/fuse-device-plugin ? |
|
@elgalu I was not aware of that project, no. And I do not think I will test it - I am a little concerned about its age, and it not being mentioned as a device plugin in the official documentation. But thanks for the reference. |
|
By the way; we now use Podman to make builds in non-privileged containers. Still using NFS backend and VFS, so performance is not great. But it does work (same instructions as for Buildah, https://github.com/containers/buildah/blob/master/docs/tutorials/05-openshift-rootless-bud.md) So I will close this issue. |
github-actions
bot
added
the
locked - please file new issue/PR
/kind bug
Description
Podman is not able to pull images when running in an OpenShift container.
There are elements seen in many other reported issues. I hope to get some help in tracking down what I am missing, so it can be added to the podman documentation proper (I will be happy to help with this).
Steps to reproduce the issue:
Full description of the steps taken - with bog standard images and very little setup - in this repository:
https://github.com/jskovjyskebankdk/openshift-podman
Describe the results you received:
Podman fails image pull with "there might not be enough IDs available in the namespace".
Describe the results you expected:
The image pull should complete :)
Additional information you deem important (e.g. issue happens only occasionally):
This may be a side effect of how OpenShift configures containers, compared to running a container locally in docker/podman.
It must be similar to whatever people face on cloud hosting, but I have not been able to find anything documenting how to get things running.
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Additional environment details (AWS, VirtualBox, physical, etc.):
On-premise OpenShift 4.3.21
The text was updated successfully, but these errors were encountered: