rawhide, rootless: podman is defaulting to runc (and failing)

[edsantiago]

Brand-new rawhide virt, brand-new user, brand-new first-time invocation of podman:

$ podman run alpine date
ERRO[0003] oci runtime "runc" does not support CGroups V2: use system migrate to mitigate
Error: this version of runc doesn't work on cgroups v2: OCI runtime error

My hunch is that the new config code is not writing a libpod.conf, and then it's not picking up the proper crun default:

$ ls -l ~/.config/containers/
total 4
-rw-rw-r--. 1 fedora fedora 724 Nov  6 13:48 storage.conf

$ grep '^runtime =' /usr/share/containers/libpod.conf
runtime = "crun"

I can't actually get it to work even if I manually create a libpod.conf:

$ grep -1 --no-group-separator crun /usr/share/containers/libpod.conf  | tee ~/.config/containers/libpod.conf
# Default OCI runtime
runtime = "crun"

# libpod will use it for reporting nicer errors.
runtime_supports_json = ["crun", "runc"]

# creation of CGroups for containers.
runtime_supports_nocgroups = ["crun"]


crun = [
            "/usr/bin/crun",
            "/usr/local/bin/crun",
]
$ podman run alpine date
Error: could not get runtime: default OCI runtime "crun" not found: invalid argument

I'm getting into diminishing-returns territory. Please ping me for more info or if there's something obvious I haven't tried. (Before you ask: yes, /usr/bin/crun exists and is 755 and untouched by me).

podman-1.6.4-0.7.dev.git581a7ec.fc32.x86_64 but also happening in earlier builds; I kind of suspect that this started with #4352 but it was being masked by the much-louder issue #4456

[mheon]

I think this is an unanticipated consequence of our reverting the Fedora-specific CGroups v2 patches for the 1.6.3 stable release for RHEL.

[edsantiago]

But root works?

# podman info  --format '{{ .host.OCIRuntime.name }}'
crun
# podman run alpine date
Wed Nov  6 19:15:51 UTC 2019

[mheon]

Hm.

@vrothberg Could this be config file merging getting us?

[mheon]

I'm sort of suspecting that the default compiled-in configuration is overriding the default libpod.conf setting? Somehow?

[vrothberg]

Rootless is not looking at the system-root config at /etc/containers/libpod.conf and the hard-coded default config always points to runc. We decided to remove forcing crun on cgroups v2 nodes and I guess that's the unfortunate consequence.

What we could do is to make the runtime of the hard-coded default config use crun on cgroups v2 nodes. That's way less brutal than forcing crun unconditionally and over-writing a config. My eyes are too tired to follow that thought, but maybe @mheon can tackle it? The code in question is https://github.com/containers/libpod/blob/master/libpod/config/default.go#L49. Setting it to crun will fix this specific issue in the gating tests.

[rhatdan]

I think we should read the system wide config and then read the local config. That way we would see the default as crun.

[vrothberg]

I think we should read the system wide config and then read the local config.

We decided against doing that in the design doc to have root be root and rootless be rootless. If a user wants specific setting, they can copy and edit the libpod.conf.

That way we would see the default as crun.

Only if there is an /etc/containers/libpod.conf. If not, we run into very same problem. That's why I think we should default to crun in the hard-coded default config. Once runc is working, we default to it again (and maybe do a version check to make sure it's not an old one).

[vrothberg]

@rhatdan @edsantiago PTAL at #4470