distrobox (podman containers) can fail to run after an rpm-ostree upgrade

[bsherman]

What happened?

Multiple users have reported in discord that after doing an rpm-ostree upgrade (or rebase), they reboot, and then their distrobox cannot be entered.

For me it looked like:

$ distrobox enter fedora
Container fedora is not running.
Starting container fedora
run this command to follow along:

 podman logs -f fedora

Error: unable to start container "79ac894fa239b849a92715a0b196861f645d1351eb5722af135740a006dc6aa1": crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

"@scotty" reported: "I just upgraded to 38.20230509.0 and I amgetting this now. No rebase, just rpm-ostree upgrade."

Error: unable to start container "<container>": crun: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI permission denied

https://discord.com/channels/1072614816579063828/1074422586894712912/1105698916797792308

Relevant log output

$ rpm-ostree ex history
NOTICE: Experimental commands are subject to change.
BootTimestamp: 2023-05-09T20:12:37Z (7h ago)
CreateTimestamp: 2023-05-10T00:29:58Z (3h 13min ago)

  ostree-unverified-image:docker://ghcr.io/ublue-os/silverblue-main:38
                   Digest: sha256:974e6c2dc125ec79986b3d52afe91ab1370cb97582b9c8802554e35a22237561
                  Version: 38.20230509.0 (2023-05-10T00:28:31Z)
          LayeredPackages: langpacks-en pass

BootTimestamp: 2023-05-09T17:21:30Z (10h ago)
CreateTimestamp: 2023-04-29T01:24:16Z (1 week 4 days ago)
CreateCommand: upgrade --trigger-automatic-update-policy

  ostree-unverified-image:docker://ghcr.io/ublue-os/silverblue-main:38
                   Digest: sha256:f0636f5a97d4133fc90278dc6f0608a209afd6a235971c4765fc0eddc9f1fbec
                  Version: 38.20230428.0 (2023-04-29T01:23:40Z)
          LayeredPackages: langpacks-en pass

BootTimestamp: 2023-04-26T17:20:05Z (1 week 6 days ago)
CreateTimestamp: 2023-04-27T00:19:36Z (1 week 6 days ago)
CreateCommand: install pass

  ostree-unverified-image:docker://ghcr.io/ublue-os/silverblue-main:38
                   Digest: sha256:4bc0858e0f37bf40057d707bf61d5b06189148ab93a26fca0a1176a9ccdd9a9a
                  Version: 38.20230421.0 (2023-04-27T00:09:35Z)
          LayeredPackages: langpacks-en pass

[baconicsynergy]

I joined the discussion on discord and have been following along with the conversation since this happened to me. Im up north and away from my ublue machine, but once im back home im willing to run some experiments and trials to help out as much as i can

[bsherman]

I believe this is the issue: containers/podman#18555

[bsherman]

I believe this is the issue: containers/podman#18555

based on this issue, I imagine that we started seeing this problem intermittently after Fedora upgraded to podman 4.5.0

[Cydox]

The other issue is related to using kube, however a very similar issue happens without using kube. Upstream issue:
containers/podman#18714

[AshleySchaeffer]

It seems that exporting the container to an image and reimporting it to a new container allows you to fix existing containers without having to nuke them entirely. This is only useful for those that don't have a reproducible build process (e.g. you're using toolbox/distrobox and you've manually installed packages).

Distrobox provides instructions on how to do this here:

https://github.com/89luca89/distrobox/blob/main/docs/useful_tips.md#container-save-and-restore

I'm not sure how you'd approach this with toolbox unfortunately.

[quazar-omega]

I think you should be able to do the exact same thing, you just snapshot the container into an image an then you use that as the origin image for a new toolbox container