Containers broken after systemctl daemon-reload

[mprasil]

I have a simple container running with docker run --name test --rm alpine sleep 100000, I can exec to it with docker exec -ti test sh and inside the container I can access /dev/null (among other things) as usual.

However after running systemctl daemon-reload, the container ends up being in broken state, where I can't exec into it:

# docker exec -ti alp2 sh
OCI runtime exec failed: exec failed: unable to start container process: open /dev/ptmx: operation not permitted: unknown

And processes inside the container can no longer access /dev/null (and possibly other devices, this one is immediately obvious):

container-shell: # cat /dev/null
cat: can't open '/dev/null': Operation not permitted

That seems to be vaguely related to #3551 however it's only happening with docker inside LXC container and with a version of runc that should have this fixed, so I figured I'll ask for any pointers towards any possible misconfiguration on my side before reporting bug.

When observing the container systemd scope configuration, the container seems to be missing couple DeviceAllow options:

# systemctl cat docker-e5671a0a214b720c55d5d5800b1bbbe4c5fda1373bcd200ea3c5be47be75de44.scope | grep DeviceAllow
# /run/systemd/transient/docker-e5671a0a214b720c55d5d5800b1bbbe4c5fda1373bcd200ea3c5be47be75de44.scope.d/50-DeviceAllow.conf
DeviceAllow=
DeviceAllow=char-pts rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

Here's a comparison with bare metal system using same OS and package versions, that does not have this problem:

# systemctl cat docker-396cac8772b1a22184623b78d43cdd6af823a61d090821ec397f8bfc3419a28a.scope | grep DeviceAllow
# /run/systemd/transient/docker-396cac8772b1a22184623b78d43cdd6af823a61d090821ec397f8bfc3419a28a.scope.d/50-DeviceAllow.conf
DeviceAllow=
DeviceAllow=char-pts rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

My understanding is that these device access settings are applied to the container directly, but are not reflected in the scope configuration, so when systemctl daemon-reload runs, it will apply the configuration it has - effectively breaking the container due to bunch of char devices becoming inaccessible. (like /dev/null or /dev/ptmx above)

The system is using version 1.1.4 of runc, which should have fix for the #3551

# runc --version
runc version 1.1.4-0ubuntu1~22.04.1
spec: 1.0.2-dev
go: go1.18.1
libseccomp: 2.5.3

OS is Ubuntu 22.04 running in privileged LXC container using cgroup v2.

Any ideas what could be causing this?

[mprasil]

I think I have answer to my own question here. As I was typing this out, runc 1.1.5 was released, which contains #3620 - which seems to be exactly the issue I'm observing.

And sure enough, if I upgrade runc to the latest version, the DeviceAllow configuration is added as expected:

# /run/systemd/transient/docker-38b11285c939aa3ab2398321d6230e6b1d79997994a6951f12fff7829e2ea38b.scope.d/50-DeviceAllow.conf
DeviceAllow=
DeviceAllow=char-pts rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

So thanks to @evanphx, this is solved. 👍