"ro" in /dev mount options in rc93 and later

[djs55]

While updating https://github.com/linuxkit/linuxkit to a more modern runc, I noticed that some containers would fail to start.

I tracked this down to linuxkit adding "ro" to the mount options for /dev e.g.

   "mounts": [
        {
            "destination": "/dev",
            "type": "tmpfs",
            "source": "tmpfs",
            "options": [
                "nosuid",
                "strictatime",
                "mode=755",
                "size=65536k",
                "ro"
            ]
        },

I think this works in rc92 and earlier, but now fails with an error trying to write to /dev.

In linuxkit I saw

linuxkit-0e465ebdb34b:~# runc create --bundle /containers/onboot/000-binfmt foobar
[   27.379103] IPVS: ftp: loaded support on port[0] = 21
ERRO[0000] container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:88: creating device nodes caused: read-only file system 

I can reproduce the same type of error (slightly different message) in 1.0.2 from the runc spec --help example by adding the "ro" argument:

$ diff -u config.json.orig config.json
--- config.json.orig	2021-10-19 07:55:59.037628359 +0100
+++ config.json	2021-10-19 07:55:43.337924319 +0100
@@ -69,7 +69,7 @@
 				"nosuid",
 				"strictatime",
 				"mode=755",
-				"size=65536k"
+				"size=65536k", "ro"
 			]
 		},
 		{

$ sudo runc run foobar
[sudo] password for dave: 
ERRO[0000] container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "devpts" to rootfs at "/dev/pts" caused: mkdir /tmp/runc/rootfs/dev/pts: read-only file system 

$ runc --version
runc version 1.0.2
commit: v1.0.2-0-g52b36a2
spec: 1.0.2-dev
go: go1.16.8
libseccomp: 2.5.1

If I remove the "ro" mount option it works correctly.

I propose to remove the "ro" mount option from linuxkit to fix the issue, see linuxkit/linuxkit#3718 (comment)

The containers involved all have

        "root": {
                "path": "rootfs",
                "readonly": true
        },

which is perhaps enough "readonly-ness".

Was it a mistake for us to add "ro" in the first place?

[thaJeztah]

@kolyshkin @cyphar any idea what changed in this area? As mentioned in linuxkit/linuxkit#3718 (comment), the goal in linuxkit was to have a have a read-only filesystem; previously, runc was able to set this up before making the filesystem readonly (did the order of processing change perhaps?)

[kolyshkin]

There are a few filesystems that are mounted under /dev, and a few devices that need to be created.

So we mount /dev first and then do all that (other mounts under /dev, and device nodes). Obviously, this fails if /dev is read-only. A workaround would be to mount it read-write, do all we need to do, then remount ro.

I think this works in rc92 and earlier, but now fails with an error trying to write to /dev.

Indeed it works for rc91 and rc92, but do not work with rc93.

I have git bisected that to commit fb4c27c.