Error mounting tmpfs on /var/lib/mock/$chroot-bootstrap/root/proc in docker container

[brianjmurrell]

Discussed in #1095

^{Originally posted by brianjmurrell June 1, 2023}
I am trying to build a docker image for mock building. To speed up it's usage, I want to prime it with the root_cache tarballs.

In my Dockerfile I have added a:

RUN for chroot in {opensuse-leap-15.4,rocky+epel-{8,9}}-x86_64; do \
        su - mockbuilder -c "mock -r \"$chroot\" --shell id";      \
    done;

But when I try to build that image, mock hits an error trying to mount the bootstap /proc:

[2023-06-01T11:44:03.152Z] Step 16/18 : RUN for chroot in {opensuse-leap-15.4,rocky+epel-{8,9}}-x86_64; do         ls -l /var/cache/mock/$chroot/root_cache/;         mock -r "$chroot" --shell id;                                      ls -l /var/cache/mock/$chroot/root_cache/;     done;
[2023-06-01T11:44:03.152Z]  ---> Running in 6c0496186695
[2023-06-01T11:44:03.409Z] ls: cannot access '/var/cache/mock/opensuse-leap-15.4-x86_64/root_cache/': No such file or directory
[2023-06-01T11:44:03.976Z] INFO: mock.py version 3.5 starting (python version = 3.11.3, NVR = mock-3.5-1.fc37)...
[2023-06-01T11:44:03.976Z] Start(bootstrap): init plugins
[2023-06-01T11:44:04.231Z] INFO: selinux disabled
[2023-06-01T11:44:04.231Z] Finish(bootstrap): init plugins
[2023-06-01T11:44:04.231Z] Start: init plugins
[2023-06-01T11:44:04.231Z] INFO: selinux disabled
[2023-06-01T11:44:04.231Z] Finish: init plugins
[2023-06-01T11:44:04.232Z] INFO: Signal handler active
[2023-06-01T11:44:04.232Z] Start: run
[2023-06-01T11:44:04.232Z] ERROR: Namespace unshare failed.
[2023-06-01T11:44:04.232Z] ERROR: It seems we are running inside of Docker. Let skip unsharing.
[2023-06-01T11:44:04.232Z] ERROR: You should *not* run anything but Mock in this container. You have been warned!
[2023-06-01T11:44:09.614Z] Start(bootstrap): chroot init
[2023-06-01T11:44:09.614Z] INFO: calling preinit hooks
[2023-06-01T11:44:09.614Z] INFO: enabled root cache
[2023-06-01T11:44:09.614Z] INFO: enabled package manager cache
[2023-06-01T11:44:09.614Z] Start(bootstrap): cleaning package manager metadata
[2023-06-01T11:44:09.614Z] Finish(bootstrap): cleaning package manager metadata
[2023-06-01T11:44:09.614Z] INFO: enabled HW Info plugin
[2023-06-01T11:44:09.614Z] ERROR: Command failed: 
[2023-06-01T11:44:09.614Z]  # /bin/mount -n -t tmpfs -o rprivate tmpfs /var/lib/mock/opensuse-leap-15.4-x86_64-bootstrap/root/proc

Is what I am trying to do impossible?

[xsuchy]

Please read
https://rpm-software-management.github.io/mock/#mock-inside-podman-fedora-toolbox-or-docker-container and mind that this was updated to reflect 4.x version.

You are running 3.5 version and there were significant changes done in 4.x version. You will save a lot of time if you move to 4.x.

[brianjmurrell]

Please read https://rpm-software-management.github.io/mock/#mock-inside-podman-fedora-toolbox-or-docker-container

I did actually read that before posting here and found nothing that seemed to me to be relevant.

and mind that this was updated to reflect 4.x version.

So that's a problem then because …

You are running 3.5 version

That's right.

and there were significant changes done in 4.x version. You will save a lot of time if you move to 4.x.

I would love to but #1062, caused by rpm-software-management/rpm-sequoia#46 makes using F38, which is where mock 4.x is first available (outside of building my own backport for F37), impossible to use.

[brianjmurrell]

Hrm. Same issue with mock 4.0:

+ su - build -c 'id; mock -r "opensuse-leap-15.4-x86_64" --shell id'
[2023-06-06T21:13:44.470Z] uid=1101(build) gid=1101(build) groups=1101(build),135(mock)
[2023-06-06T21:13:45.033Z] INFO: mock.py version 4.0 starting (python version = 3.11.3, NVR = mock-4.0-1.fc38)...
[2023-06-06T21:13:45.033Z] Start(bootstrap): init plugins
[2023-06-06T21:13:45.033Z] INFO: selinux disabled
[2023-06-06T21:13:45.033Z] Finish(bootstrap): init plugins
[2023-06-06T21:13:45.289Z] Start: init plugins
[2023-06-06T21:13:45.289Z] INFO: selinux disabled
[2023-06-06T21:13:45.289Z] Finish: init plugins
[2023-06-06T21:13:45.289Z] INFO: Signal handler active
[2023-06-06T21:13:45.289Z] Start: run
[2023-06-06T21:13:45.289Z] ERROR: Namespace unshare failed.
[2023-06-06T21:13:45.289Z] ERROR: It seems we are running inside of Docker. Let skip unsharing.
[2023-06-06T21:13:45.289Z] ERROR: You should *not* run anything but Mock in this container. You have been warned!
[2023-06-06T21:13:50.556Z] Start(bootstrap): chroot init
[2023-06-06T21:13:50.556Z] INFO: calling preinit hooks
[2023-06-06T21:13:50.556Z] INFO: enabled root cache
[2023-06-06T21:13:50.556Z] INFO: enabled package manager cache
[2023-06-06T21:13:50.556Z] Start(bootstrap): cleaning package manager metadata
[2023-06-06T21:13:50.556Z] Finish(bootstrap): cleaning package manager metadata
[2023-06-06T21:13:50.556Z] INFO: enabled HW Info plugin
[2023-06-06T21:13:50.881Z] ERROR: Command failed: 
[2023-06-06T21:13:50.881Z]  # /bin/mount -n -t tmpfs -o rprivate tmpfs /var/lib/mock/opensuse-leap-15.4-x86_64-bootstrap/root/proc

[praiskup]

I don't have docker in hand, are you running a --privileged conatianer? For mounting
you need to have CAP_SYS_ADMIN capability.

[brianjmurrell]

I have both of those enabled in fact.

[praiskup]

Weird, can you test the mount command in isolation?

$ mkdir /test
$ /bin/mount -n -t tmpfs -o rprivate tmpfs /test

.. and diagnose why it is failing?

[brianjmurrell]

Should I be doing that test within su - build in the same way that mock is run under that uid?

[praiskup]

Mock switches to root on demand, per /etc/security/console.apps/mock. So please try as root, if it fails you likely miss some capabilities. If it works - the question is why mock can not switch to root using consolehelper.

[brianjmurrell]

...
[2023-06-12T16:58:36.032Z] ERROR: Namespace unshare failed.
[2023-06-12T16:58:36.032Z] ERROR: It seems we are running inside of Docker. Let skip unsharing.
[2023-06-12T16:58:36.033Z] ERROR: You should *not* run anything but Mock in this container. You have been warned!
[2023-06-12T16:58:41.281Z] Start(bootstrap): chroot init
[2023-06-12T16:58:41.282Z] INFO: calling preinit hooks
[2023-06-12T16:58:41.282Z] INFO: enabled root cache
[2023-06-12T16:58:41.282Z] INFO: enabled package manager cache
[2023-06-12T16:58:41.282Z] Start(bootstrap): cleaning package manager metadata
[2023-06-12T16:58:41.282Z] Finish(bootstrap): cleaning package manager metadata
[2023-06-12T16:58:41.282Z] INFO: enabled HW Info plugin
[2023-06-12T16:58:41.282Z] ERROR: Command failed: 
[2023-06-12T16:58:41.282Z]  # /bin/mount -n -t tmpfs -o rprivate tmpfs /var/lib/mock/rocky+epel-8-x86_64-bootstrap/root/proc
[2023-06-12T16:58:41.282Z] 
[2023-06-12T16:58:41.549Z] + mkdir -p /test
[2023-06-12T16:58:41.549Z] + /bin/mount -n -t tmpfs -o rprivate tmpfs /test
[2023-06-12T16:58:41.549Z] mount: /test: permission denied.
[2023-06-12T16:58:41.549Z]        dmesg(1) may have more information after failed mount system call.
[2023-06-12T16:58:41.549Z] + echo 'Failed to mount onto /test'
[2023-06-12T16:58:41.549Z] + dmesg
[2023-06-12T16:58:41.549Z] Failed to mount onto /test
[2023-06-12T16:58:41.549Z] dmesg: read kernel buffer failed: Operation not permitted

But unfortunately, as you can see, even root in the Docker container can't read dmesg so that we can see if there is any more useful information.

[brianjmurrell]

if it fails you likely miss some capabilities

Aren't Docker's capabilities a runtime (i.e. docker run) concept?

In case it was not clear, I am having the problem of running mock during the image build docker build, not docker run. To refresh, I want to build and store the root_cache tarballs in the created docker image to speed up the (very repeated) run-time usage of the images/containers.

[brianjmurrell]

Any more ideas here?

[praiskup]

Can you perhaps check with getpcaps 1? Dunno. Podman build e.g. seems to support the --cap-add=CAP_xxx option. For me it is harder to experiment with Docker (switched years ago, and have zero motivation to go back to when Podman works, last time I tried it was not properly working on Fedora with cgroups v2).

[praiskup]

I want to build and store the root_cache tarballs in the created docker image to speed up the (very repeated) run-time usage of the images/containers.

This is interesting, I'm curious if you could dump the tarball out from podman run first, and then bind-mount it for the other runs.

[brianjmurrell]

+ getpcaps 1
1: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep

This is interesting, I'm curious if you could dump the tarball out from podman run first, and then bind-mount it for the other runs.

That would mean each workspace would need to create it's own tarballs and we could have hundreds of workspaces. That's why I want to bake the tarball right into the image (which we rebuild weekly, just to keep the mock host updated).

[praiskup]

Can you try docker build --cap-add=CAP_SYS_ADMIN or something like that?

[praiskup]

I was able to run mock at container build time using:
$ podman build --security-opt=label=disable --cap-add=CAP_SYS_ADMIN . -t mock