LXD 5.0 fails to build rocks on Ubuntu 22.04 #195

twovican · 2023-02-15T18:00:34Z

When using LXD 5.0 on 22.04, rockcraft pack will fail to build. Throwing the following error:

2023-02-14 17:59:07.565 :: 2023-02-14 23:59:06.997 fuse-overlayfs mountpoint='/root/overlay/overlay', args=('-olowerdir=/root/bundles/ubuntu-20.04/rootfs,upperdir=/root/overlay/packages,workdir=/root/overlay/work',) 2023-02-14 17:59:07.565 :: 2023-02-14 23:59:07.020 Failed to mount overlay on /root/overlay/overlay: Command '['fuse-overlayfs', '-olowerdir=/root/bundles/ubuntu-20.04/rootfs,upperdir=/root/overlay/packages,workdir=/root/overlay/work', '/root/overlay/overlay']' returned non-zero exit status 1.

The current workaround is to upgrade LXD to 5.9 stable.

Run this:
sudo snap install lxd --channel=5.9/stable

Instead of this:
sudo snap install lxd --channel=5.0/stable

The text was updated successfully, but these errors were encountered:

gboutry · 2023-03-06T13:59:50Z

I have the same errors using lxd 5.11/stable

2023-03-06 14:54:30.168 :: 2023-03-06 13:54:29.785 fuse-overlayfs mountpoint='/root/overlay/overlay', args=('-olowerdir=/root/bundles/ubuntu-20.04/rootfs,upperdir=/root/overlay/packages,workdir=/root/overlay/work',)
2023-03-06 14:54:30.168 :: 2023-03-06 13:54:29.817 Failed to mount overlay on /root/overlay/overlay: Command '['fuse-overlayfs', '-olowerdir=/root/bundles/ubuntu-20.04/rootfs,upperdir=/root/overlay/packages,workdir=/root/overlay/work', '/root/overlay/overlay']' returned non-zero exit status 1.

Only when using overlay instructions such as overlay-packages

sergiusens · 2023-03-07T00:02:36Z

@tomponline can you help out with this?

tomponline · 2023-03-07T06:42:22Z

@sergiusens do you have reproducer steps (ideally not requiring rockcraft and just the problem commands it is calling, or if not then for someone who is not familiar with rockcraft?) Thanks

gboutry · 2023-03-07T08:31:19Z

@sergiusens do you have reproducer steps (ideally not requiring rockcraft and just the problem commands it is calling, or if not then for someone who is not familiar with rockcraft?) Thanks

I can't really give you steps without rockcraft, but with rockcraft:
In an empty dir, add rockcraft.yaml:

name: my-rock-name
base: ubuntu:22.04
version: '0.1'
summary: summary
description: description
license: Apache-2.0
platforms:
    amd64:

parts:
    my-part:
        plugin: nil
        overlay-packages:
          - software-properties-common

Then, run rockcraft -v pack.

I ran this commands using:

rockcraft installed: 0+git.63135d7 (633) 83MB classic
lxd installed: 5.11-ad0b61e (24483) 149MB

tomponline · 2023-03-07T08:32:44Z

Thanks!

Does it work with LXD 5.9/5.10? I'm trying to understand if this is a regression or a new bug? Thanks

tomponline · 2023-03-07T08:35:33Z

Also can you show me the lxc config show <instance> --expanded for the container rockcraft is using please?

tomponline · 2023-03-07T08:35:53Z

And the output of uname -a please.

gboutry · 2023-03-07T08:41:11Z

It also fails with lxd 5.10, did not try with 5.9 and I can't install it anymore.

Output of lxc --project rockcraft config show rockcraft-my-rock-name-258206 --expanded:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (buildd release) (20220426)
  image.label: buildd release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20220426"
  image.type: tar.gz
  image.version: "22.04"
  raw.idmap: both 1000 0
  security.syscalls.intercept.mknod: "true"
  volatile.base_image: 5de93bcc5cddf2ede3e74f88c313deb056eab006dd131618a5d412dc4a5e5534
  volatile.cloud-init.instance-id: 14e6eade-1ec8-4052-9d23-6261ea469bfd
  volatile.eth0.hwaddr: 00:16:3e:34:df:d2
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000001,"Nsid":1,"Maprange":999999999},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000001,"Nsid":1,"Maprange":999999999}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000001,"Nsid":1,"Maprange":999999999},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000001,"Nsid":1,"Maprange":999999999}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: a22997db-4890-44f1-a59e-7e50273972c4
devices:
  eth0:
    name: eth0
    network: lxdbr1
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Output of uname -a (reproduced the behavior on both):

Linux gboutry-bastion 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux <-- this is a vm
Linux infinity-book 6.1.0-1009-tuxedo #11 SMP PREEMPT_DYNAMIC Tue Feb 21 19:40:50 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux <-- this is my laptop

Running the command manually works:

root@rockcraft-my-rock-name-258206:~# /snap/rockcraft/633/usr/bin/fuse-overlayfs -olowerdir=/root/bundles/ubuntu-22.04/rootfs,upperdir=/root/overlay/packages,workdir=/root/overlay/work /root/overlay/overlay
root@rockcraft-my-rock-name-258206:~# mount | grep /root/overlay/overlay
fuse-overlayfs on /root/overlay/overlay type fuse.fuse-overlayfs (rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)

tomponline · 2023-03-07T08:47:52Z

Thanks. I'm wondering if its to do with this:

https://discuss.linuxcontainers.org/t/linux-6-1-lts-confirmed-but-no-fix-for-mknod-yet/16436

Although the kernel versions don't match.

Although:

As far as I can see this problem with interception is present on 5.15 kernel too.

https://discuss.linuxcontainers.org/t/linux-6-1-lts-confirmed-but-no-fix-for-mknod-yet/16436/15?u=tomp

tomponline · 2023-03-07T08:48:40Z

Do you think this is the same issue as https://discuss.linuxcontainers.org/t/linux-6-1-lts-confirmed-but-no-fix-for-mknod-yet/16436 @mihalicyn ?

mihalicyn · 2023-03-07T16:48:56Z

Hello everyone,

I don't think that it's related to mknod intercept, because here we are failing on the mount stage. And here we have overlayfs-fuse, but there we have in-kernel overlayfs.
I think it makes sense to patch this place https://github.com/canonical/craft-parts/blob/62c65491a55a2d6b59979d27b40bd325744ac5a9/craft_parts/utils/os_utils.py#L249 and replace check_call to check_output or something else (I'm not a python expert), that allows to catch stdout/stderr. Otherwise we can only try to guess what's happening here. But having more verbose logs can help to quickly determine where is the problem.

tomponline · 2023-03-07T21:10:43Z

Thanks!

cmatsuoka · 2023-03-07T21:11:38Z

I've played a bit with fuse-overlayfs in a plain jammy container and in the container created by Rockcraft, and the results are different.

If I just launch a jammy container with lxc 5.11 and run fuse-overlayfs, it works:

root@test-overlayfs:~# mkdir upper lower work
root@test-overlayfs:~# fuse-overlayfs -olowerdir=lower,upperdir=upper,workdir=work /mnt
root@test-overlayfs:~# echo $?
0

However, if I enter the rockcraft container (rockcraft pull --shell), create the directories and run the same command, it fails:

root@rockcraft-my-rock-name-2378573:~/project# fuse-overlayfs -olowerdir=lower,upperdir=upper,workdir=work /mnt
fuse: mount failed: Invalid argument
fuse-overlayfs: cannot mount: Invalid argument

Running it through strace shows:

root@rockcraft-my-rock-name-2378573:~/project# strace fuse-overlayfs -f -olowerdir=lower,upperdir=upper,workdir=work /mnt
execve("/snap/rockcraft/638/usr/bin/fuse-overlayfs", ["fuse-overlayfs", "-f", "-olowerdir=lower,upperdir=upper,"..., "/mnt"], 0x7ffc37889f58 /* 27 vars */) = 0
...
stat("/mnt", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
openat(AT_FDCWD, "/dev/fuse", O_RDWR|O_CLOEXEC) = 6
getgid()                                = 0
getuid()                                = 0
mount("fuse-overlayfs", "/mnt", "fuse.fuse-overlayfs", MS_NODEV|MS_NOATIME, "default_permissions,allow_other,"...) = -1 EINVAL (Invalid argument)

@mr-cal, what are the container creation parameters currently being used in Rockcraft?

tomponline · 2023-03-07T21:14:08Z

Can we see lxc config show <instance> --expanded for each one please?

cmatsuoka · 2023-03-07T21:29:29Z

Here they are:

Plain container

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (release) (20230302)
  image.label: release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20230302"
  image.type: squashfs
  image.version: "22.04"
  raw.lxc: |
    lxc.mount.entry = tmpfs tmp tmpfs rw,nosuid,nodev,size=2G
    lxc.mount.entry = tmpfs var/tmp tmpfs rw,nosuid,nodev,size=2G
  volatile.base_image: 72565f3fbae414d317b90569b6d7aa308c482fdf562aaf0c2eaa6e50fa39747b
  volatile.cloud-init.instance-id: 0b1a09bb-f746-4812-8344-07a5955c32c0
  volatile.eth0.host_name: vethbb897c4f
  volatile.eth0.hwaddr: 00:16:3e:a5:c9:dd
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 064a8457-c7d5-4ed5-9536-e8562cc40452
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: fast
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Rockcraft container

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (buildd release) (20220426)
  image.label: buildd release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20220426"
  image.type: tar.gz
  image.version: "22.04"
  raw.idmap: both 1000 0
  raw.lxc: |
    lxc.mount.entry = tmpfs tmp tmpfs rw,nosuid,nodev,size=2G
    lxc.mount.entry = tmpfs var/tmp tmpfs rw,nosuid,nodev,size=2G
  security.syscalls.intercept.mknod: "true"
  volatile.base_image: 5de93bcc5cddf2ede3e74f88c313deb056eab006dd131618a5d412dc4a5e5534
  volatile.cloud-init.instance-id: fd5f9784-b2cc-4785-9777-1cd31dc93919
  volatile.eth0.hwaddr: 00:16:3e:e2:63:e9
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000001,"Nsid":1,"Maprange":999999999},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000001,"Nsid":1,"Maprange":999999999}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000001,"Nsid":1,"Maprange":999999999},{"Isuid":true,"Isgid":true,"Hostid":1000,"Nsid":0,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000001,"Nsid":1,"Maprange":999999999}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 91c32aa6-e09f-4e90-92ee-bcb4eb7d1e40
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: fast
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

tomponline · 2023-03-07T21:33:28Z

Looks like there is some custom Id mapping going on which could be worth investigating. Also are you using the same base image? The configs suggest not.

cmatsuoka · 2023-03-07T21:41:47Z

It's not the same base image, but Id mapping is likely the cause given that kernel overlayfs doesn't work with idmapped mounts. Has something been changed in lxd 5 to enforce the same behavior?

tomponline · 2023-03-07T21:44:52Z

Idmapped mounts is the default if possible (not supported on zfs, cephfs or tmpfs currently AFAIK).

tomponline · 2023-03-07T21:45:56Z

Could you try it in a zfs storage pool and see if that works, that would prove idmapped mounts is the issue.

tomponline · 2023-03-07T21:46:34Z

Its not changed recently to my knowledge though.

cmatsuoka · 2023-03-07T21:55:15Z

I set raw.idmap to both 1000 0 in the test container (this seems to be what Rockcraft does when launching its container) and it's still working. I'll investigate this further tomorrow.

mihalicyn · 2023-03-07T22:37:48Z

@cmatsuoka please, show stat /dev/fuse inside the rockcraft-my-rock-name-2378573 container, where the problem is reproducible.

This -EINVAL from the kernel most probably means that provided /dev/fuse fd is not fuse char device fd, but something else...

gboutry · 2023-03-08T08:57:16Z

Could you try it in a zfs storage pool and see if that works, that would prove idmapped mounts is the issue.

I use a ZFS storage pool

Result of lxc storage list --format yaml | yq 'del(.[].used_by)'

- config:
    size: 500GiB
    source: /var/snap/lxd/common/lxd/disks/lxd.img
    zfs.pool_name: lxd
  description: ""
  name: lxd
  driver: zfs
  status: Created
  locations:
    - none

tomponline · 2023-03-08T09:13:37Z

OK so its not idmapped mounts then thats causing the issue.

cmatsuoka · 2023-03-08T11:32:40Z

@mihalicyn is correct, /dev/fuse and all other files in /dev are not device nodes and have no permissions set:

  File: /dev/fuse
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: f6h/246d        Inode: 3           Links: 1
Access: (0000/----------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-08 11:23:56.304300204 +0000
Modify: 2023-03-08 11:23:56.304300204 +0000
Change: 2023-03-08 11:23:56.304300204 +0000
 Birth: -

root@rockcraft-my-rock-name-2378573:~/project# ls -l /dev
total 0
---------- 1 root root  0 Mar  8 11:23 console
lrwxrwxrwx 1 root root 11 Mar  8 11:23 core -> /proc/kcore
lrwxrwxrwx 1 root root 13 Mar  8 11:23 fd -> /proc/self/fd
---------- 1 root root  0 Mar  8 11:23 full
---------- 1 root root  0 Mar  8 11:23 fuse
lrwxrwxrwx 1 root root 12 Mar  8 11:23 initctl -> /run/initctl
lrwxrwxrwx 1 root root 28 Mar  8 11:23 log -> /run/systemd/journal/dev-log
drwxr-xr-x 2 root root 40 Mar  8 11:23 lxd
drwxr-xr-x 2 root root 40 Mar  8 11:23 mqueue
drwxr-xr-x 2 root root 60 Mar  8 11:23 net
---------- 1 root root  0 Mar  8 11:23 null
---------- 1 root root  0 Mar  8 11:23 ptmx
drw-r--r-- 2 root root 40 Mar  8 11:23 pts
---------- 1 root root  0 Mar  8 11:23 random
drwxr-xr-x 2 root root 40 Mar  8 11:23 shm
lrwxrwxrwx 1 root root 15 Mar  8 11:23 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Mar  8 11:23 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Mar  8 11:23 stdout -> /proc/self/fd/1
---------- 1 root root  0 Mar  8 11:23 tty
---------- 1 root root  0 Mar  8 11:23 urandom
---------- 1 root root  0 Mar  8 11:23 zero

In contrast the test container has device nodes:

root@test-overlayfs:~# ls -l /dev
total 0
crw--w---- 1 root   tty     136,   0 Mar  8 11:28 console
lrwxrwxrwx 1 root   root          11 Mar  8 11:28 core -> /proc/kcore
lrwxrwxrwx 1 root   root          13 Mar  8 11:28 fd -> /proc/self/fd
crw-rw-rw- 1 nobody nogroup   1,   7 Feb 27 12:30 full
crw-rw-rw- 1 nobody nogroup  10, 229 Mar  8 09:35 fuse
lrwxrwxrwx 1 root   root          12 Mar  8 11:28 initctl -> /run/initctl
lrwxrwxrwx 1 root   root          28 Mar  8 11:28 log -> /run/systemd/journal/dev-log
drwxr-xr-x 2 nobody nogroup       60 Feb 23 00:11 lxd
drwxrwxrwt 2 nobody nogroup       40 Jan  5 18:22 mqueue
drwxr-xr-x 2 root   root          60 Mar  8 11:28 net
crw-rw-rw- 1 nobody nogroup   1,   3 Feb 27 12:30 null
crw-rw-rw- 1 root   root      5,   2 Mar  8 11:28 ptmx
drwxr-xr-x 2 root   root           0 Mar  8 11:28 pts
crw-rw-rw- 1 nobody nogroup   1,   8 Feb 27 12:30 random
drwxrwxrwt 2 root   root          40 Mar  8 11:28 shm
lrwxrwxrwx 1 root   root          15 Mar  8 11:28 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root   root          15 Mar  8 11:28 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root   root          15 Mar  8 11:28 stdout -> /proc/self/fd/1
crw-rw-rw- 1 nobody nogroup   5,   0 Mar  7 21:02 tty
crw-rw-rw- 1 nobody nogroup   1,   9 Feb 27 12:30 urandom
crw-rw-rw- 1 nobody nogroup   1,   5 Feb 27 12:30 zero

cmatsuoka · 2023-03-08T11:48:22Z

This is running on 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64.

mihalicyn · 2023-03-08T12:18:23Z

The only suspicious difference between this containers is the image format. In the "good" case we see squashfs, but in the "bad" we see tar.gz. I think that something terribly wrong during tar.gz unpacking, probably tar isn't running as root, so it doesn't capable to call mknod and fails to create device nodes.

mr-cal · 2023-03-08T14:50:30Z

Hello!

Does it work with LXD 5.9/5.10? I'm trying to understand if this is a regression or a new bug? Thanks

The failure does not occur with LXD 5.9. I've reproduced it with 5.0, 5.10, and 5.11.

In the rockcraft spread tests, we install LXD 5.9 using a bug in the snap store:

snap install lxd --channel=latest/stable
snap refresh lxd --channel=5.9/stable

@mr-cal, what are the container creation parameters currently being used in Rockcraft?

We set two parameters:

raw.idmap = both <current-user-uid> 0
security.syscalls.intercept.mknod = true

In the spread tests, we run lxd init --auto during setup, so nothing out of the ordinary during initialization.

tomponline · 2023-03-08T15:03:07Z

Thanks!

Are you able to explain the difference in the base image being used from #195 (comment) ?

tomponline · 2023-03-08T15:03:57Z

Because if we can identify the cause of that we may be able to isolate to that based on #195 (comment) comment.

mr-cal · 2023-03-08T16:58:23Z

Are you able to explain the difference in the base image being used from #195 (comment) ?

Rockcraft uses the jammy/core22 image from https://cloud-images.ubuntu.com/buildd/releases/, (the tar.gz image).

Claudio created the other container from https://cloud-images.ubuntu.com/releases (the squashfs image).

I rebuilt rockcraft + craft-providers and tried both images. Using LXD 5.11, I can reproduce the original error using either image.

I think Claudio is going to manually create an instance from the same buildd image that Rockcraft uses and check the device nodes.

tomponline · 2023-03-08T18:05:45Z

Sounds good thanks

mr-cal · 2023-03-08T18:09:31Z

@cmatsuoka, I can't reproduce your example with missing device nodes with rockcraft.

Using LXD 5.11 and rockcraft latest/edge, I see device nodes as expected:

root@rockcraft-my-rock-name-27145908:~# ls -l /dev
total 0
crw--w---- 1 root   tty     136,   0 Mar  8 18:07 console
lrwxrwxrwx 1 root   root          11 Mar  8 18:07 core -> /proc/kcore
lrwxrwxrwx 1 root   root          13 Mar  8 18:07 fd -> /proc/self/fd
crw-rw-rw- 1 nobody nogroup   1,   7 Mar  8 14:14 full
crw-rw-rw- 1 nobody nogroup  10, 229 Mar  8 17:54 fuse
lrwxrwxrwx 1 root   root          12 Mar  8 18:07 initctl -> /run/initctl
lrwxrwxrwx 1 root   root          28 Mar  8 18:07 log -> /run/systemd/journal/dev-log
drwxr-xr-x 2 nobody nogroup       60 Mar  8 15:14 lxd
drwxrwxrwt 2 nobody nogroup       40 Mar  8 14:14 mqueue
drwxr-xr-x 2 root   root          60 Mar  8 18:07 net
crw-rw-rw- 1 nobody nogroup   1,   3 Mar  8 14:14 null
crw-rw-rw- 1 root   root      5,   2 Mar  8 18:07 ptmx
drwxr-xr-x 2 root   root           0 Mar  8 18:07 pts
crw-rw-rw- 1 nobody nogroup   1,   8 Mar  8 14:14 random
drwxrwxrwt 2 root   root          40 Mar  8 18:07 shm
lrwxrwxrwx 1 root   root          15 Mar  8 18:07 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root   root          15 Mar  8 18:07 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root   root          15 Mar  8 18:07 stdout -> /proc/self/fd/1
crw-rw-rw- 1 nobody nogroup   5,   0 Mar  8 18:04 tty
crw-rw-rw- 1 nobody nogroup   1,   9 Mar  8 14:14 urandom
crw-rw-rw- 1 nobody nogroup   1,   5 Mar  8 14:14 zero

cmatsuoka · 2023-03-08T18:39:44Z

This is interesting. Have you entered the instance using rockcraft pull --shell? What kernel version are you using?

cmatsuoka · 2023-03-08T23:18:27Z

So here's the simplest way to reproduce the problem:

Launch a container
Inside the container, rbind-mount /dev
Lazy-unmount it
See what happened to /dev entries

This is what happens here:

$ lxc launch ubuntu:22.04 test-container
Creating test-container
Starting test-container
$ lxc exec test-container bash
root@test-container:~# ls -l /dev/fuse
crw-rw-rw- 1 nobody nogroup 10, 229 Mar  8 11:44 /dev/fuse
root@test-container:~# mount --rbind /dev /mnt
root@test-container:~# umount --lazy /mnt
root@test-container:~# ls -l /dev/fuse
---------- 1 root root 0 Mar  8 23:02 /dev/fuse

Rockcraft rbind-mounts /dev and chroots into the filesystem it's creating to install packages. After /dev in the new root is unmounted, the external /dev becomes unusable and that causes fuse-overlayfs to fail. Apparently this started to happen in lxd 5 and worked correctly with previous versions (and, for some reason, lxd 5.9 also seems to work).

mihalicyn · 2023-03-09T08:55:15Z

I think it's related to lxc/lxc#4229 change.

Container rootfs became shared mount, it means that when you doing umount, this unmount operation propagates through shared group and original mounts are unmounted too.

I can suggest to use mount --make-rprivate /mnt after mount --rbind /dev /mnt to prevent umount from affecting to the original /dev subtree.

tomponline · 2023-03-09T09:03:03Z

Thanks @mihalicyn I was thinking last night that this might be something to do with the changes that @brauner added for the mount tunnel and was going to dig out this same PR in liblxc.

Address shared mount issues affecting /dev mount in chroots. This is a result of lxc/lxc#4229 (container rootfs became a shared mount, meaning that unmounts propagates through the shared group and original mounts are unmounted too). See canonical/rockcraft#195 for details. Signed-off-by: Claudio Matsuoka <[email protected]>

cmatsuoka · 2023-03-09T13:04:49Z

This seems to have fixed the issue, thank you. I'll update craft-parts and close this issue when the fix lands on Rockcraft.

cmatsuoka · 2023-03-09T13:12:30Z

Result of running Rockcraft after changing the mount to rprivate:

$ rockcraft pack
Launching instance...                                                                                                                                                                 
Retrieved base ubuntu:22.04 for amd64                                                                                                                                                 
Extracted ubuntu:22.04                                                                                                                                                                
Executed: pull my-part                                                                                                                                                                
Executed: pull pebble                                                                                                                                                                 
Executed: overlay my-part                                                                                                                                                             
Executed: overlay pebble                                                                                                                                                              
Executed: build my-part                                                                                                                                                               
Executed: build pebble                                                                                                                                                                
Executed: stage my-part                                                                                                                                                               
Executed: stage pebble                                                                                                                                                                
Executed: prime my-part                                                                                                                                                               
Executed: prime pebble                                                                                                                                                                
Executed parts lifecycle                                                                                                                                                              
Created new layer                                                                                                                                                                     
Labels and annotations set to ['org.opencontainers.image.version=0.1', 'org.opencontainers.image.title=my-rock-name', 'org.opencontainers.image.ref.name=my-rock-name', 'org.opencontainers.image.licenses=Apache-2.0', 'org.opencontainers.image.created=2023-03-09T13:09:57.480643+00:00', 'org.opencontainers.image.base.digest=2adf22367284330af9f832ffefb717c78239f6251d9d0f58de50b86229ed1427']                                                                                                                                                               
Exported to OCI archive 'my-rock-name_0.1_amd64.rock'

tomponline · 2023-03-09T13:15:23Z

Thanks @mihalicyn ! :)

mihalicyn · 2023-03-09T13:26:04Z

Not at all! Always glad to help! ;-)

Address shared mount issues affecting /dev mount in chroots. This is a result of lxc/lxc#4229 (container rootfs became a shared mount, meaning that unmounts propagates through the shared group and original mounts are unmounted too). See canonical/rockcraft#195 for details. Signed-off-by: Claudio Matsuoka <[email protected]>

Craft-parts 1.18.4 makes the /dev rbind mount private, fixing chroot and overlayfs mounts when using lxd 5.11. Fixes canonical#195. Signed-off-by: Claudio Matsuoka <[email protected]>

Lxd was pinned down to version 5.9 to work around the overlayfs issue reported in canonical#195. With this fix, this should no longer be necessary. Signed-off-by: Claudio Matsuoka <[email protected]>

Craft-parts 1.18.4 makes the /dev rbind mount private, fixing chroot and overlayfs mounts when using lxd 5.11. Fixes #195. Signed-off-by: Claudio Matsuoka <[email protected]>

Lxd was pinned down to version 5.9 to work around the overlayfs issue reported in #195. With this fix, this should no longer be necessary. Signed-off-by: Claudio Matsuoka <[email protected]>

Craft-parts 1.18.4 makes the /dev rbind mount private, fixing chroot and overlayfs mounts when using lxd 5.11. Fixes canonical#195. Signed-off-by: Claudio Matsuoka <[email protected]>

Lxd was pinned down to version 5.9 to work around the overlayfs issue reported in canonical#195. With this fix, this should no longer be necessary. Signed-off-by: Claudio Matsuoka <[email protected]>

gboutry mentioned this issue Mar 6, 2023

Can't prime symlink in resulting oci #208

Closed

cmatsuoka mentioned this issue Mar 9, 2023

chroot: mark /dev mount as private canonical/craft-parts#400

Merged

1 task

cmatsuoka mentioned this issue Mar 9, 2023

requirements: update craft-parts to 1.18.4 #210

Merged

1 task

sergiusens closed this as completed in #210 Mar 9, 2023

sergiusens pushed a commit that referenced this issue Mar 9, 2023

requirements: update craft-parts to 1.18.4

d252916

Craft-parts 1.18.4 makes the /dev rbind mount private, fixing chroot and overlayfs mounts when using lxd 5.11. Fixes #195. Signed-off-by: Claudio Matsuoka <[email protected]>

AmberCharitos mentioned this issue Aug 15, 2024

CSS-10231 Adds apache2-utils and fixes source-commit canonical/charmed-trino-rock#8

Merged

LXD 5.0 fails to build rocks on Ubuntu 22.04 #195

LXD 5.0 fails to build rocks on Ubuntu 22.04 #195

Comments

twovican commented Feb 15, 2023

gboutry commented Mar 6, 2023 • edited Loading

sergiusens commented Mar 7, 2023

tomponline commented Mar 7, 2023 • edited Loading

gboutry commented Mar 7, 2023

tomponline commented Mar 7, 2023 • edited Loading

tomponline commented Mar 7, 2023

tomponline commented Mar 7, 2023

gboutry commented Mar 7, 2023 • edited Loading

tomponline commented Mar 7, 2023

tomponline commented Mar 7, 2023

mihalicyn commented Mar 7, 2023

tomponline commented Mar 7, 2023

cmatsuoka commented Mar 7, 2023

tomponline commented Mar 7, 2023

cmatsuoka commented Mar 7, 2023

tomponline commented Mar 7, 2023 • edited Loading

cmatsuoka commented Mar 7, 2023

tomponline commented Mar 7, 2023

tomponline commented Mar 7, 2023

tomponline commented Mar 7, 2023

cmatsuoka commented Mar 7, 2023

mihalicyn commented Mar 7, 2023 • edited Loading

gboutry commented Mar 8, 2023

tomponline commented Mar 8, 2023

cmatsuoka commented Mar 8, 2023

cmatsuoka commented Mar 8, 2023

mihalicyn commented Mar 8, 2023

mr-cal commented Mar 8, 2023

tomponline commented Mar 8, 2023 • edited Loading

tomponline commented Mar 8, 2023

mr-cal commented Mar 8, 2023

tomponline commented Mar 8, 2023

mr-cal commented Mar 8, 2023

cmatsuoka commented Mar 8, 2023

cmatsuoka commented Mar 8, 2023

mihalicyn commented Mar 9, 2023

tomponline commented Mar 9, 2023

cmatsuoka commented Mar 9, 2023

cmatsuoka commented Mar 9, 2023

tomponline commented Mar 9, 2023

mihalicyn commented Mar 9, 2023

gboutry commented Mar 6, 2023 •

edited

Loading

tomponline commented Mar 7, 2023 •

edited

Loading

tomponline commented Mar 7, 2023 •

edited

Loading

gboutry commented Mar 7, 2023 •

edited

Loading

tomponline commented Mar 7, 2023 •

edited

Loading

mihalicyn commented Mar 7, 2023 •

edited

Loading

tomponline commented Mar 8, 2023 •

edited

Loading