[Asahi Linux] ubuntu 22.x (?) core dumps in `linux/amd64`, `linux/s390x` emulation on arm64 host (apple silicon) with `p_rcu_reader->depth != 0` #90

maxheld83 · 2022-06-05T16:09:25Z

To reproduce, on an arm64 host (Apple Silicon) I ran:

sudo docker run --privileged --rm tonistiigi/binfmt:qemu-v6.1.0 --install all
# {
#  "supported": [
#    "linux/arm64",
#    "linux/amd64",
#    "linux/riscv64",
#    "linux/ppc64le",
#    "linux/s390x",
#    "linux/386",
#    "linux/mips64le",
#    "linux/mips64",
#    "linux/arm/v7",
#    "linux/arm/v6"
#  ],
#  "emulators": [
#    "qemu-arm",
#    "qemu-i386",
#    "qemu-mips64",
#    "qemu-mips64el",
#    "qemu-ppc64le",
#    "qemu-riscv64",
#    "qemu-s390x",
#    "qemu-x86_64"
#  ]
#}

This is:

docker run --privileged --rm tonistiigi/binfmt --version
# binfmt/1aa2eba qemu/v6.2.0 go/1.17.8

These fail:

docker run --platform linux/amd64 ubuntu apt-get update
# Assertion failed: p_rcu_reader->depth != 0 (/qemu/include/qemu/rcu.h: rcu_read_unlock: 101)

Same thing for linux/s390x, but all other above enabled emulators work fine (in so far as they the platform is supported by ubuntu).

I'm guessing this affects more than just apt-get update; I just needed that first.
Some downstream (of ubuntu 22.x) images, such as r-base are also affected (already on their default ENTRYPOINT/CMD).

A bunch of other official images also seemed to be unaffected on cursory inspection, including alpine, debian and fedora.

The problem also does not occur (even for linux/s290x on an amd64 host).

So, TL,DR, it seems you need all of these for the problem to arise:

arm64 host (maybe just Apple Silicon?)
linux/amd64 or linux/s390x image architecture
Ubuntu 22.x (20.04 gives another odd error from apt-get key list, but not a coredump -- 18.04 works without a hitch)

I'm running this on a vanilla Asahi Linux Apple M1 machine.
(I don't have access to another arm64 host, so couldn't cross-validate whether the issue is somehow related to Apple Silicon).

The text was updated successfully, but these errors were encountered:

maxheld83 · 2022-06-05T16:19:44Z

uh, nevermind the limitation to Ubuntu 22.x, these also fail with the same coredump:

docker run -it --platform linux/amd64 rocker/r-ver
# based on ubuntu 20.04

docker run -it --platform linux/amd64 r-base
# based on debian bookworm/sid

tonistiigi · 2022-06-05T20:21:41Z

Under regular Apple M1 the command seems to work fine:

 # uname -a
Darwin MacBook-Pro.local 21.4.0 Darwin Kernel Version 21.4.0: Mon Feb 21 20:35:58 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T6000 arm64
 # docker run --platform linux/amd64 --rm ubuntu apt-get update
Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [160 kB]
Get:6 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [156 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:10 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [71.8 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [4653 B]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [4653 B]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [178 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [126 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [298 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [2009 B]
Fetched 21.3 MB in 12s (1749 kB/s)
Reading package lists...

I don't have Asahi Linux though. Not sure if anything special in that kernel. You might also want to check your docker version, sometimes seccomp causes issues on older versions. You can also just try the test command without seccomp to be sure that is not the case.

maxheld83 · 2022-06-06T16:18:17Z

You might also want to check your docker version

docker --version
# Docker version 20.10.16, build aa7e414fdc

That's current, I think.

You can also just try the test command without seccomp to be sure that is not the case.

docker run --rm --security-opt seccomp=unconfined --platform linux/amd64 ubuntu apt-get update
# Assertion failed: p_rcu_reader->depth != 0 (/qemu/include/qemu/rcu.h: rcu_read_unlock: 101)

I think this disables seccomp?
Same issue.

maxheld83 · 2022-06-06T16:32:34Z

I'll crosspost this on Asahi.

maxheld83 · 2022-06-06T20:47:27Z

maybe related:

qemu works out of the box on M1 Pro/Max and should work with highmem=off on M1 (until a newer qemu version is released, which should fix that), I believe.

-- asahi lead dev @marcan, via reddit

@tonistiigi is there a way for me to pass highmem=off to qemu-user in tonistiigi/binfmt:qemu-v6.1.0 to test whether this fixes the issue?
Or to try qemu-v7.0.0?

crazy-max · 2022-07-11T13:27:47Z

@maxheld83 Can you test with #93 (comment) and let me know?

maxheld83 · 2022-07-11T16:28:56Z

@crazy-max don't think I can (on asahi linux):

docker run --privileged --rm crazymax/binfmt:v7.0.0 --install all
# WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
# exec /usr/bin/binfmt: exec format error

As far as I can see from https://hub.docker.com/r/crazymax/binfmt/tags, there is no linux/arm64 version of the image, correct?

crazy-max · 2022-07-11T17:03:52Z

@maxheld83 Sorry forgot to push a multi platform image. Should be ok now.

maxheld83 · 2022-07-11T18:02:15Z

@crazy-max thanks 🙏, done ☑️.

Same problem (coredump), unfortunately 😔.

docker run --privileged --rm crazymax/binfmt:v7.0.0 --install all
# installing: ppc64le OK
# installing: riscv64 OK
# installing: 386 OK
# installing: mips64 OK
# installing: amd64 OK
# installing: arm OK
# installing: s390x OK
# installing: mips64le OK
# {
#   "supported": [
#     "linux/arm64",
#     "linux/amd64",
#     "linux/riscv64",
#     "linux/ppc64le",
#     "linux/s390x",
#     "linux/386",
#     "linux/mips64le",
#     "linux/mips64",
#     "linux/arm/v7",
#     "linux/arm/v6"
#   ],
#   "emulators": [
#     "qemu-arm",
#     "qemu-i386",
#     "qemu-mips64",
#     "qemu-mips64el",
#     "qemu-ppc64le",
#     "qemu-riscv64",
#     "qemu-s390x",
#     "qemu-x86_64"
#   ]
# }
docker run --privileged --rm crazymax/binfmt:v7.0.0 --version
# binfmt/eb3fd34 qemu/v7.0.0 go/1.17.11
docker run --platform linux/amd64 ubuntu apt-get update  
# Assertion failed: p_rcu_reader->depth != 0 (/qemu/include/qemu/rcu.h: rcu_read_unlock: 102)
docker run --rm --security-opt seccomp=unconfined --platform linux/amd64 ubuntu apt-get update
# Assertion failed: p_rcu_reader->depth != 0 (/qemu/include/qemu/rcu.h: rcu_read_unlock: 102)

Lesiuk · 2023-02-16T12:48:06Z

I compiled tonistiigi/binfmt with qemu master and many amd64 apps are stuck in infinite loop (100% cpu consumption). One of the examples is cmake --version.

I am using Asahi Linux on M1 Mac Mini

arch-fan · 2023-12-05T04:01:04Z

It's happening. Trying to run Microsoft SQL Server on Raspberry Pi 5

pi@raspberrypi:~/docker/portainer $ docker run --platform linux/amd64 --rm mcr.microsoft.com/mssql/server
SQL Server 2022 will run as non-root by default.
This container is running as user mssql.
To learn more visit https://go.microsoft.com/fwlink/?linkid=2099216.
Assertion failed: p_rcu_reader->depth != 0 (/qemu/include/qemu/rcu.h: rcu_read_unlock: 102)

maxheld83 changed the title ~~ubuntu 22.x core dumps in linux/amd64, linux/s390x emulation on arm64 host (apple silicon)~~ ubuntu 22.x (?) core dumps in linux/amd64, linux/s390x emulation on arm64 host (apple silicon) with p_rcu_reader->depth != 0 Jun 5, 2022

maxheld83 mentioned this issue Jun 6, 2022

qemu-user emulator images core dump when emulating linux/amd64, linux/s390x with p_rcu_reader->depth != 0 AsahiLinux/linux#33

Open

crazy-max mentioned this issue Jul 11, 2022

update qemu version to v7.0.0 #93

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Asahi Linux] ubuntu 22.x (?) core dumps in `linux/amd64`, `linux/s390x` emulation on arm64 host (apple silicon) with `p_rcu_reader->depth != 0` #90

[Asahi Linux] ubuntu 22.x (?) core dumps in `linux/amd64`, `linux/s390x` emulation on arm64 host (apple silicon) with `p_rcu_reader->depth != 0` #90

maxheld83 commented Jun 5, 2022 •

edited

Loading

maxheld83 commented Jun 5, 2022

tonistiigi commented Jun 5, 2022

maxheld83 commented Jun 6, 2022

maxheld83 commented Jun 6, 2022 •

edited

Loading

maxheld83 commented Jun 6, 2022 •

edited

Loading

crazy-max commented Jul 11, 2022

maxheld83 commented Jul 11, 2022

crazy-max commented Jul 11, 2022

maxheld83 commented Jul 11, 2022

Lesiuk commented Feb 16, 2023 •

edited

Loading

arch-fan commented Dec 5, 2023

[Asahi Linux] ubuntu 22.x (?) core dumps in linux/amd64, linux/s390x emulation on arm64 host (apple silicon) with p_rcu_reader->depth != 0 #90

[Asahi Linux] ubuntu 22.x (?) core dumps in linux/amd64, linux/s390x emulation on arm64 host (apple silicon) with p_rcu_reader->depth != 0 #90

Comments

maxheld83 commented Jun 5, 2022 • edited Loading

maxheld83 commented Jun 5, 2022

tonistiigi commented Jun 5, 2022

maxheld83 commented Jun 6, 2022

maxheld83 commented Jun 6, 2022 • edited Loading

maxheld83 commented Jun 6, 2022 • edited Loading

crazy-max commented Jul 11, 2022

maxheld83 commented Jul 11, 2022

crazy-max commented Jul 11, 2022

maxheld83 commented Jul 11, 2022

Lesiuk commented Feb 16, 2023 • edited Loading

arch-fan commented Dec 5, 2023

[Asahi Linux] ubuntu 22.x (?) core dumps in `linux/amd64`, `linux/s390x` emulation on arm64 host (apple silicon) with `p_rcu_reader->depth != 0` #90

[Asahi Linux] ubuntu 22.x (?) core dumps in `linux/amd64`, `linux/s390x` emulation on arm64 host (apple silicon) with `p_rcu_reader->depth != 0` #90

maxheld83 commented Jun 5, 2022 •

edited

Loading

maxheld83 commented Jun 6, 2022 •

edited

Loading

maxheld83 commented Jun 6, 2022 •

edited

Loading

Lesiuk commented Feb 16, 2023 •

edited

Loading