binfmt_misc flags: 'OC' not working

[mailfcly]

• Your Windows build number: (Type ver at a Windows Command Prompt)
  OS Name: Microsoft Windows 10 Pro
OS Version: 10.0.16299 N/A Build 16299

• What you're doing and what's happening: (Copy&paste specific commands and their output, or include screen shots)

"~/wheezy/" is arm rootfs.
I have installed qemu-user-static.

For a simple test qemu-arm-static executable, ~/wheezy/bin/ls, expected results would be:
$ ~/wheezy/bin/ls
/lib/ld-linux.so.3: No such file or directory

While actual results are:
$ ~/wheezy/bin/ls
prepare_binprm: Bad file descriptor

Explicit invocation of qemu-arm-static, however, works:
$ qemu-arm-static ~/wheezy/bin/ls
/lib/ld-linux.so.3: No such file or directory

Show the binfmts list, it is OK:
`update-binfmts --display qemu-arm

qemu-arm (enabled):
package = qemu-user-static
type = magic
offset = 0
magic = \x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
interpreter = /usr/bin/qemu-arm-static
detector =
`

[DDoSolitary]

Same issue here.

[mcmordie]

And here

[Goodwu]

Same here under version 10.0.16299.192

[Froosh]

Windows Version:
Microsoft Windows [Version 10.0.17115.1]

Installed debootstrap:
sudo apt install debootstrap

Prepare a Debian armel root:
sudo debootstrap --verbose --arch=armel --variant=minbase --foreign stretch debian-armel/

Try to run the armel bash prompt:
strace -vff -o bash.armel.strace ./debian-armel/bin/bash

prepare_binprm: Bad file descriptor

Resulting strace files: https://gist.github.com/Froosh/142d0da5dddd15fdc65f66e52a2d644d

[therealkenc]

From strace.102:

read(3, 0x7fffce27eca0, 1024) = -1 EBADF (Bad file descriptor)

File descriptor 3 is nowhere to be found in the two files you posted though. Need the contents of strace -f -o bash.armel.strace ./debian-armel/bin/bash (and some luck). There's some problematic stuff in there already, for example MADV_HUGEPAGE and /proc/sys/vm/mmap_min_addr. Those might be mortal wounds on their own, but not the kill shot.

There is still no repro here though; which is the place to start. Those three commands aren't going to fly on Real Linux either, naturally. [You'll get missing /lib/ld-linux.so.3: No such file or directory just like the OP. Because no qemu.]

This almost certainly isn't going to fly; or someone would have showed off a long time ago. But I can at least give you the first blocker, in return for a repro or/and a complete strace.

[Froosh]

Hehe, yeah, this was a bit rushed in the middle of doing other work, will try and get some better data later today/tonight. Full repro steps on a fresh wsl ubuntu instance, with some comparison strace between relying on binfmt and explicitly calling qemu-arm-static. I might also try the ETL logs to see if they show anything, and perhaps a native ubuntu comparison if I have time.

You've absolutely identified the missing element though: When calling qemu-arm-static explicitly, there is a call to open("/bin/bash", O_RDONLY) which returns FD 3 that doesn't happen through the binfmt assisted call.

[Froosh]

Windows version:
Microsoft Windows [Version 10.0.17120.1]

WSL App 'Ubuntu' version:
1604.2017.922.0

Steps to reproduce on a fresh WSL Ubuntu instance:

sudo apt install debootstrap qemu-user-static
sudo service binfmt-support start
sudo debootstrap --verbose --arch=armel --variant=minbase --foreign stretch debian-armel/
sudo cp /usr/bin/qemu-arm-static debian-armel/usr/bin/
sudo mount --bind /dev debian-armel/dev
sudo mount --bind /dev/pts debian-armel/dev/pts
sudo mount --bind /proc debian-armel/proc
sudo mount --bind /sys debian-armel/sys
sudo strace -vff -o implicit.strace chroot debian-armel/ /bin/bash

Result (doesn't work):
prepare_binprm: Bad file descriptor

Strace log: https://gist.github.com/Froosh/dc472dc953dd534806d68a1040e4c10e

Alternate explicit qemu-arm-static invocation:
sudo strace -vff -o explicit.strace chroot debian-armel/ /usr/bin/qemu-arm-static /bin/bash

Result (works):
I have no name!@Froosh-Book:/#

Strace log: https://gist.github.com/Froosh/267e1c2db65e0359e56034610cf328f0

The main difference (apart from memory addresses/times/etc) is a missing call to open() which returns FD 3.

Implict invocation:

gettid()                                = 169
fstat(3, {st_dev=makedev(0, 2), st_ino=1407374884004467, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=2000, st_size=1021372, st_atime=2018/03/16-12:47:13.221254200, st_mtime=2017/05/16-05:45:32, st_ctime=2018/03/16-12:47:13.268238900}) = 0
geteuid()                               = 0
getegid()                               = 0
read(3, 0x7fffc8464780, 1024)           = -1 EBADF (Bad file descriptor)

Explicit invocation:

gettid()                                = 175
open("/bin/bash", O_RDONLY)             = 3
fstat(3, {st_dev=makedev(0, 2), st_ino=1407374884004467, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=2000, st_size=1021372, st_atime=2018/03/16-12:47:13.221254200, st_mtime=2017/05/16-05:45:32, st_ctime=2018/03/16-12:47:13.268238900}) = 0
geteuid()                               = 0
getegid()                               = 0
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\2\0(\0\1\0\0\0\300\327\2\0004\0\0\0"..., 1024) = 1024

[therealkenc]

The main difference ... is a missing call to open() which returns FD 3.

Thanks for the repro. Possibly ptrace() is dropping ENOSYS which has come up before; although that's pretty weird on an open().

I haven't looked at the problem in detail yet, but at your first apt install step there's the usual invoke-rc.d: could not determine current runlevel on Setting up binfmt-support (2.1.6-1), which is a rocky start out of the gate.

The fact armel "works" is pretty damn cool though. I take back "almost certainly isn't going to fly", because, the silly pig flies. From inside the chroot you can do:

I have no name!@DESKTOP-4UTIQSF:/# /usr/bin/qemu-arm-static /usr/bin/lscpu
Architecture:          armv7l
CPU op-mode(s):        32-bit, 64-bit
[...]

Whatever the problem, I strongly suspect it is going to be generic to binfmt-support not qemu-arm. That's the place to look.

[edit,aside: I suspect the binfmt-support can be overcome. If the way we end up getting x86 32-bit support is by running qemu inside WSL that is going to be a special kind of amusing.]

[therealkenc]

Okay -- @benhillis, here is a repro that is a little easier to eat than a full blown debootstrap:

$ sudo apt install qemu-user-static
$ # make the binfmt-support postinstall script happy just to be safe, but I doubt it matters
$ sudo bash -c "echo '[1] [00049] [~~  ] [runlevel] [~           ] [4.4.0-17115-Micoroso] [0.0.0.0        ] [Wed Feb 28 13:27:14 2018 STD]' | utmpdump -r > /var/run/utmp"
$ sudo apt install --reinstall binfmt-support
$ sudo service binfmt-support start # registers qemu-arm binfmt
 * Enabling additional executable binary formats binfmt-support                                       [ OK ]
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: OC
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
$ wget https://github.com/andrew-d/static-binaries/raw/master/binaries/linux/arm/file
$ hexdump ./file | head -n 2
0000000 457f 464c 0101 0001 0000 0000 0000 0000
0000010 0002 0028 0001 0000 340c 0001 0034 0000
$ chmod 755 ./file
$ ./file ./file
prepare_binprm: Bad file descriptor

The strace -f -o file.strace ./file ./file looks the same as reported, including the absent open() [edit: wait, maybe it's not an open()].

mmap(NULL, 151552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2b1e090000
gettid()                                = 890
[edit: something happens here]
fstat(3, {st_mode=S_IFREG|0755, st_size=194976, ...}) = 0
geteuid()                               = 1000
getegid()                               = 1000
read(3, 0x7ffffa628160, 1024)           = -1 EBADF (Bad file descriptor)

I don't think the fact that qemu-arm-static is the registered interpreter matters here. That much appears to be working fine.

[therealkenc]

WSL's builtin binfmt support doesn't seem to like the fancy flags: OC. Or that's how it looks. So from the top with repetition from Froosh:

$ sudo apt install debootstrap qemu-user-static
$ sudo service binfmt-support start
$ sudo debootstrap --verbose --arch=armel --variant=minbase --foreign stretch debian-armel/
$ sudo cp /usr/bin/qemu-arm-static debian-armel/usr/bin/
$ # pop the interpreter as registered by qemu-user-static package
$ sudo update-binfmts --package qemu-user-static --remove qemu-arm /usr/bin/qemu-arm-static
$ # add it back without any special options
$ sudo update-binfmts --install qemu-arm /usr/bin/qemu-arm-static --magic '\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00' --mask '\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff'
$ sudo mount --bind /dev debian-armel/dev
$ sudo mount --bind /dev/pts debian-armel/dev/pts
$ sudo mount --bind /proc debian-armel/proc
$ sudo mount --bind /sys debian-armel/sys
$ # don't forget WSL's binfmt_misc filesystem
$ sudo mount --bind /proc/sys/fs/binfmt_misc debian-armel/proc/sys/fs/binfmt_misc
$ sudo cp -afL /etc/resolv.conf debian-armel/etc  # we'll want this for apt
$ sudo chroot debian-armel/ /bin/bash

And, after doing the second stage and installing a few packages with the armel apt binary, would you look at that:

I'm getting Unsupported syscall: 384 (notably on lsb_release, shrug) but otherwise it is working surprisingly well. Performance seems alright to the naked eye.

[therealkenc]

WSL's builtin binfmt support doesn't seem to like the fancy flags: OC. Or that's how it looks.

Ref #2103 (message), which I should have looked for harder before I started. If the flags: OC is supported, then it must be something else and removing/re-installing the interpreter works despite itself.

[Froosh]

Awesome work!

Agreed, some experimentation shows it doesn't work with flag 'O' or flag 'C' separately, but does work without either (no flags).

ARMel appears to be working nicely, though MIPSel segfaults with access errors after running a command (e.g. ls) successfully. That's a matter for a separate issue though, when I've got more time to test and document the various architectures ;)

Fulfilling the law of WSL issues: now if only we could load loop devices and/or other filesystems (FUSE!) …

Love the idea of using qemu to get 32-bit compatibility running, inspired!

[therealkenc]

some experimentation shows it doesn't work with flag 'O' or flag 'C' separately

That makes #2103 really weird. 🤷

Fulfilling the law of WSL issues

Yep.

Love the idea of using qemu to get 32-bit compatibility running, inspired!

If qemu is in your wheelhouse (it isn't especially in mine) a cookbook on how to get x86 emulation working in situ would be valuable to a number of people. Clearly it is doable, even if not necessarily out of the box. With a little more work, maybe even some other software that still depends a lot on x86 legacy code.

[mcmordie]

Your workaround of removing and readding ARM binfmt suport works for me! Thanks for your great work on this!

[edit] now no longer working-- I get:
update-binfmts: warning: current package is qemu-user-static, but binary format already installed by <local>; not removing.

Not clear to me what this means but this has broken the package for sure-- can't remove and reinstall it.

[therealkenc]

WSL's builtin binfmt support doesn't seem to like the fancy flags: OC. Or that's how it looks.

This was never enough of a showstopper to get grease since 2017. The binfmt flags support is complete in WSL2.