"sh: write error: Invalid argument" errors when register #38

heyi-arm · 2018-05-23T08:00:10Z

Hi, on my x86_64 server:

$ uname -a
Linux a010291 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64

Run the command "docker run --rm --privileged multiarch/qemu-user-static:register" encounters errors:

sh: write error: Invalid argument
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: Invalid argument

But run command ""docker run --rm --privileged multiarch/qemu-user-static:register --reset" actually installs all the interpreters successfully:

Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa

junaruga · 2019-07-28T12:16:15Z

@heyi-arm hi the behavior you reported is expected behavior.

The error message sh: write error: Invalid argument is not understandable.
But the actual meaning is when "docker run --rm --privileged multiarch/qemu-user-static:register" try to add "binfmt_misc" entry files /proc/sys/fs/binfmt_misc/qemu-$arch, those files already exist.
/proc/sys/fs/binfmt_misc/qemu-$arch files are special files . [1]
When updating the file, we have to delete the file at once.

I am not sure which Linux distribution you are using.
When installing RPM qemu-user-staticon CentOS or Fedora, it adds the /proc/sys/fs/binfmt_misc/qemu-$arch files as a post script after yum install qemu-user-static.
There is deb package qemu-user-static package on Debian or Ubuntu.
It is possible those files already exist when you run "docker run --rm --privileged multiarch/qemu-user-static:register".

Here is the way to change flags: to flags: F for /proc/sys/fs/binfmt_misc/qemu-aarch64.

# prompt is executed by root user. sudo does not work for the operation.
But if you try this operation, that is by your own responsibility.

If the environment is really clean without binfmt_misc files. Maybe condition is like this.

$ ls /proc/sys/fs/binfmt_misc
register  status

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access '/proc/sys/fs/binfmt_misc/qemu-aarch64': No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-aarch64

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags:  
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access '/proc/sys/fs/binfmt_misc/qemu-aarch64': No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /qus/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

I would close this ticket, as it is expected behavior.
But you like to reopen, you can do it.
Thanks for reporting!

By the way today we release a new image "multiarch/qemu-user-static" that can build and run standard architecture specific container instead of multiarch compatible imagtes.

Like this.

$ uname -m
x86_64

$ docker run --rm -t arm64v8/ubuntu uname -m
standard_init_linux.go:211: exec user process caused "exec format error"

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ docker run --rm -t arm64v8/ubuntu uname -m
aarch64

[1] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

junaruga · 2019-07-28T12:40:33Z

The register image is using below QEMU's scripts/qemu-binfmt-conf.sh script internally.
So, if the message is not understandable, you might be able to report to QEMU project.

Below is the logic to add binfmt_misc entry.

https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh#L269-L270

qemu_register_interpreter() {
    echo "Setting $qemu as binfmt interpreter for $cpu"
    qemu_generate_register > /proc/sys/fs/binfmt_misc/register
}

hoshsadiq · 2019-08-26T20:13:32Z

@junaruga I'm not sure I follow all of what you said. Could you please expand a little bit on it?

My build output can be found here: https://travis-ci.com/hoshsadiq/qemu-testing/builds/124639278

As you can the /proc/sys/fs/binfmt_misc/ is empty and doesn't have the files as suggested in your comment (unless I misunderstood).

junaruga · 2019-08-27T17:35:13Z

https://travis-ci.com/hoshsadiq/qemu-testing/builds/124639278

@hoshsadiq in your Travis log, below command outputs error message then finishes with exit status: 0. that is what we have to fix.

$ docker run --rm --privileged multiarch/qemu-user-static:$QUS_VERSION --reset -p yes
Status: Downloaded newer image for multiarch/qemu-user-static:v4.0.0-4
sh: write error: Invalid argument
...

That means docker run --rm --privileged multiarch/qemu-user-static:$QUS_VERSION --reset -p yes failed to create /proc/sys/fs/binfmt_misc/qemu-* files.
I do not know why for now. To know it, we need to debug /qemu-binfmt-conf.sh injecting set -x in it.

But in your case I found why the issue happened.

Here is my repository forked from your qemu testing repository.
https://github.com/junaruga/qemu-testing/tree/feature/debug
https://travis-ci.org/junaruga/qemu-testing/builds/577460704

diff --git a/.travis.yml b/.travis.yml
index 8a2fc7a..b31a117 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,4 +1,4 @@
-dist: trusty
+dist: xenial
 # sudo: required
 language: minimal

After changing dist from trusty to xenial, you can run it without errors.

junaruga · 2019-08-27T23:05:18Z

This repository's .travis.yml might be also useful.
https://github.com/junaruga/ci-multi-arch-test

hoshsadiq · 2019-08-28T15:39:17Z

Ah. I was being stupid. I was meant to be using bionic. Seems this is supported on Xenial and above. Thank you!

lugeng · 2019-10-18T05:45:08Z

echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F" > /proc/sys/fs/binfmt_misc/register
bash: line 0: echo: write error: Invalid argument

ls /proc/sys/fs/binfmt_misc/qemu-aarch64
ls: cannot access /proc/sys/fs/binfmt_misc/qemu-aarch64: No such file or directory

But
echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

ls /proc/sys/fs/binfmt_misc/qemu-aarch64
/proc/sys/fs/binfmt_misc/qemu-aarch64 works

cannot set flag with F

yuanzhedong · 2020-02-01T07:02:12Z

@lugeng I got the same error, did you find a solution? Thanks!

junaruga · 2020-02-01T12:45:50Z

The F flag is a relatively new feature of the kernel binfmt_misc. Maybe it does not work if the host os kernel version (= uname -r) < 4. See #100 (comment) for the solution.

Deepak275 · 2021-07-25T08:39:53Z

facing the same issue, kernel is 4.4, OS: Ubuntu 18.04.4 LTS. Tried running both, with and without -p yes, but no progress. Any help is appreciated ?

lqhandsome · 2021-09-18T07:37:19Z

must uname -r > 4.8

gasgithub · 2022-06-10T10:33:48Z

I have similar issue in docker.io/library/docker:19.03.14 container (Linux reynwz-containerize-pod 3.10.0-1160.62.1.el7.x86_64 #1 SMP Wed Mar 23 09:04:02 UTC 2022 x86_64 Linux) running the following but getting errors:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

Status: Downloaded newer image for multiarch/qemu-user-static:latest 
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha sh: write error: Invalid argument
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm sh: write error: Invalid argument
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64 sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64 sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32 sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64 sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4 sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64 sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32 sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64 sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb sh: write error: Invalid argument
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze sh: write error: Invalid argument
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel sh: write error: Invalid argument
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k sh: write error: Invalid argument
Setting /usr/bin/qemu-hexagon-static as binfmt interpreter for hexagon sh: write error: Invalid argument

Any hints whats the problem and what can I do to fix this issue?

My environment is:

+ docker version
Client: Docker Engine - Community
 Version:           19.03.14
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        5eb3275
 Built:             Tue Dec  1 19:14:24 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.15
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       99e3ed8
  Built:            Sat Jan 30 03:18:13 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

orgads · 2022-11-08T09:36:49Z

@gasgithub Your kernel is probably too old. Try to upgrade it.

Deepak275 · 2023-01-15T08:47:49Z

must uname -r > 4.8
@lqhandsome Yeah, that was the issue, thanks. :)

kfunaoka mentioned this issue Jul 18, 2018

[Feature] Cross compile Autoware for aarch64 autowarefoundation/autoware#1375

Merged

junaruga closed this as completed Jul 28, 2019

jc21 mentioned this issue Dec 16, 2019

sh: write error: Invalid argument - Centos 7 #100

Closed

Silex mentioned this issue Mar 15, 2020

failed to load LLB: runtime execution on platform linux/arm64 not supported docker/buildx#138

Closed

metaist mentioned this issue Aug 16, 2024

Fix invalid file (bad magic number) Exec format error on Linux ahgamut/superconfigure#50

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"sh: write error: Invalid argument" errors when register #38

"sh: write error: Invalid argument" errors when register #38

heyi-arm commented May 23, 2018

junaruga commented Jul 28, 2019

junaruga commented Jul 28, 2019

hoshsadiq commented Aug 26, 2019

junaruga commented Aug 27, 2019

junaruga commented Aug 27, 2019

hoshsadiq commented Aug 28, 2019

lugeng commented Oct 18, 2019 •

edited

Loading

yuanzhedong commented Feb 1, 2020

junaruga commented Feb 1, 2020 •

edited

Loading

Deepak275 commented Jul 25, 2021 •

edited

Loading

lqhandsome commented Sep 18, 2021

gasgithub commented Jun 10, 2022

orgads commented Nov 8, 2022

Deepak275 commented Jan 15, 2023

"sh: write error: Invalid argument" errors when register #38

"sh: write error: Invalid argument" errors when register #38

Comments

heyi-arm commented May 23, 2018

junaruga commented Jul 28, 2019

junaruga commented Jul 28, 2019

hoshsadiq commented Aug 26, 2019

junaruga commented Aug 27, 2019

junaruga commented Aug 27, 2019

hoshsadiq commented Aug 28, 2019

lugeng commented Oct 18, 2019 • edited Loading

yuanzhedong commented Feb 1, 2020

junaruga commented Feb 1, 2020 • edited Loading

Deepak275 commented Jul 25, 2021 • edited Loading

lqhandsome commented Sep 18, 2021

gasgithub commented Jun 10, 2022

orgads commented Nov 8, 2022

Deepak275 commented Jan 15, 2023

lugeng commented Oct 18, 2019 •

edited

Loading

junaruga commented Feb 1, 2020 •

edited

Loading

Deepak275 commented Jul 25, 2021 •

edited

Loading