7.4.21-fpm-alpine3.14 missing linux/ppc64le and linux/s390x #1178

jokay · 2021-07-04T13:38:55Z

Using 7.4.21-fpm-alpine3.14 linux/ppc64le and linux/s390x are missing.

7.4.20-fpm-alpine3.13

7.4.21-fpm-alpine3.14

Can we expect them to get build or won't this happen anymore for these architectures? 🧐

The text was updated successfully, but these errors were encountered:

tianon · 2021-07-06T22:30:30Z

Interestingly, we're getting build failures with all the hallmarks of opencontainers/runc#2750 but we're on Docker 20.10.7 with runc 1.0.0-rc95 (so that shouldn't be the problem). When I try the offending command with strace, I see that indeed, it is getting EPERM on faccessat2 (which shouldn't be happening), but if I run the same exact command directly it works fine, so I have to imagine this is something specific to make on these platforms?? 😬

...
writev(1, [{iov_base="", iov_len=0}, {iov_base="/bin/sh /usr/src/php/libtool --s"..., iov_len=563}], 2/bin/sh /usr/src/php/libtool --silent --preserve-dup-deps --mode=compile cc -Iext/opcache/ -I/usr/src/php/ext/opcache/ -I/usr/src/php/include -I/usr/src/php/main -I/usr/src/php -I/usr/src/php/ext/date/lib -I/usr/include/libxml2 -I/usr/src/php/TSRM -I/usr/src/php/Zend  -D__MUSL__  -Wno-clobbered -Wall -Wextra -Wno-strict-aliasing -Wno-implicit-fallthrough -Wno-unused-parameter -Wno-sign-compare -g -O2 -fvisibility=hidden -DZEND_SIGNALS    -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -c /usr/src/php/ext/opcache/zend_shared_alloc.c -o ext/opcache/zend_shared_alloc.lo 
) = 563
madvise(0x3ff85d25000, 4096, MADV_FREE) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
faccessat2(AT_FDCWD, "/bin/sh", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: /bin/sh: Operation not per"..., iov_len=39}], 2make: /bin/sh: Operation not permitted
) = 39
...

tianon · 2021-08-10T20:56:00Z

I've reproduced with a much simpler reproducer:

$ docker run -it --rm -w /tmp alpine:3.14
/tmp # apk add --no-cache make strace
...
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp # make
echo yay
make: echo: Operation not permitted
make: *** [Makefile:2: all] Error 127
/tmp # strace make
...
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
faccessat2(AT_FDCWD, "/usr/local/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/local/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: echo: Operation not permit"..., iov_len=36}], 2make: echo: Operation not permitted
) = 36
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: *** [Makefile:2: all] Erro"..., iov_len=38}], 2make: *** [Makefile:2: all] Error 127
) = 38
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
munmap(0x3ffabb4d000, 4096)             = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
chdir("/tmp")                           = 0
close(1)                                = 0
exit_group(2)                           = ?
+++ exited with 2 +++

Docker 20.10.7, runc 1.0.1, containerd 1.5.4, libseccomp 2.4.1 😞

(Should be plenty new in all respects for us to not see this.)

thaJeztah · 2021-08-10T21:36:19Z

@nirmannarang @anujajakhade FYI; perhaps one of you has ideas what's different here for s390x? FWIW, the faccessat2 has been on the "allow" list in docker's seccomp profile since docker 20.10.0 (see moby/moby#41353), so seccomp shouldn't be blocking it.

nirmannarang · 2021-08-11T11:59:30Z

@nirmannarang @anujajakhade FYI; perhaps one of you has ideas what's different here for s390x? FWIW, the faccessat2 has been on the "allow" list in docker's seccomp profile since docker 20.10.0 (see moby/moby#41353), so seccomp shouldn't be blocking it.

Will take a look.

clnperez · 2021-08-12T19:24:11Z

@tianon thanks for all those details. I don't know why this would happen only for Power and z. I remember the horrendous problem of another syscall in golang for which the value for p and z was different than that for x86/amd -- but I double-checked and it's 439 across the board.

The requirements in the faccessat2 section here are a little confusing: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0. Could we try with libseccomp 2.4.4?

tianon · 2021-08-12T22:28:41Z

Apparently my reproducer works fine (as in, does not reproduce any issue and properly prints yay) on ppc64le, however I can definitely reproduce with that minimal make on s390x even with libseccomp version 2.5.1 from Debian Bullseye.

tianon · 2021-08-12T22:41:06Z

Confirmed, apparently ppc64le has been fixed since this issue was filed! All those tags are built/pushed successfully. 😄

anujajakhade · 2021-08-13T10:17:21Z

I think this issue is resolved on s390x as well. I am not able to reproduce it. @tianon @jokay can you please confirm.

dropper # docker run -it --rm -w /tmp alpine:3.14
/tmp # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.1
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/tmp # uname -a
Linux 71fca25f7540 5.3.18-57-default #1 SMP Wed Apr 28 10:54:41 UTC 2021 (ba3c2e9) s390x Linux
/tmp

jokay · 2021-08-13T10:40:29Z

@anujajakhade sorry I can't test this, I only use the Docker image to build multi-arch Docker images :)

thaJeztah · 2021-08-13T11:45:18Z

I wasn't able to fit an s390x server in my closet, so not sure if it adds anything to the table, but I can confirm it works on Docker Desktop for Mac (using QEMU emulation), or at least, it prints yay;

docker run -it --rm --platform=linux/s390x -w /tmp alpine:3.14
Unable to find image 'alpine:3.14' locally
3.14: Pulling from library/alpine
625f57562315: Pull complete
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Downloaded newer image for alpine:3.14
/tmp # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.14.1
PRETTY_NAME="Alpine Linux v3.14"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/tmp # uname -a
Linux f4b3c3366135 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 s390x Linux
/tmp # apk add --no-cache make strace
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/s390x/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/s390x/APKINDEX.tar.gz
(1/2) Installing make (4.3-r0)
(2/2) Installing strace (5.12-r0)
Executing busybox-1.33.1-r3.trigger
OK: 7 MiB in 16 packages
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp # make
echo yay
yay
/tmp # strace make
/usr/bin/strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
/usr/bin/strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
/usr/bin/strace: PTRACE_SETOPTIONS: Function not implemented
/usr/bin/strace: detach: waitpid(37): No child process
/usr/bin/strace: Process 37 detached

clnperez · 2021-08-13T16:38:01Z

Good news all around. I would really love to know what fixed it where, if anyone has more info!

tianon · 2021-08-13T16:56:21Z

😕

$ date
Fri 13 Aug 2021 09:51:49 AM PDT

$ dpkg -l | grep libseccomp
ii  libseccomp2:s390x              2.4.1-2~bpo10+1              s390x        high level interface to Linux seccomp filter

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ docker version
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.6
 Git commit:        20.10.7-1~deb10u0
 Built:             Tue Jul 20 03:36:24 2021
 OS/Arch:           linux/s390x
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.6
  Git commit:       20.10.7-2~deb10u0
  Built:            Mon Jul 19 20:13:31 2021
  OS/Arch:          linux/s390x
  Experimental:     false
 containerd:
  Version:          1.5.4-1~deb10u0
  GitCommit:        
 runc:
  Version:          1.0.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

$ docker pull alpine:3.14
3.14: Pulling from library/alpine
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Image is up to date for alpine:3.14
docker.io/library/alpine:3.14

$ docker run -it --rm -w /tmp alpine:3.14
/tmp # apk add --no-cache make strace
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/s390x/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/s390x/APKINDEX.tar.gz
(1/2) Installing make (4.3-r0)
(2/2) Installing strace (5.12-r0)
Executing busybox-1.33.1-r3.trigger
OK: 7 MiB in 16 packages
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp # make
echo yay
make: echo: Operation not permitted
make: *** [Makefile:2: all] Error 127
/tmp # strace make
execve("/usr/bin/make", ["make"], 0x3ffdecff280 /* 6 vars */) = 0
...
munmap(0x3ffbd9ce000, 28672)            = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
faccessat2(AT_FDCWD, "/usr/local/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/local/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: echo: Operation not permit"..., iov_len=36}], 2make: echo: Operation not permitted
) = 36
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: *** [Makefile:2: all] Erro"..., iov_len=38}], 2make: *** [Makefile:2: all] Error 127
) = 38
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
munmap(0x3ffbd9cd000, 4096)             = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
chdir("/tmp")                           = 0
close(1)                                = 0
exit_group(2)                           = ?
+++ exited with 2 +++

$ date
Fri Aug 13 16:54:05 UTC 2021

$ dpkg -l | grep libseccomp
ii  libseccomp2:s390x             2.5.1-1                      s390x        high level interface to Linux seccomp filter

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ docker version
Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.15.9
 Git commit:        20.10.7-1~deb11u0
 Built:             Tue Jul 20 03:45:31 2021
 OS/Arch:           linux/s390x
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.9
  Git commit:       20.10.7-2~deb11u0
  Built:            Mon Jul 19 20:13:31 2021
  OS/Arch:          linux/s390x
  Experimental:     false
 containerd:
  Version:          1.5.4-1~deb11u0
  GitCommit:        
 runc:
  Version:          1.0.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

$ docker pull alpine:3.14
3.14: Pulling from library/alpine
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Image is up to date for alpine:3.14
docker.io/library/alpine:3.14

$ docker run -it --rm -w /tmp alpine:3.14
/tmp # apk add --no-cache make strace
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/s390x/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/s390x/APKINDEX.tar.gz
(1/2) Installing make (4.3-r0)
(2/2) Installing strace (5.12-r0)
Executing busybox-1.33.1-r3.trigger
OK: 7 MiB in 16 packages
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp # make
echo yay
make: echo: Operation not permitted
make: *** [Makefile:2: all] Error 127
/tmp # strace make
execve("/usr/bin/make", ["make"], 0x3ffd947f930 /* 6 vars */) = 0
...
munmap(0x3ffbae4e000, 28672)            = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
faccessat2(AT_FDCWD, "/usr/local/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/local/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/usr/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/sbin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
faccessat2(AT_FDCWD, "/bin/echo", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: echo: Operation not permit"..., iov_len=36}], 2make: echo: Operation not permitted
) = 36
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
writev(2, [{iov_base="", iov_len=0}, {iov_base="make: *** [Makefile:2: all] Erro"..., iov_len=38}], 2make: *** [Makefile:2: all] Error 127
) = 38
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
munmap(0x3ffbae4d000, 4096)             = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
chdir("/tmp")                           = 0
close(1)                                = 0
exit_group(2)                           = ?
+++ exited with 2 +++

Both of these were on:

$ uname -a
Linux lxdeb05 4.19.0-9-s390x #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) s390x GNU/Linux

nirmannarang · 2021-08-19T11:03:37Z

@tianon That's strange.
Works fine for me for both Docker versions: 19.x and 20.x.

$ docker run -it --rm -w /tmp alpine:3.14
Unable to find image 'alpine:3.14' locally
3.14: Pulling from library/alpine
625f57562315: Pull complete
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Downloaded newer image for alpine:3.14
/tmp # uname -a
Linux 1abdee79c44b 5.11.0-31-generic #33-Ubuntu SMP Wed Aug 11 13:18:18 UTC 2021 s390x Linux
/tmp # apk add --no-cache make strace
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/s390x/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/s390x/APKINDEX.tar.gz
(1/2) Installing make (4.3-r0)
(2/2) Installing strace (5.12-r0)
Executing busybox-1.33.1-r3.trigger
OK: 7 MiB in 16 packages
/tmp # echo -e 'all:\n\techo yay' > Makefile
/tmp #  make
echo yay
yay

nirmannarang · 2021-08-19T11:32:41Z

@tianon I just noticed, you are using Docker on Debian s390x. How did you install it?
The above working result I had posted is on Docker on Ubuntu s390x.

tianon · 2021-08-19T21:35:52Z

Not sure how our host system was installed (it was a VM provided to us by IBM), but the Bullseye system is a Docker-in-Docker instance on that same system. I want to update the host to Bullseye too, but honestly I'm a bit scared to do so because the last time we did things like that we broke the DASD configuration somehow and it required a higher escalation to fix. 😅

tianon · 2021-08-20T17:30:21Z

Yessss, after a conversation with @clnperez to make sure we could find the right folks if things went poorly, I updated to Bullseye -- the upgrade went flawlessly (minus having to remove an old kernel package because our /boot is so so tiny), and now my reproducer above no longer does!! (now on kernel 5.10.46-4)

So, gonna trigger the real builds and see how it goes. 🤞

tianon · 2021-08-20T18:31:17Z

Confirmed, all is well -- builds are in progress, but there are quite a few, so it's going to be a bit. 😅

thaJeztah · 2021-08-20T21:23:54Z

Great news! Thanks @tianon @clnperez

wglambert added the Issue label Jul 8, 2021

This was referenced Jul 9, 2021

The 1.21.1-alpine tag on Docker Hub is missing multple OS/ARCH entries nginxinc/docker-nginx#561

Closed

ppc64le and s390x archs are missing in Node 16 (current) Alpine3.14 images nodejs/docker-node#1525

Closed

timbru31 mentioned this issue Aug 9, 2021

build: upgrade Alpine base image to 3.14 timbru31/docker-node-alpine-git#137

Merged

yosifkit mentioned this issue Aug 20, 2021

Alpine based golang docker image for linux/s390x architecture missing docker-library/golang#383

Closed

tianon closed this as completed Aug 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

7.4.21-fpm-alpine3.14 missing linux/ppc64le and linux/s390x #1178

7.4.21-fpm-alpine3.14 missing linux/ppc64le and linux/s390x #1178

jokay commented Jul 4, 2021 •

edited

Loading

tianon commented Jul 6, 2021

tianon commented Aug 10, 2021

thaJeztah commented Aug 10, 2021

nirmannarang commented Aug 11, 2021

clnperez commented Aug 12, 2021

tianon commented Aug 12, 2021

tianon commented Aug 12, 2021

anujajakhade commented Aug 13, 2021

jokay commented Aug 13, 2021

thaJeztah commented Aug 13, 2021

clnperez commented Aug 13, 2021

tianon commented Aug 13, 2021

nirmannarang commented Aug 19, 2021

nirmannarang commented Aug 19, 2021

tianon commented Aug 19, 2021

tianon commented Aug 20, 2021

tianon commented Aug 20, 2021

thaJeztah commented Aug 20, 2021

7.4.21-fpm-alpine3.14 missing linux/ppc64le and linux/s390x #1178

7.4.21-fpm-alpine3.14 missing linux/ppc64le and linux/s390x #1178

Comments

jokay commented Jul 4, 2021 • edited Loading

7.4.20-fpm-alpine3.13

7.4.21-fpm-alpine3.14

tianon commented Jul 6, 2021

tianon commented Aug 10, 2021

thaJeztah commented Aug 10, 2021

nirmannarang commented Aug 11, 2021

clnperez commented Aug 12, 2021

tianon commented Aug 12, 2021

tianon commented Aug 12, 2021

anujajakhade commented Aug 13, 2021

jokay commented Aug 13, 2021

thaJeztah commented Aug 13, 2021

clnperez commented Aug 13, 2021

tianon commented Aug 13, 2021

nirmannarang commented Aug 19, 2021

nirmannarang commented Aug 19, 2021

tianon commented Aug 19, 2021

tianon commented Aug 20, 2021

tianon commented Aug 20, 2021

thaJeztah commented Aug 20, 2021

jokay commented Jul 4, 2021 •

edited

Loading