misc/cgo/testsigfwd: Test fails starting with 1.19 on x86 Alpine Linux

[nmeum]

What version of Go are you using (go version)?

$ go version
go version go1.19 linux/386

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="386"
GOBIN="/home/buildozer/src/aports/community/go/src/go/bin"
GOCACHE="/home/buildozer/.cache/go-build"
GOENV="/home/buildozer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS="-buildmode=pie -modcacherw"
GOHOSTARCH="386"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/buildozer/.local/share/go/modcache"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/buildozer/src/aports/community/go/src"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/home/buildozer/src/aports/community/go/src/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/home/buildozer/src/aports/community/go/src/go/pkg/tool/linux_386"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GO386="sse2"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m32 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1188913618=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Compiled Go from source using ./make.bash on x86 Alpine Linux Edge with 365ca69 backported. Afterwards, invoked the test suite using ./run.bash -run='!(^cgo_test$)' (cgo tests are known to fail on Alpine due to #39857).

What did you expect to see?

Successful test run.

What did you see instead?

A test failure of ../misc/cgo/testsigfwd:

##### ../misc/cgo/testsigfwd
# misc/cgo/testsigfwd
cgo: cannot load DWARF output from $WORK/b001//_cgo_.o: zlib: invalid header
2022/08/12 18:10:59 Failed: exit status 2

Full build log: alpine-linux-x86-edge-go-1.19.txt

I am aware that Alpine/musl is not fully supported by the test suite (see #19938 etc.), however, I am still reporting this as the test case passed with Go 1.18 (and also passes on all other architectures supported by Alpine). Furthermore, this seems to be related to #52919 and #54313 as the test case passes successfully with the following change (as discussed in the referenced issues, Alpine uses -fstack-protector by default):

diff -upr go.orig/misc/cgo/testsigfwd/main.go go/misc/cgo/testsigfwd/main.go
--- go.orig/misc/cgo/testsigfwd/main.go	2022-08-12 20:28:17.222319036 +0200
+++ go/misc/cgo/testsigfwd/main.go	2022-08-12 20:28:45.752384953 +0200
@@ -7,7 +7,7 @@ package main
 import "fmt"
 
 /*
-#cgo CFLAGS: -pthread
+#cgo CFLAGS: -pthread -fno-stack-protector
 #cgo LDFLAGS: -pthread
 
 #include <signal.h>

If -fstack-protector is generally unsupported on all Go architectures, then I would suggest having cgo add that to the compiler flags by default unconditionally. If it is just about this test case it would also be sufficient to commit the diff from above (with this diff the run.bash invocation from above succeeds). However, I don't fully understand why it worked prior to Go 1.19 without -fno-stack-protector.

[thanm]

@golang/runtime

[ianlancetaylor]

This may be fixed on tip by https://go.dev/cl/421935. If so, we could consider backporting that to 1.19.

[nmeum]

This may be fixed on tip by https://go.dev/cl/421935. If so, we could consider backporting that to 1.19.

That is the same as 365ca69, no?

Because if so I already have that backported (see What did you do? in my original issue description).

[ianlancetaylor]

Sorry for the confusion.

I agree that we shouldn't start adding -fno-stack-protector everywhere, and we should figure out what changed to make it required.

Looking further at the issue I don't understand why not adding it would lead to the "invalid header" error that you mention.

[nmeum]

Looking further at the issue I don't understand why not adding it would lead to the "invalid header" error that you mention.

git-bisect(1) tells me that testsigfwd started failing on Alpine Linux x86 with commit
d34287a prior to that commit the test passes fine without -fno-stack-protector.

I don't understand though how -fstack-protector relates to ELF compression.

Any ideas?

[nmeum]

git-bisect(1) tells me that testsigfwd started failing on Alpine Linux x86 with commit d34287a prior to that commit the test passes fine without -fno-stack-protector.

I don't understand though how -fstack-protector relates to ELF compression.

@mengzhuo since you authored the aforementioned commit, do you have an idea how this might be related?

I was also able to extract the $WORK/b001//_cgo_.o object file, which the go build invocation from above complains about. Furthermore, I also extracted this object file in a build with -fno-stack-protector to make it easier to compare the two:

• With -fstack-protector (zlib: invalid header error): _cgo_with_stack_protector.o
• With -fno-stack-protector (no error): _cgo_without_stack_protector.o

Maybe this helps with debugging the error regarding compressed dwarf information?

I am personally not sure what to look for in these files though.

[mengzhuo]

I had no idea how they are related, sorry.
By the way, I can't reproduce this on the latest Alpine release.

cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.16.2
PRETTY_NAME="Alpine Linux v3.16"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
Using built-in specs.

gcc -v
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/11.2.1/lto-wrapper
Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-11.2.1_git20220219/configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --build=x86_64-alpine-linux-musl --host=x86_64-alpine-linux-musl --target=x86_64-alpine-linux-musl --with-pkgversion='Alpine 11.2.1_git20220219' --enable-checking=release --disable-fixed-point --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-werror --disable-symvers --enable-__cxa_atexit --enable-default-pie --enable-default-ssp --enable-cloog-backend --enable-languages=c,c++,d,objc,go,fortran,ada,jit --disable-libssp --disable-libmpx --disable-libmudflap --disable-libsanitizer --enable-shared --enable-threads --enable-tls --enable-host-shared --with-system-zlib --with-linker-hash-style=gnu
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 11.2.1 20220219 (Alpine 11.2.1_git20220219)

alpine:~/godev/src# ./run.bash -run='testsigfwd'

##### Building packages and commands.

##### Test execution environment.
# GOARCH: amd64
# CPU: Intel(R) Xeon(R) W-2150B CPU @ 3.00GHz
# GOOS: linux
# OS Version: Linux 5.15.62-0-lts #1-Alpine SMP Tue, 23 Aug 2022 13:50:24 +0000 x86_64

##### ../misc/cgo/testsigfwd
ok      testsigfwd

ALL TESTS PASSED (some were excluded)

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/godev"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/godev/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/godev/src/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1767409198=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.19 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.19
uname -sr: Linux 5.15.62-0-lts

[nmeum]

By the way, I can't reproduce this on the latest Alpine release.

Thanks for pointing this out. We ran into this on Alpine Linux Edge (the development version) since cgo is involved here this might then be related to a specific binutils/gcc/… version. On Edge we use binutils 2.39 (2.38 in the latest Alpine release) and gcc 12.1.1 (11.2.1 on the latest Alpine release).

[nmeum]

I had no idea how they are related, sorry. By the way, I can't reproduce this on the latest Alpine release.

@mengzhuo while debugging this further I just noticed that you tried to reproduce this issue on amd64 (at least GOARCH is set to amd64 in your test environment output). However, as described above this issue can only be reproduced on x86 (i.e. GOARCH=i386) even on Alpine Linux Edge.

The code that seems to be emitting the error message is:

go/src/debug/elf/file.go

Lines 1183 to 1189 in d34287a

	r, err := zlib.NewReader(bytes.NewBuffer(b[s.compressionOffset:]))
	if err != nil {
	return nil, err
	}
	if _, err := io.ReadFull(r, dbuf); err != nil {
	return nil, err
	}

I added the following printf debug output:

diff --git a/src/debug/elf/file.go b/src/debug/elf/file.go
index aff2b00aae..6af82828d0 100644
--- a/src/debug/elf/file.go
+++ b/src/debug/elf/file.go
@@ -1253,6 +1253,10 @@ func (f *File) DWARF() (*dwarf.Data, error) {
                        }
                }
                if dlen > 0 {
+                       fmt.Printf("Section: %s\n", s.Name)
+                       fmt.Printf("b = %x\n", b)
+                       fmt.Printf("compressionOffset = %v\n", s.compressionOffset)
+                       fmt.Printf("decoding: %x\n", b[s.compressionOffset:])
                        r, err := zlib.NewReader(bytes.NewBuffer(b[s.compressionOffset:]))
                        if err != nil {
                                return nil, err

With this patch the output is:

Section: .debug_ranges
b = 010000000600000028000000310000003400000036000000000000000000000000000000b500000000000000820000000000000000000000
compressionOffset = 12
decoding: 310000003400000036000000000000000000000000000000b500000000000000820000000000000000000000
cgo: cannot load DWARF output from $WORK/b001//_cgo_.o: zlib: invalid header

And 0x31 is indeed not a valid zlib header.

In fact, it seems that the .debug_ranges section is not compressed at all?

$ objdump --dwarf _cgo_with_stack_protector.o
Contents of the .debug_ranges section:


    Offset   Begin    End
    00000000 00000001 00000006
    00000000 00000028 00000031
    00000000 00000034 00000036
    00000000 <End of list>
    00000020 00000000 000000b5
    00000020 00000000 00000082
    00000020 <End of list

I am not sure though if .debug_ranges section should be compressed or if Go should not attempt to decompress is as it is not compressed. Any hints on that?

The file is available here: https://dev.alpinelinux.org/~nmeum/go-issue-%2354422/_cgo_with_stack_protector.o

[thanm]

I note that when the Go linker does DWARF compression, if a given section doesn't benefit from compression it will leave it as is (e.g. see https://go.googlesource.com/go/+/2551324cd01b295915c10c6d3d06625676401610/src/cmd/link/internal/ld/dwarf.go#2225). Could this be what's happening here?

[nmeum]

@mengzhuo is there any chance that you can take another look at this given that you tried to reproduce the issue on x86_64 and not x86? Maybe the additional information provided in #54422 (comment) is also useful to you.

[bcmills]

@nmeum, is this still an issue? It appears that the Alpine patch was deleted in https://git.alpinelinux.org/aports/commit/community/go?id=57b7f2005454faff9f036f04c11a7c78a0281948 for Go 1.20.

[nmeum]

@nmeum, is this still an issue?

I can no longer reproduce this so yea, let's close this for now.