Error loading shared library libresolv.so.2 on Alpine in Go 1.20

[bep]

This is a reopening of #59304 -- which was closed with a not very helpful "working as intended" message.

Searching this GitHub repo for musl libc shows lots of existing relevant issues/code adaptations. People, like me, could maybe be lead to believe that Musl Libc was supported (and Alpine is a very popular Docker runtime). I'm not saying that this is a bug, but it's at least a documentation issue.

I understand that with the big number of issues in your back log, you need to somehow be very effective about this, but these short and cranky brush offs with "working as intended" is, I suspect, not having the effect you want in the long run.

[seankhliao]

As of 1.20, you're responsible for linking against C libraries properly if you build with CGo enabled.
Previously, go was distributed with prebuilt archives linked against ancient c library versions. As documented, we no longer do so.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

[ianlancetaylor]

@bep Can you show us how to recreate the problem? Right now I don't understand it, and I don't see any clear reproduction instructions in the earlier issues. Thanks.

[bep]

As of 1.20, you're responsible for linking against C libraries properly if you build with CGo enabled.

@seankhliao That I understand, but this is a C library that the Go standard library (this repo) is linking, I assume to do DNS lookups.

@ianlancetaylor I will put on my list and try to reproduce this in a simpler setup, but as this involves thinkering with Docker (not my favourite), it may take a few days. But please don't close this issue prematurely.

[Nuru]

You can see that minikube v1.30.0 suffers from this problem.

[ianlancetaylor]

@Nuru Is it possible that you are building with glibc and then running in a container that uses Alpine?

[Nuru]

@ianlancetaylor I am not building minikube at all. I am using the binary they distribute. See kubernetes/minikube#16228

[ianlancetaylor]

@Nuru Is it possible that the binary that they distribute was built with glibc, but that you are running it in a container that uses Alpine?

[afbjorklund]

The main difference is that due to the change above (in 1.20), there is now a new libresolv.so.2 depency:

$ go version ./minikube-1.29.0 
./minikube-1.29.0: go1.19.5
$ ldd minikube-1.29.0 
	linux-vdso.so.1 (0x00007ffcdc5f7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f312da7d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f312da5a000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f312dc86000)
$ go version ./minikube-1.30.0 
./minikube-1.30.0: go1.20.2
$ ldd minikube-1.30.0 
	linux-vdso.so.1 (0x00007ffc583db000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f40eb1b2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f40eb18f000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f40eb173000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f40eb3bb000)

The glibc compatibility in Alpine (gcompat) needs to be updated to handle this too - or run a musl binary....

[Julio-Guerra]

@Nuru Is it possible that the binary that they distribute was built with glibc, but that you are running it in a container that uses Alpine?

That's exactly it @ianlancetaylor. We have a "smoke test" now failing with this issue where we build a simple hello world HTTP API using the default golang docker image and copy the resulting binary to the default alpine image. (We introduced that smoke tests because a lot of our package users compile their Go programs this way and wanted to catch any issues asap.)

Something like:

FROM golang:1.20 AS build
WORKDIR /app
COPY . .
RUN go build -o program .

FROM alpine
COPY --from=build /app/program .
RUN ./program

See the output we now get when starting it:

Error loading shared library libresolv.so.2: No such file or directory (needed by /usr/local/bin/go-dvwa)
[...]
Error relocating /usr/local/bin/go-dvwa: __res_search: symbol not found

So we are basically saying that starting from go1.20, the portability we used to have between glibc-based compilations, and musl-based ones, is no longer possible out of the box, without directly compiling with a "musl-gcc".

[afbjorklund]

If you want a "real" alpine binary (instead of troubleshooting the glibc compatibility), you can build with CC=musl-gcc*.

* You do need to install the musl-tools package

For minikube we didn't want to ship double (or static) binaries, but maybe someone else want dynamic musl binaries ?

$ CC=musl-gcc make
$ musl-ldd out/minikube 
	/lib/ld-musl-x86_64.so.1 (0x7f6b07e40000)
	libc.so => /lib/ld-musl-x86_64.so.1 (0x7f6b07e40000)

[ianlancetaylor]

It's likely that a binary linked against glibc and dynamically linked against Alpine does not actually need a libresolv.so.2 shared library. So a workaround might be to add a dummy shared library with that name, or to link libc.so.6 to that name.

[afbjorklund]

There is a libgcompat.so library, that exports the needed symbols from the real libc. So it would be possible to fix this in the Alpine distribution, so that the default glibc compatibility layer handles this new "libresolv" like it handles "libpthread"

---

Note that there is no libc.so.6 either, it is also replaced.

/ # ldd /usr/bin/minikube 
	/lib64/ld-linux-x86-64.so.2 (0x7f8c8161d000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f8c8161d000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f8c8161d000)
Error loading shared library libresolv.so.2: No such file or directory (needed by /usr/bin/minikube)

ln -sf ../../lib/libgcompat.so.0 /usr/lib/libresolv.so.2

/ # ldd /usr/bin/minikube 
	/lib64/ld-linux-x86-64.so.2 (0x7fbb07446000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fbb07446000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fbb07446000)
	libresolv.so.2 => /usr/lib/libresolv.so.2 (0x7fbb07433000)
	libucontext.so.1 => /lib/libucontext.so.1 (0x7fbb0742e000)
	libobstack.so.1 => /usr/lib/libobstack.so.1 (0x7fbb07429000)

[Julio-Guerra]

If you want a "real" alpine binary (instead of troubleshooting the glibc compatibility), you can build with CC=musl-gcc*.

Thanks, that's exactly what we just did and it works. We will probably provide this answer to our users raising this specific issue from now (cross-compiling is the still in the usual "bring-your-own-toolchain" case).

[afbjorklund]

Do you have a link to the Alpine issue, about this ? I guess it would be in their interest, to be able to run go1.20 binaries

[Nuru]

I opened this issue with Alpine.

[161424]

apk add gcompat
gcompat include libresolv.so.2
https://pkgs.alpinelinux.org/contents?file=libresolv.so.2&path=&name=&branch=edge

[cyw3]

apk add gcompat gcompat include libresolv.so.2 https://pkgs.alpinelinux.org/contents?file=libresolv.so.2&path=&name=&branch=edge

I also meet this problem. After following your repair steps, it was not successful and another error message appeared:

Segmentation fault (core dumped)

But I compiled the program using go1.19 and then executed it successfully under alpine.