__getauxval symbols cannot be found after 2.0.66 on musl Linux

[BewareMyPower]

Summary

With 2.0.66, the __getauxval symbol can be found after installing the gcompat dependency. However, 2.0.67 to 2.0.69 all failed.

How to reproduce

Run an alpine:3.20 container on macOS (my OS version is Ventura 13.6.4). Then go to the container and download these 4 JARs:

netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar
netty-tcnative-boringssl-static-2.0.67.Final-linux-aarch_64.jar
netty-tcnative-boringssl-static-2.0.68.Final-linux-aarch_64.jar
netty-tcnative-boringssl-static-2.0.69.Final-linux-aarch_64.jar

Install the dependencies:

apk add gcc
apk add gcompat

Take 2.0.69 for example:

~ # unzip -q netty-tcnative-boringssl-static-2.0.69.Final-linux-aarch_64.jar 
~ # ldd META-INF/native/libnetty_tcnative_linux_aarch_64.so 
	/lib/ld-musl-aarch64.so.1 (0xffffa888a000)
	librt.so.1 => /lib/ld-musl-aarch64.so.1 (0xffffa888a000)
	libdl.so.2 => /lib/ld-musl-aarch64.so.1 (0xffffa888a000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xffffa862a000)
	libc.so.6 => /lib/ld-musl-aarch64.so.1 (0xffffa888a000)
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: __getauxval: symbol not found

As you can see, the __getauxval symbol is not found.

The outputs of 2.0.68 is the same, while 2.0.67 is worse:

~ # unzip -q netty-tcnative-boringssl-static-2.0.67.Final-linux-aarch_64.jar 
~ # ldd META-INF/native/libnetty_tcnative_linux_aarch_64.so 
	/lib/ld-musl-aarch64.so.1 (0xffff9f41b000)
	librt.so.1 => /lib/ld-musl-aarch64.so.1 (0xffff9f41b000)
	libdl.so.2 => /lib/ld-musl-aarch64.so.1 (0xffff9f41b000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xffff9f1cd000)
	libc.so.6 => /lib/ld-musl-aarch64.so.1 (0xffff9f41b000)
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: __getauxval: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZdlPvm: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: _ZTVN10__cxxabiv120__si_class_type_infoE: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_aarch_64.so: __gxx_personality_v0: symbol not found

2.0.66 works well:

~ # unzip -q netty-tcnative-boringssl-static-2.0.66.Final-linux-aarch_64.jar 
~ # ldd META-INF/native/libnetty_tcnative_linux_aarch_64.so 
	/lib/ld-musl-aarch64.so.1 (0xffff9c6d0000)
	librt.so.1 => /lib/ld-musl-aarch64.so.1 (0xffff9c6d0000)
	libcrypt.so.1 => /lib/libcrypt.so.1 (0xffff9c4a0000)
	libdl.so.2 => /lib/ld-musl-aarch64.so.1 (0xffff9c6d0000)
	libc.so.6 => /lib/ld-musl-aarch64.so.1 (0xffff9c6d0000)
	libucontext.so.1 => /lib/libucontext.so.1 (0xffff9c47f000)
	libobstack.so.1 => /usr/lib/libobstack.so.1 (0xffff9c45e000)
~ # strings /lib/libcrypt.so.1 | grep __getauxval
__getauxval
~ # ls -l /lib/libcrypt.so.1 
lrwxrwxrwx    1 root     root            15 Dec 12 06:34 /lib/libcrypt.so.1 -> libgcompat.so.0

Not sure why __getauxval cannot be found from libgcompat.so in 2.0.68 or later.

[lhotari]

Since it breaks in 2.0.67, diff between 2.0.66 and 2.0.67: Diff between 2.0.66 and 2.0.67: netty-tcnative-parent-2.0.66.Final...netty-tcnative-parent-2.0.67.Final .

[normanmaurer]

I kind of suspect this is related to updating to latest boringssl and static compilation.

[lhotari]

I kind of suspect this is related to updating to latest boringssl and static compilation.

@normanmaurer Are there plans to provide native libraries compiled explicitly for musl on Linux? That could be a more stable option in the long term.

[lhotari]

I digged more into this. It looks like checking linking requires using LD_PRELOAD=/lib/libgcompat.so.0 ldd as explained in comment grpc/grpc-java#10930 (comment) and grpc/grpc-java#10930 (comment) .

It appears that 2.0.69.Final succeed on both linux/arm64 and linux/amd64. This is the script I used for testing:

function test_tcnative() {
    local TCNATIVE_VERSION=$1
    local PLATFORM=$2
    local ALPINE_VERSION=3.20
    
    echo "--------------------------------------------------------------------"
    echo "Testing ${TCNATIVE_VERSION} on $PLATFORM"
    
    docker run --platform $PLATFORM --env TCNATIVE_VERSION=$TCNATIVE_VERSION --rm -it alpine:$ALPINE_VERSION sh -c '
        set -e
        ARCH=$(arch)
        [ "$ARCH" = "aarch64" ] && ARCH="aarch_64"
        
        # Install dependencies
        apk update
        apk add gcompat libgcc
        
        # Download and extract
        wget -q https://repo1.maven.org/maven2/io/netty/netty-tcnative-boringssl-static/${TCNATIVE_VERSION}/netty-tcnative-boringssl-static-${TCNATIVE_VERSION}-linux-$ARCH.jar
        unzip -q netty-tcnative-boringssl-static-${TCNATIVE_VERSION}-linux-$ARCH.jar
        
        # Test library loading
        echo "Checking library dependencies:"
        LD_PRELOAD=/lib/libgcompat.so.0 ldd META-INF/native/libnetty_tcnative_linux_$ARCH.so
    '
}
test_tcnative 2.0.69.Final linux/arm64
test_tcnative 2.0.69.Final linux/amd64
test_tcnative 2.0.66.Final linux/arm64
test_tcnative 2.0.66.Final linux/amd64