netty-tcnative 2.0.61 dependency might break when used with Alpine

[henrik242]

I got this when running the Google SDK (which depends on grpc-netty-shaded 1.61.0) with Bellsoft's alpine docker image of jdk 21:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000021866, pid=7, tid=42
#
# JRE version: OpenJDK Runtime Environment (21.0.2+14) (build 21.0.2+14-LTS)
# Java VM: OpenJDK 64-Bit Server VM (21.0.2+14-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libio_grpc_netty_shaded_netty_tcnative_linux_x86_6415938253211895176092.so+0x28f83]  _init+0x79cb
#
# Core dump will be written. Default location: /core.%e.7.%t
#
# An error report file with more information is saved as:
# /app/hs_err_pid7.log
[8.331s][warning][os] Loading hsdis library failed

This seems to have been fixed in netty-tcnative 2.0.62.

Related:

• https://gitlab.alpinelinux.org/alpine/aports/-/issues/15582
• netty/netty-tcnative@netty-tcnative-parent-2.0.61.Final...netty-tcnative-parent-2.0.62.Final

• grpc-java/gradle/libs.versions.toml

  Line 6 in 0d39c2c

  nettytcnative = '2.0.61.Final'

• grpc-java/repositories.bzl

  Line 38 in 0d39c2c

  "io.netty:netty-tcnative-classes:2.0.61.Final",

[ejona86]

I'm going to blame Alpine much sooner than Java 21. Especially for a crash in native code. I assume you meant BellSoft. It looks like the package you used was most likely bellsoft-jre21.0.2+14-linux-x64-musl.

netty/netty-tcnative#789 is equally similar to the alpine aports issue, but both are in a different function.

This is almost certainly a missing symbol. These can generally be fixed by using LD_PRELOAD.

See #10096 (comment) for the last time I looked into one.

[henrik242]

@ejona86 Thanks! LD_PRELOAD fixed it for us.

[normanmaurer]

I think this might be fixed by netty-tcnative 2.0.63.Final without the need of LD_PRELOAD

[ejona86]

@normanmaurer, looks like 2.0.63 still needs LD_PRELOAD of gcompat (because the java in alpine doesn't use the gcompat linker, but that varies depending on which java you use). And now there's missing sys_siglist, which puts us back to #10096 (comment) . I didn't find a library to provide that symbol before.

$ docker run --rm -it docker.io/library/alpine:latest
/ # wget https://search.maven.org/remotecontent?filepath=io/netty/netty-tcnative-boringssl-static/2.0.63.Final/netty-tcnative-boringssl-static-2.0.63.Final-linux-x86_64.jar
/ # unzip netty-tcnative-boringssl-static-2.0.63.Final-linux-x86_64.jar 
/ # ldd META-INF/native/libnetty_tcnative_linux_x86_64.so
	/lib/ld-musl-x86_64.so.1 (0x7fa155c5d000)
	librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fa155c5d000)
	libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7fa155c5d000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7fa155c5d000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7fa155c5d000)
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __isnan: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __isinf: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: __strdup: symbol not found
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: sys_siglist: symbol not found
/ # apk add gcompat
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
(1/3) Installing musl-obstack (1.2.3-r2)
(2/3) Installing libucontext (1.2-r2)
(3/3) Installing gcompat (1.1.0-r4)
OK: 8 MiB in 18 packages
/ # LD_PRELOAD=/lib/libgcompat.so.0 ldd META-INF/native/libnetty_tcnative_linux_x86_64.so
	/lib/ld-musl-x86_64.so.1 (0x7f835ba7f000)
	/lib/libgcompat.so.0 => /lib/libgcompat.so.0 (0x7f835ba6d000)
	librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f835ba7f000)
	libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7f835ba7f000)
	libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7f835ba7f000)
	libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f835ba7f000)
	libucontext.so.1 => /lib/libucontext.so.1 (0x7f835ba68000)
	libobstack.so.1 => /usr/lib/libobstack.so.1 (0x7f835ba63000)
Error relocating META-INF/native/libnetty_tcnative_linux_x86_64.so: sys_siglist: symbol not found

[ejona86]

Seems like this is resolved, as LD_PRELOAD is required on Alpine. Seems we may be even better off now than after the planned upgrade to 2.0.63. If things aren't good, comment, and it can be reopened.

I've said it before, but I recommend against Alpine. We don't officially support or test against it. If it happens to work, fine, but there are no guarantees. (In the olden days you had to compile tcnative yourself. At some point gcompat got good enough you no longer needed to. But it is fickle.)

[codefromthecrypt]

I opened a follow-up on tcnative as this is also used by armeria (I didn't think to check here). Thanks for all the tips @ejona86! netty/netty-tcnative#853