Pubsub Subscriber initialization crashes JVM on Alpine Linux with libc6-compat

[vitaly-sourikov]

Pubsub Subscriber initialization crashes JVM on Alpine Linux with libc6-compat. Here is the stacktrace from hs_err crash log:

Stack: [0x00007fad5f2d8000,0x00007fad5f3d8aa8],  sp=0x00007fad5f3d2668,  free space=1001k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x0000000000025426
C  [libio_grpc_netty_shaded_netty_tcnative_linux_x86_64646374909380874916.so+0x29585]  JNI_OnLoad_netty_tcnative+0x55
C  [libjava.so+0xda62]  Java_java_lang_ClassLoader_00024NativeLibrary_load+0xf2

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.lang.ClassLoader$NativeLibrary.load(Ljava/lang/String;Z)V+0
j  java.lang.ClassLoader.loadLibrary0(Ljava/lang/Class;Ljava/io/File;)Z+328
j  java.lang.ClassLoader.loadLibrary(Ljava/lang/Class;Ljava/lang/String;Z)V+48
j  java.lang.Runtime.load0(Ljava/lang/Class;Ljava/lang/String;)V+57
j  java.lang.System.load(Ljava/lang/String;)V+7
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(Ljava/lang/String;Z)V+5
v  ~StubRoutines::call_stub
J 1656  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007fad4ec3aa37 [0x00007fad4ec3a9c0+0x77]
J 1655 C1 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007fad4ec42554 [0x00007fad4ec41380+0x11d4]
J 587 C1 sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x00007fad4e8cfe0c [0x00007fad4e8cfd00+0x10c]
J 586 C1 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x00007fad4e8d05dc [0x00007fad4e8d01e0+0x3fc]
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run()Ljava/lang/Object;+53
v  ~StubRoutines::call_stub
J 697  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fad4e9597cf [0x00007fad4e959780+0x4f]
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(Ljava/lang/Class;Ljava/lang/String;Z)V+10
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(Ljava/lang/ClassLoader;Ljava/lang/String;Z)V+15
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(Ljava/lang/String;Ljava/lang/ClassLoader;)V+402
j  io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(Ljava/lang/ClassLoader;[Ljava/lang/String;)V+33
j  io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.loadTcNative()V+173
j  io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.<clinit>()V+152
v  ~StubRoutines::call_stub
j  io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider()Lio/grpc/netty/shaded/io/netty/handler/ssl/SslProvider;+0
j  io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContextBuilder;)Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContextBuilder;+1
j  io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forClient()Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContextBuilder;+3
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$NettyTranspo
rtFactory;Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContext;)V+23
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory$DefaultNettyTransportCreationParamsFilterFactory.<init>(Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$NettyTranspo
rtFactory;Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContext;Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$1;)V+3
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory.<init>(Lio/grpc/netty/shaded/io/grpc/netty/NettyChannelBuilder$TransportCreationParamsFilterFactory;Ljava/lang/Class;Ljava/u
til/Map;Lio/grpc/netty/shaded/io/grpc/netty/NegotiationType;Lio/grpc/netty/shaded/io/netty/handler/ssl/SslContext;Lio/grpc/netty/shaded/io/netty/channel/EventLoopGroup;IIIJJZLio/grpc/internal/TransportTra
cer;)V+45
j  io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.buildTransportFactory()Lio/grpc/internal/ClientTransportFactory;+59
j  io.grpc.internal.AbstractManagedChannelImplBuilder.build()Lio/grpc/ManagedChannel;+10
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel()Lio/grpc/ManagedChannel;+261
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel()Lcom/google/api/gax/rpc/TransportChannel;+19
j  com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel()Lcom/google/api/gax/rpc/TransportChannel;+52
j  com.google.cloud.pubsub.v1.Subscriber.doStart()V+25
j  com.google.api.core.AbstractApiService$InnerService.doStart()V+4
j  com.google.common.util.concurrent.AbstractService.startAsync()Lcom/google/common/util/concurrent/Service;+33
j  com.google.api.core.AbstractApiService.startAsync()Lcom/google/api/core/ApiService;+4
j  com.google.cloud.pubsub.v1.Subscriber.startAsync()Lcom/google/api/core/ApiService;+1

[pongad]

@garrettjonesgoogle Do you happen to know what our story is on Alpine support? @vitaly-sourikov seems to have libc installed.

[edencakir]

We're having the same issue. cc @ahmetsemsettinozdemirden

[chrisribble]

I have the same problem and have been using openjdk:8-jre-slim instead of openjdk:8-jre-alpine, which is unfortunate because it doubles the final size of my applications final image.

[sduskis]

We believe that the problem relates to grpc and SSL. You can see some information about the issue here. I haven't tried it myself, but as best I can tell, you would have to exclude io.grpc:grpc-netty-shaded, include io.grpc:grpc-netty and include netty-tcnative-alpine.

[chrisribble]

In case it helps anyone else: we just switched to gcr.io/distroless/java as our base image and it has most of the space benefits of Alpine (it's only 34MB larger) without the Musl incompatibility issues (since it uses glibc). The pubsub library seems to work fine with this image.

[ajaaym]

@chrisribble Here is how you can make this work.

[chrisribble]

@ajaaym the issue you linked to suggests installing libc-compat, but this issue clearly states that the issue happens with libc-compat installed.

It's been a while since we moved away from Alpine, but IIRC the issue is that the netty tcnative needs to be built specially for Alpine's SSL implementation (i.e. https://github.com/pires/netty-tcnative-alpine) and gRPC is pulling in a netty-tcnative implementation that uses JNI against the symbols in OpenSSL in a way that doesn't work on Alpine.

I played around with trying to get that netty-tcnative-alpine project to build and figured I would force the dependency resolution to use my built version instead of the version of netty-tcnative that was specified by the gRPC pom, but I couldn't get the netty-tcnative-alpine build to work so I gave up and switched to grc distroless (as noted above).