Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available #5369

Closed
ankitshubham97 opened this issue Feb 17, 2019 · 35 comments

Comments

@ankitshubham97
Copy link

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.13.1

What did you expect to see?

The jar should be running fine.

I am using java 8 to build an executable jar. Below is the java version:

$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-2~14.04-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

I am using gradle 3.4.1 to generate the jar as follows:
/opt/gradle-3.4.1/bin/gradle jar -Dorg.gradle.java.home=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
Below are the dependencies defined in the module-level build.gradle file:

dependencies {
  compile files("$TOOLCHAIN_VERSION_DIR/lib/commons-io-2.6.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/grpc-all.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/java-protobuf.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-core-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-slf4j-impl-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/netty-tcnative-boringssl-static-2.0.20.Final.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/netty-tcnative-2.0.20.Final.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/picocli-3.8.2.jar")
  compile group: 'com.google.guava', name: 'guava', version: '20.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/javassist-3.19.0-GA.jar")
  compile project(':annotation')
}

After building, I am running the jar on an AIX7.2 machine as follows:
/usr/java8_64/jre/bin/java -jar agent-1.0.jar

The java version on the AIX machine is as follows:

$ /usr/java8_64/jre/bin/java -version           
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.26 - pap6480sr5fp26-20181115_03(SR5 FP26))
IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64-Bit Compressed References 20181106_401576 (JIT enabled, AOT enabled)
OpenJ9   - fde1d6f
OMR      - d8c3617
IBM      - 5c4a9f0)
JCL - 20181022_01 based on Oracle jdk8u191-b26

But I am getting the following error while running:

Feb 17, 2019 3:11:33 AM io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: netty-tcnative unavailable (this may be normal)
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_aix_ppc_64, netty_tcnative_ppc_64, netty_tcnative]
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:93)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:430)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:242)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:456)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:55)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:91)
        at com.ankit.agents.AgentMain.main(AgentMain.java:132)
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_aix_ppc_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_aix_ppc_64.a
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_ppc_64
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative_ppc_64.a
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)
                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_tcnative.a
                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
                        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: netty_tcnative (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                                ... 11 more

Feb 17, 2019 3:11:33 AM io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Conscrypt not found (this may be normal)
Feb 17, 2019 3:11:33 AM io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Jetty ALPN unavailable (this may be normal)
java.lang.ClassNotFoundException: org.eclipse.jetty.alpn.ALPN
        at java.lang.Class.forNameImpl(Native Method)
        at java.lang.Class.forName(Class.java:403)
        at io.grpc.netty.JettyTlsUtil.isJettyAlpnConfigured(JettyTlsUtil.java:64)
        at io.grpc.netty.JettyTlsUtil.getJettyAlpnUnavailabilityCause(JettyTlsUtil.java:75)
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:255)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:456)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:55)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:91)
        at com.ankit.agents.AgentMain.main(AgentMain.java:132)

Exception in thread "main" java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:456)
        at io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:55)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:91)
        at com.ankit.agents.AgentMain.main(AgentMain.java:132)

However, it is running fine on an Ubuntu machine. I am running the jar as follows:
/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java -jar agent-1.0.jar

The java version here is:

$ /usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/bin/java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-2~14.04-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

What is the problem here and how to solve this?

@ST-DDT
Copy link
Contributor

ST-DDT commented Feb 18, 2019

You need to have an (exact) version match for the netty and tcnative versions to work together.
See also the table at https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

grpc-version: 1.13.1
tcnative-version: 2.0.20.Final (but should be 2.0.8.Final)

Off-Topic: Could we add a headline/link anchor there so that the table is easier to link?

@ankitshubham97
Copy link
Author

@ST-DDT I tried to use grpc 1.18.0 which matches tcnative-version 2.0.20.Final. But it is giving the same error. My guess is that since I have built it on an ubuntu machine using openjdk, it is creating problem for IBM JRE. By the way, is there any option other than netty-tcnative that can be used in my use case? Are you aware of some? I would appreciate any help.

@ST-DDT
Copy link
Contributor

ST-DDT commented Feb 18, 2019

Usually I just use grpc-netty with netty-tcnative-boringssl-static no need for an extra netty-tcnative.

As an alternative have you tried using grpc-netty-shaded? AFAICT this has the necessary/compatibile grpc/netty/tcnative libraries included.

@ankitshubham97
Copy link
Author

@ST-DDT Changed the dependency in the build.gradle to following:

dependencies {
  compile files("$TOOLCHAIN_VERSION_DIR/lib/commons-io-2.6.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/grpc-all-1.18.0.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/java-protobuf.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-core-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-slf4j-impl-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/netty-tcnative-boringssl-static-2.0.20.Final.jar")
  compile group: 'io.grpc', name: 'grpc-netty', version: '1.18.0'
  compile group: 'io.grpc', name: 'grpc-netty-shaded', version: '1.18.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/picocli-3.8.2.jar")
  compile group: 'com.google.guava', name: 'guava', version: '20.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/javassist-3.19.0-GA.jar")
  compile project(':annotation')
}

It is still not working. I think openjdk is not build aix native libraries (as seen in the log).

@khodayarj
Copy link

I've been trying to solve the same issue for two weeks, I've tried this also, it happens during configure stackdriver logging configuration in a cluster
RuntimeException in Action for tag [appender] com.google.cloud.logging.LoggingException: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available

@ST-DDT
Copy link
Contributor

ST-DDT commented Feb 18, 2019

Can you paste the logs if you just use the dependencies as described in the readme (+plus the non-grpc/non-netty/non-tcnative libs)?

compile 'io.grpc:grpc-netty-shaded:1.18.0'
compile 'io.grpc:grpc-protobuf:1.18.0'
compile 'io.grpc:grpc-stub:1.18.0'

@khodayarj
Copy link

khodayarj commented Feb 18, 2019

this is the error log when spring boot app wants to start :

ERROR in ch.qos.logback.core.joran.spi.Interpreter@23:16 - RuntimeException in Action for tag [appender] com.google.cloud.logging.LoggingException: java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
        at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:169)
        at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:82)
        at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
        at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:117)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:264)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:237)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:200)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:173)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74)
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:338)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:297) ....

this is the dependencies tree:

 --- maven-dependency-plugin:3.0.2:tree (default-cli) @ bootservice ---
 com.kody.c:boot-service:jar:0.0.1-SNAPSHOT
 +- org.springframework.boot:spring-boot-starter-actuator:jar:2.0.8.RELEASE:compile
 |  +- org.springframework.boot:spring-boot-starter:jar:2.0.8.RELEASE:compile
 |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.8.RELEASE:compile
 |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
 |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
 |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
 |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
 |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
 |  +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:2.0.8.RELEASE:compile
 |  |  +- org.springframework.boot:spring-boot-actuator:jar:2.0.8.RELEASE:compile
 |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.8:compile
 |  \- io.micrometer:micrometer-core:jar:1.0.9:compile
 |     +- org.hdrhistogram:HdrHistogram:jar:2.1.10:compile
 |     \- org.latencyutils:LatencyUtils:jar:2.0.3:compile
 +- org.springframework.boot:spring-boot-starter-web:jar:2.0.8.RELEASE:compile
 |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.8.RELEASE:compile
 |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.8:compile
 |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.8:compile
 |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.8.RELEASE:compile
 |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.37:compile
 |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.37:compile
 |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.37:compile
 |  +- org.hibernate.validator:hibernate-validator:jar:6.0.14.Final:compile
 |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
 |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
 |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
 |  +- org.springframework:spring-web:jar:5.0.12.RELEASE:compile
 |  \- org.springframework:spring-webmvc:jar:5.0.12.RELEASE:compile
 |     \- org.springframework:spring-expression:jar:5.0.12.RELEASE:compile
 +- org.springframework.security.oauth.boot:spring-security-oauth2-autoconfigure:jar:2.0.0.RELEASE:compile
 |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
 |  +- org.springframework.boot:spring-boot:jar:2.0.8.RELEASE:compile
 |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.8.RELEASE:compile
 |  \- org.springframework.security:spring-security-jwt:jar:1.0.9.RELEASE:compile
 |     \- org.bouncycastle:bcpkix-jdk15on:jar:1.56:compile
 +- org.springframework.boot:spring-boot-starter-test:jar:2.0.8.RELEASE:test
 |  +- org.springframework.boot:spring-boot-test:jar:2.0.8.RELEASE:test
 |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.0.8.RELEASE:test
 |  +- com.jayway.jsonpath:json-path:jar:2.4.0:test
 |  |  \- net.minidev:json-smart:jar:2.3:test
 |  |     \- net.minidev:accessors-smart:jar:1.2:test
 |  +- junit:junit:jar:4.12:test
 |  +- org.assertj:assertj-core:jar:3.9.1:test
 |  +- org.mockito:mockito-core:jar:2.15.0:test
 |  |  +- net.bytebuddy:byte-buddy:jar:1.7.11:test
 |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
 |  |  \- org.objenesis:objenesis:jar:2.6:test
 |  +- org.hamcrest:hamcrest-core:jar:1.3:test
 |  +- org.hamcrest:hamcrest-library:jar:1.3:test
 |  +- org.skyscreamer:jsonassert:jar:1.5.0:test
 |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
 |  +- org.springframework:spring-core:jar:5.0.12.RELEASE:compile
 |  |  \- org.springframework:spring-jcl:jar:5.0.12.RELEASE:compile
 |  +- org.springframework:spring-test:jar:5.0.12.RELEASE:test
 |  \- org.xmlunit:xmlunit-core:jar:2.5.1:test
 +- org.springframework.boot:spring-boot-starter-aop:jar:2.0.8.RELEASE:compile
 |  +- org.springframework:spring-aop:jar:5.0.12.RELEASE:compile
 |  \- org.aspectj:aspectjweaver:jar:1.8.13:compile
 +- org.springframework.cloud:spring-cloud-gcp-starter-logging:jar:1.0.0.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-gcp-starter:jar:1.0.0.RELEASE:compile
 |  |  +- org.springframework.cloud:spring-cloud-gcp-core:jar:1.0.0.RELEASE:compile
 |  |  \- org.springframework.cloud:spring-cloud-gcp-autoconfigure:jar:1.0.0.RELEASE:compile
 |  +- ch.qos.logback.contrib:logback-json-classic:jar:0.1.5:compile
 |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
 |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
 |  |  \- ch.qos.logback.contrib:logback-json-core:jar:0.1.5:compile
 |  +- com.google.cloud:google-cloud-logging-logback:jar:0.56.0-alpha:compile
 |  |  \- com.google.cloud:google-cloud-logging:jar:1.38.0:compile
 |  |     +- com.google.api.grpc:proto-google-cloud-logging-v2:jar:0.21.0:compile
 |  |     \- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
 |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
 +- io.netty:netty-tcnative-boringssl-static:jar:2.0.20.Final:compile
 +- org.springframework.cloud:spring-cloud-gcp-starter-trace:jar:1.1.0.M3:compile
 |  +- org.springframework.cloud:spring-cloud-starter-sleuth:jar:2.0.1.RELEASE:compile
 |  |  \- org.springframework.cloud:spring-cloud-sleuth-core:jar:2.0.1.RELEASE:compile
 |  |     +- org.aspectj:aspectjrt:jar:1.8.13:compile
 |  |     +- io.zipkin.brave:brave-context-log4j2:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-spring-web:jar:5.1.4:compile
 |  |     |  \- io.zipkin.brave:brave-instrumentation-http:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-spring-rabbit:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-kafka-clients:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-httpclient:jar:5.1.4:compile
 |  |     +- io.zipkin.brave:brave-instrumentation-httpasyncclient:jar:5.1.4:compile
 |  |     \- io.zipkin.brave:brave-instrumentation-spring-webmvc:jar:5.1.4:compile
 |  |        \- io.zipkin.brave:brave-instrumentation-servlet:jar:5.1.4:compile
 |  +- io.zipkin.gcp:zipkin-sender-stackdriver:jar:0.8.3:compile
 |  |  +- io.zipkin.gcp:zipkin-translation-stackdriver:jar:0.8.3:compile
 |  |  +- io.zipkin.reporter2:zipkin-reporter:jar:2.7.10:compile
 |  |  +- com.google.api.grpc:grpc-google-devtools-cloudtrace-v1:jar:0.1.1:compile
 |  |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.1:compile
 |  |  +- io.grpc:grpc-core:jar:1.18.0:compile
 |  |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
 |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.2.0:compile
 |  |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
 |  |  |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
 |  |  |  +- org.checkerframework:checker-compat-qual:jar:2.5.2:compile
 |  |  |  +- io.opencensus:opencensus-api:jar:0.18.0:compile
 |  |  |  \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.18.0:compile
 |  |  +- io.grpc:grpc-protobuf:jar:1.18.0:compile
 |  |  |  \- io.grpc:grpc-protobuf-lite:jar:1.18.0:compile
 |  |  +- io.zipkin.zipkin2:zipkin:test-jar:tests:2.11.7:compile
 |  |  \- io.zipkin.zipkin2:zipkin:jar:2.11.7:compile
 |  +- io.zipkin.gcp:brave-propagation-stackdriver:jar:0.8.3:compile
 |  |  \- io.zipkin.brave:brave:jar:5.4.3:compile
 |  +- org.springframework.cloud:spring-cloud-sleuth-zipkin:jar:2.0.1.RELEASE:compile
 |  |  +- org.springframework.cloud:spring-cloud-commons:jar:2.0.1.RELEASE:compile
 |  |  |  \- org.springframework.security:spring-security-crypto:jar:5.0.11.RELEASE:compile
 |  |  +- io.zipkin.reporter2:zipkin-sender-kafka11:jar:2.7.7:compile
 |  |  \- io.zipkin.reporter2:zipkin-sender-amqp-client:jar:2.7.7:compile
 |  \- com.google.cloud:google-cloud-core-grpc:jar:1.53.0:compile
 |     +- com.google.auth:google-auth-library-credentials:jar:0.12.0:compile
 |     +- com.google.cloud:google-cloud-core:jar:1.53.0:compile
 |     |  +- joda-time:joda-time:jar:2.9.9:compile
 |     |  +- com.google.http-client:google-http-client:jar:1.27.0:compile
 |     |  +- com.google.api:api-common:jar:1.7.0:compile
 |     |  +- com.google.api:gax:jar:1.35.0:compile
 |     |  \- com.google.api.grpc:proto-google-iam-v1:jar:0.12.0:compile
 |     +- com.google.guava:guava:jar:26.0-android:compile
 |     |  \- com.google.j2objc:j2objc-annotations:jar:1.1:compile
 |     +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
 |     +- com.google.protobuf:protobuf-java-util:jar:3.6.1:compile
 |     +- io.grpc:grpc-context:jar:1.16.1:compile
 |     +- io.grpc:grpc-netty-shaded:jar:1.18.0:compile
 |     +- io.grpc:grpc-stub:jar:1.18.0:compile
 |     +- io.grpc:grpc-auth:jar:1.16.1:compile
 |     \- com.google.api:gax-grpc:jar:1.35.0:compile
 |        +- org.threeten:threetenbp:jar:1.3.3:compile
 |        \- com.google.auth:google-auth-library-oauth2-http:jar:0.11.0:compile
 |           \- com.google.http-client:google-http-client-jackson2:jar:1.24.1:compile
 +- org.springframework.cloud:spring-cloud-starter-netflix-hystrix:jar:2.0.1.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-starter:jar:2.0.1.RELEASE:compile
 |  |  +- org.springframework.cloud:spring-cloud-context:jar:2.0.1.RELEASE:compile
 |  |  \- org.springframework.security:spring-security-rsa:jar:1.0.5.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-netflix-core:jar:2.0.1.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-netflix-ribbon:jar:2.0.1.RELEASE:compile
 |  |  \- org.springframework.cloud:spring-cloud-netflix-archaius:jar:2.0.1.RELEASE:compile
 |  +- org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar:2.0.1.RELEASE:compile
 |  |  +- com.netflix.archaius:archaius-core:jar:0.7.6:compile
 |  |  \- commons-configuration:commons-configuration:jar:1.8:compile
 |  |     \- commons-lang:commons-lang:jar:2.6:compile
 |  +- com.netflix.hystrix:hystrix-core:jar:1.5.12:compile
 |  |  \- io.reactivex:rxjava:jar:1.3.8:compile
 |  +- com.netflix.hystrix:hystrix-serialization:jar:1.5.12:compile
 |  |  \- com.fasterxml.jackson.module:jackson-module-afterburner:jar:2.9.8:runtime
 |  +- com.netflix.hystrix:hystrix-metrics-event-stream:jar:1.5.12:compile
 |  +- com.netflix.hystrix:hystrix-javanica:jar:1.5.12:compile
 |  |  +- org.apache.commons:commons-lang3:jar:3.7:runtime
 |  |  \- org.ow2.asm:asm:jar:5.0.4:runtime
 |  \- io.reactivex:rxjava-reactive-streams:jar:1.2.1:compile
 |     \- org.reactivestreams:reactive-streams:jar:1.0.2:compile
 +- org.springframework.cloud:spring-cloud-starter-contract-stub-runner:jar:2.0.1.RELEASE:test
 |  +- org.springframework.cloud:spring-cloud-contract-stub-runner:jar:2.0.1.RELEASE:test
 |  |  +- org.springframework.cloud:spring-cloud-contract-shade:jar:2.0.1.RELEASE:test
 |  |  +- org.springframework.cloud:spring-cloud-contract-spec:jar:2.0.1.RELEASE:test
 |  |  |  +- dk.brics.automaton:automaton:jar:1.11-8:test
 |  |  |  \- org.apache.commons:commons-text:jar:1.1:test
 |  |  +- org.eclipse.jgit:org.eclipse.jgit:jar:4.11.0.201803080745-r:test
 |  |  |  +- com.jcraft:jsch:jar:0.1.54:test
 |  |  |  +- com.jcraft:jzlib:jar:1.1.1:test
 |  |  |  \- com.googlecode.javaewah:JavaEWAH:jar:1.1.6:test
 |  |  +- com.jcraft:jsch.agentproxy.sshagent:jar:0.0.9:test
 |  |  |  \- com.jcraft:jsch.agentproxy.core:jar:0.0.9:test
 |  |  +- com.jcraft:jsch.agentproxy.jsch:jar:0.0.9:test
 |  |  \- com.jcraft:jsch.agentproxy.usocket-jna:jar:0.0.9:test
 |  |     +- net.java.dev.jna:jna:jar:4.5.2:test
 |  |     \- net.java.dev.jna:jna-platform:jar:4.5.2:test
 |  +- org.springframework.cloud:spring-cloud-stream-test-support:jar:2.0.1.RELEASE:test
 |  |  \- org.springframework.cloud:spring-cloud-stream:jar:2.0.1.RELEASE:test
 |  |     +- org.springframework.boot:spring-boot-starter-validation:jar:2.0.8.RELEASE:test
 |  |     +- org.springframework:spring-messaging:jar:5.0.12.RELEASE:test
 |  |     +- org.springframework.integration:spring-integration-core:jar:5.0.11.RELEASE:test
 |  |     |  +- org.springframework:spring-tx:jar:5.0.12.RELEASE:test
 |  |     |  \- io.projectreactor:reactor-core:jar:3.1.14.RELEASE:compile
 |  |     +- org.springframework.integration:spring-integration-jmx:jar:5.0.11.RELEASE:test
 |  |     +- org.springframework:spring-tuple:jar:1.0.0.RELEASE:test
 |  |     |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:test
 |  |     |     \- com.esotericsoftware:minlog:jar:1.3.0:test
 |  |     +- org.springframework.integration:spring-integration-tuple:jar:1.0.0.RELEASE:test
 |  |     \- org.springframework.retry:spring-retry:jar:1.2.3.RELEASE:test
 |  +- org.springframework.cloud:spring-cloud-contract-wiremock:jar:2.0.1.RELEASE:test
 |  |  +- com.github.tomakehurst:wiremock-standalone:jar:2.18.0:test
 |  |  \- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
 |  |     \- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
 |  \- org.eclipse.aether:aether-api:jar:1.0.2.v20150114:test
 +- org.springframework.cloud:spring-cloud-starter-contract-verifier:jar:2.0.1.RELEASE:test
 |  +- org.springframework.cloud:spring-cloud-contract-verifier:jar:2.0.1.RELEASE:test
 |  |  +- javax.inject:javax.inject:jar:1:test
 |  |  +- org.codehaus.groovy:groovy:jar:2.4.16:test
 |  |  +- org.codehaus.groovy:groovy-nio:jar:2.4.16:test
 |  |  +- org.codehaus.groovy:groovy-json:jar:2.4.16:test
 |  |  +- org.codehaus.groovy:groovy-xml:jar:2.4.16:test
 |  |  +- com.github.jknack:handlebars:jar:4.0.6:test
 |  |  |  +- org.antlr:antlr4-runtime:jar:4.5.1-1:test
 |  |  |  \- org.mozilla:rhino:jar:1.7R4:test
 |  |  \- commons-beanutils:commons-beanutils:jar:1.9.3:test
 |  |     +- commons-logging:commons-logging:jar:1.2:test
 |  |     \- commons-collections:commons-collections:jar:3.2.2:test
 |  +- io.rest-assured:rest-assured:jar:3.0.7:test
 |  |  +- org.apache.httpcomponents:httpmime:jar:4.5.6:test
 |  |  +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:test
 |  |  +- io.rest-assured:json-path:jar:3.0.7:test
 |  |  |  \- io.rest-assured:rest-assured-common:jar:3.0.7:test
 |  |  \- io.rest-assured:xml-path:jar:3.0.7:test
 |  |     \- javax.xml.bind:jaxb-api:jar:2.3.1:test
 |  |        \- javax.activation:javax.activation-api:jar:1.2.0:test
 |  +- io.rest-assured:spring-mock-mvc:jar:3.0.7:test
 |  +- com.toomuchcoding.jsonassert:jsonassert:jar:0.4.13:test
 |  \- cglib:cglib:jar:3.2.4:test
 |     \- org.apache.ant:ant:jar:1.9.6:test
 |        \- org.apache.ant:ant-launcher:jar:1.9.6:test
 +- com.googlecode.junit-toolbox:junit-toolbox:jar:1.5:test
 |  +- commons-io:commons-io:jar:2.4:test
 |  \- org.codehaus.jsr166-mirror:jsr166y:jar:1.7.0:test
 +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.8:compile
 +- com.fasterxml.jackson.core:jackson-core:jar:2.9.8:compile
 +- commons-codec:commons-codec:jar:1.9:compile
 +- org.bouncycastle:bcprov-jdk15on:jar:1.60:compile
 \- org.springframework.security.oauth:spring-security-oauth2:jar:2.1.3.RELEASE:compile
    +- org.springframework:spring-beans:jar:5.0.12.RELEASE:compile
    +- org.springframework:spring-context:jar:5.0.12.RELEASE:compile
    +- org.springframework.security:spring-security-core:jar:5.0.11.RELEASE:compile
    +- org.springframework.security:spring-security-config:jar:5.0.11.RELEASE:compile
    +- org.springframework.security:spring-security-web:jar:5.0.11.RELEASE:compile
    \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
       \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile

@ankitshubham97
Copy link
Author

@khodayar Are you trying to run this on AIX too?

@khodayarj
Copy link

@ankitshubham97
I'm trying to run it on a local kubernetes cluster or gcp, I can run the code locally without any problem, but not in the clusters

@ankitshubham97
Copy link
Author

@khodayar What is the underlying OS?

@khodayarj
Copy link

I'm using docker for windows but Linux containers

@ST-DDT
Copy link
Contributor

ST-DDT commented Feb 19, 2019

Is that the complete stacktrace?
Where is the grpc stuff?
Also you use different versions for the grpc components try updating them to the same version?

@ankitshubham97
Copy link
Author

Here is the complete stacktrace:
bash-4.4# /usr/java8_64/jre/bin/java -Xbootclasspath/p:/home/ankit/alpn-boot-8.1.13.v20180117.jar -jar agent-1.0.jar

Feb 19, 2019 12:02:34 AM io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: netty-tcnative unavailable (this may be normal)
java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_aix_ppc_64, netty_tcnative_ppc_64, netty_tcnative]
        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:104)
        at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:526)
        at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:93)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:244)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:471)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:53)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:52)
        at com.ankit.agents.AgentMain.main(AgentMain.java:93)
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative_aix_ppc_64
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative_aix_ppc_64.a
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_aix_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative_ppc_64
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative_ppc_64.a
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_ppc_64 (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative_ppc_64 (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 11 more
        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: io_grpc_netty_shaded_netty_tcnative
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:233)
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:96)
                ... 9 more
        Caused by: java.io.FileNotFoundException: META-INF/native/libio_grpc_netty_shaded_netty_tcnative.a
                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
                ... 10 more
                Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative (Not found in java.library.path)
                        at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                        at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                        at java.lang.System.loadLibrary(System.java:565)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:316)
                        at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
                        ... 10 more
                        Suppressed: java.lang.UnsatisfiedLinkError: io_grpc_netty_shaded_netty_tcnative (Not found in java.library.path)
                                at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1425)
                                at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:1395)
                                at java.lang.System.loadLibrary(System.java:565)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
                                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
                                at java.lang.reflect.Method.invoke(Method.java:508)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:336)
                                at java.security.AccessController.doPrivileged(AccessController.java:647)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:328)
                                at io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:306)
                                ... 11 more

Feb 19, 2019 12:02:34 AM io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Conscrypt not found (this may be normal)
Feb 19, 2019 12:02:34 AM io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts defaultSslProvider
INFO: Jetty ALPN unavailable (this may be normal)
java.lang.ClassNotFoundException: org.eclipse.jetty.alpn.ALPN
        at java.lang.Class.forNameImpl(Native Method)
        at java.lang.Class.forName(Class.java:403)
        at io.grpc.netty.shaded.io.grpc.netty.JettyTlsUtil.isJettyAlpnConfigured(JettyTlsUtil.java:64)
        at io.grpc.netty.shaded.io.grpc.netty.JettyTlsUtil.getJettyAlpnUnavailabilityCause(JettyTlsUtil.java:75)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:257)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:471)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:53)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:52)
        at com.ankit.agents.AgentMain.main(AgentMain.java:93)

Exception in thread "main" java.lang.IllegalStateException: Could not find TLS ALPN provider; no working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:258)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts.forServer(GrpcSslContexts.java:151)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:471)
        at io.grpc.netty.shaded.io.grpc.netty.NettyServerBuilder.useTransportSecurity(NettyServerBuilder.java:53)
        at com.ankit.agents.AgentMain.<init>(AgentMain.java:52)
        at com.ankit.agents.AgentMain.main(AgentMain.java:93)

My dependency in the build.gradle is like this now:

dependencies {
  compile files("$TOOLCHAIN_VERSION_DIR/lib/commons-io-2.6.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/alpn-api-1.0.0.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/grpc-all-1.18.0.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/java-protobuf.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/jetty-alpn-openjdk8-server-9.4.14.v20181114.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-core-2.8.jar")
  compile files("$TOOLCHAIN_VERSION_DIR/lib/log4j-slf4j-impl-2.8.jar")
  compile 'io.grpc:grpc-netty-shaded:1.18.0'
  compile 'io.grpc:grpc-protobuf:1.18.0'
  compile 'io.grpc:grpc-stub:1.18.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/picocli-3.8.2.jar")
  compile group: 'com.google.guava', name: 'guava', version: '20.0'
  compile files("$TOOLCHAIN_VERSION_DIR/lib/javassist-3.19.0-GA.jar")
  compile project(':annotation')
}

@ankitshubham97
Copy link
Author

It is not able to find some native methods("could not load a native library" error in the above log), maybe we have to compile them, but I am not sure how to do that.

@khodayarj
Copy link

Is that the complete stacktrace?
Where is the grpc stuff?
Also you use different versions for the grpc components try updating them to the same version?

I used the same versions (1.18.0) for all grpc dependencies , didn't help

@carl-mastrangelo
Copy link
Contributor

One quick thing: you probably don't want netty tcnative static. Most people should not use that. The regular, dynamic version should be correct.

@khodayarj
Copy link

I added that based on : https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty
Also I had tried the regular version before

@carl-mastrangelo carl-mastrangelo added this to the Unscheduled milestone Feb 19, 2019
@rahulKQL
Copy link

@ankitshubham97 /@khodayar
Hey, Did you find a solution for this issue? If yes, Could you please update here. I am also facing the same issue.

@khodayarj
Copy link

@rahulKQL we couldn't find a solution.

@crengutam
Copy link

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

@Ravipatel401
Copy link

Hey Guys,

Tying to identifying this issues from last 4 days but found the perfect solution that is just checked when

Experiment 1. :- Java 8 32bit with eclipse 32 bit :---- Failed to load any of the given libraries: [netty_tcnative_windows_x86_32, netty_tcnative_x86_32, netty_tcnative]
Experiment 2.:- ### Java 8 64bit with Eclipse Luna 64 bit 👍 :- Worked Without Error.

Thanks

@carl-mastrangelo
Copy link
Contributor

@Ravipatel401 Do you know why it failed to load? It may be something as simple as the filename of the dll was wrong.

@dpkirchner
Copy link

dpkirchner commented Jun 14, 2019

I don't know if this is related, but I noticed that the .so files provided by io.netty:netty-tcnative-boringssl-static:2.0.7.Final and 2.0.12.Final were compiled against a specific set of libraries that expect some symbols to exist in the system's libraries. In my case, I'm running my application in a Docker image based on alpine 3.9.4. The library that I believe grpc is loading (but please correct me if I'm wrong) expects pthread_yield, among others:

bash-4.4# ldd libnetty_tcnative_linux_x86_64.so
	ldd (0x7f3412b6f000)
	librt.so.1 => ldd (0x7f3412b6f000)
	libc.so => ldd (0x7f3412b6f000)
	libpthread.so.0 => ldd (0x7f3412b6f000)
	libdl.so.2 => ldd (0x7f3412b6f000)
Error relocating libnetty_tcnative_linux_x86_64.so: __isnan: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: __isinf: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: __strndup: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: pthread_mutex_consistent_np: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: pthread_mutexattr_setrobust_np: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: pthread_yield: symbol not found
Error relocating libnetty_tcnative_linux_x86_64.so: sys_siglist: symbol not found

The version of libpthread.so.0 that ships with this version of alpine does not provide these symbols (I haven't investigated where sys_syglist, or the __* functions come from). Is it possible this is the cause of the "could not load native" errors? If so, is it possible for grpc-java to use some other pure-Java TLS provider instead?

@carl-mastrangelo
Copy link
Contributor

cc @normanmaurer

@Ravipatel401
Copy link

Ravipatel401 commented Jun 17, 2019

@Ravipatel401 Do you know why it failed to load? It may be something as simple as the filename of the dll was wrong.

@carl-mastrangelo Issue is general while you Run program you need to JAVA 64bit (jvm,jre,jdk all should be 64bit ) otherwise it won't run. Make sure you using 64bit OS on 64bit JAVA. and don't try to change and preloaded libraries it's will create more problems.

@yalagondaakhil
Copy link

yalagondaakhil commented Jun 18, 2019

How to make it work in a 32-bit machine then? Can anyone help with that? @ravipatel410

@rahulgoti1
Copy link

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Thanks, It's working for me.

@MrStobbart
Copy link

I still faced the issue while using openjdk:8 and managed to fix it by upgrading to openjdk:11 😊
Java 11 has ALPN available by default.

@Ravipatel401
Copy link

Ravipatel401 commented Jul 31, 2019 via email

@mrojasabregu
Copy link

I had the same issue and I resolved changing the java version to 64 bit!

@damioti
Copy link

damioti commented Jan 13, 2020

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Thanks, It's working for me.

Yes, this worked for me too.. Worked like magic!!!! Thanks for sharing :)

@Ravipatel401
Copy link

Ravipatel401 commented Jan 13, 2020 via email

@appleseedexm
Copy link

appleseedexm commented Feb 14, 2020

As @Ravipatel401 already said netty does not run on 32bit.
Another factor here is that netty is not running on alpine images in general, see netty/netty-tcnative#111.

@liopalacios
Copy link

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

I had the same problem and I was using a Docker image with a kernel that was incompatible with the netty native libraries needed by my app. I changed the image in my Dockerfile(in my case from "FROM openjdk:8-jre-alpine" to "FROM openjdk:8") and the issue was fixed.

Gracias !! Fue mi solucion luego de no poder dormir por ese problema

@ejona86
Copy link
Member

ejona86 commented Dec 28, 2020

Upgrading to Java 11 was a long-standing workaround for this problem. But since Java 8u252 (April 2020) Java 8 has native support for ALPN, so netty_tcnative isn't required (although still recommended when available) even on Java 8.

@ejona86 ejona86 closed this as completed Dec 28, 2020
@ejona86 ejona86 removed this from the Unscheduled milestone Dec 28, 2020
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests