[cosmos 4.40.0]NoSuchElementException: io.opentelemetry.javaagent.instrumentation.netty.v4.common.client.NettySslInstrumentationHandler[QUERY]

[Deepa-Pathak]

I am working on a request to upgrade from cosmos 4.0.0-preview.1 to recent cosmos library: 4.40.0.

After upgrading this and the related dependencies, the application runs fine locally but on Kubernetes Pod, this exception is thrown:

api-method= operation-name= user-id= app-id=:Channel([id: 0x012067fb, L:/10.10.3.94:47258 - R:cdb-ms-prod-eastus2-fd5.documents.azure.com/52.179.141.33:14509]) health check request failed due to: java.util.NoSuchElementException: io.opentelemetry.javaagent.instrumentation.netty.v4.common.client.NettySslInstrumentationHandler|	at io.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:1073) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:302) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:290) ~[netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.userEventTriggered(RntbdRequestManager.java:397) [azure-cosmos-4.40.0.jar!/:4.40.0]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) [netty-codec-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireUserEventTriggered(CombinedChannelDuplexHandler.java:430) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) [netty-codec-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.CombinedChannelDuplexHandler.userEventTriggered(CombinedChannelDuplexHandler.java:241) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.opentelemetry.javaagent.instrumentation.netty.v4.common.client.NettySslInstrumentationHandler.userEventTriggered(NettySslInstrumentationHandler.java:104) [?:?]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1835) [netty-handler-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1359) [netty-handler-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) [netty-handler-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) [netty-handler-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) [netty-codec-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) [netty-codec-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-transport-classes-epoll-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-classes-epoll-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-classes-epoll-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.70.Final.jar!/:4.1.70.Final]|	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.70.Final.jar!/:4.1.70.Final]|	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]|

I have introduced the dependency of application insights:
com.microsoft.azure:applicationinsights-core:2.6.4
com.microsoft.azure:applicationinsights-logging-log4j2:2.6.4
com.microsoft.azure:applicationinsights-spring-boot-starter:2.6.4
com.microsoft.azure:applicationinsights-web:2.6.4

But the issue is still there. I tried to add io.telemetry also, but the pod is going into crash loop back off state however this time also locally the application runs okay.

attached is my pom:

pom.zip

I have read other posts that mention that this is a known issue with application insights but we do not know how to check/upgrade that. None of the online materials have been of any help.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar, @TheovanKraay

[joshfree]

/cc @trask @lmolkova

[trask]

hi @Deepa-Pathak!

this looks like the same issue fixed at open-telemetry/opentelemetry-java-instrumentation#6469, which was included in Application Insights Java 3.4.0 (and later).

What version of Application Insights Java (3.x) agent or OpenTelemetry Java agent are you using?

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!