[BUG] Failing Spring startup with CosmosDB on Azure

[ViktorZieg]

Describe the bug
When starting my Spring app on an Azure App Service I get a NoSuchElementException.
Application Insights shows GET /addresses/ prior to the exception

Exception or Stack Trace

Report this azure-cosmos issue to ensure it is addressed:
[ChannelHandlerContext(RntbdRequestManager#0, [id: 0xcd8ed54d, L:/***:54350 - R:***westeurope.documents.azure.com/***:17994])]
[com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.reportIssue(RntbdRequestManager.java:957)]
[SslHandshakeCompletionEvent(SUCCESS): : java.util.NoSuchElementException: io.opentelemetry.javaagent.instrumentation.netty.common.client.NettySslInstrumentationHandler
at io.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:1073)
at io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:302)
at io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:290)
at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.userEventTriggered(RntbdRequestManager.java:396)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117)
at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:368)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireUserEventTriggered(CombinedChannelDuplexHandler.java:430)
at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117)
at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:368)
at io.netty.channel.CombinedChannelDuplexHandler.userEventTriggered(CombinedChannelDuplexHandler.java:241)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
at io.opentelemetry.javaagent.instrumentation.netty.common.client.NettySslInstrumentationHandler.userEventTriggered(NettySslInstrumentationHandler.java:104)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1838)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1360)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:487)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:385)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
]

Setup:

• Spring Boot 2.7.2
• Spring Cloud Azure 4.4.0

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar, @TheovanKraay

[yiliuTo]

Hi @kushagraThapar could you help to take a look of this issue? Besides, @ViktorZieg could you provide which Spring Cloud Azure library you are using?

[TheovanKraay]

This is not an issue in Cosmos DB or Java SDK for Cosmos DB.
Please refer also to issue: #30459.
This is a known issue in Application Insights Java agent, that was fixed in a later version of that agent software (see open-telemetry/opentelemetry-java-instrumentation#6469). Please ensure you are using the latest version of the Application Insights Java agent.

[ViktorZieg]

Hi @kushagraThapar could you help to take a look of this issue? Besides, @ViktorZieg could you provide which Spring Cloud Azure library you are using?

Do you mean this?

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.azure.spring</groupId>
                <artifactId>spring-cloud-azure-dependencies</artifactId>
                <version>4.2.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

Or the dependencies of the services I am using?

[ViktorZieg]

This is not an issue in Cosmos DB or Java SDK for Cosmos DB. Please refer also to issue: #30459. This is a known issue in Application Insights Java agent, that was fixed in a later version of that agent software (see open-telemetry/opentelemetry-java-instrumentation#6469). Please ensure you are using the latest version of the Application Insights Java agent.

I have looked into this issue, but I am not aware of using Application Insights Java agent. Also, I did not find any dependency related to it.

[TheovanKraay]

This is not an issue in Cosmos DB or Java SDK for Cosmos DB. Please refer also to issue: #30459. This is a known issue in Application Insights Java agent, that was fixed in a later version of that agent software (see open-telemetry/opentelemetry-java-instrumentation#6469). Please ensure you are using the latest version of the Application Insights Java agent.

I have looked into this issue, but I am not aware of using Application Insights Java agent. Also, I did not find any dependency related to it.

Sorry, to be more precise, there is known issue in Application Insights Java agent, and this was also fixed in the upstream OpenTelemetry Java agent (which it appears you are using), see here: open-telemetry/opentelemetry-java-instrumentation#6469. So I believe you will need to take a later version of opentelemetry-instrumentation-api. If this does not solve your problem, please provide a sample app that reproduces the issue, and we will investigate.

[ViktorZieg]

Where did you see, that I am using OpenTelemetry? I could not find it anywhere and a bit confused how to take a later version

[TheovanKraay]

[SslHandshakeCompletionEvent(SUCCESS): : java.util.NoSuchElementException: io.opentelemetry.javaagent.instrumentation.netty.common.client.NettySslInstrumentationHandler

From the stack trace you included: [SslHandshakeCompletionEvent(SUCCESS): : java.util.NoSuchElementException: io.opentelemetry.javaagent.instrumentation.netty.common.client.NettySslInstrumentationHandler

Either you have the relevant dependency directly in your build, or a library that bundles it. If you do not believe you are using it, please share a sample application which reproduces this error, and we can investigate how to resolve this.

[ViktorZieg]

I could not create a sample app, where this occurs. Also I could not find anything poining to opentelemetry. For now, I just downgraded to the latest working solution. I close this, because there seems to be a solution to the issue.