Nat able to connect to cosmos DB using proxy settings for Azure SDK for Java

[ghost]

Query/Question
We have a support ticket opened by customer where they are using Java SDK for Cosmos DB (4.16.0) and using proxy configurations as per this document - https://docs.microsoft.com/en-us/java/api/com.azure.core.http.proxyoptions?view=azure-java-stable.

These settings work from one machine but fails from another machine.
Both machines have same config and env is same but still it is failing on one and working on other.
There is DNS resolution failure in both scenarios but in one case, HTTP proxy is able to connect and in other it does not.
PCAP dumps show that proxy made an HTTP request for successful connection but for failed one, there is no HTTP request.

Why is this not a Bug or a feature Request?
It is working from one machine and not from other so does not seem like a bug

Setup (please complete the following information if applicable):

• OS: Windows
• IDE: IntelliJ
• Library/Libraries: [e.g. com.azure:azure-core:1.16.0 (groupId:artifactId:version)]

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
Proxy code and error message

Client creation code:

public CosmosClient getCosmosDbClient() {
return new CosmosClientBuilder()
.credential(defaultAzureCredential)
.endpoint(endpoint)
.preferredRegions(preferredLocations)
.multipleWriteRegionsEnabled(multiregionWrite)
.consistencyLevel(ConsistencyLevel.SESSION)
.gatewayMode(getGatewayConnection())
.buildClient();
}

private GatewayConnectionConfig getGatewayConnection() {
GatewayConnectionConfig gatewayConnectionConfig = new GatewayConnectionConfig();
if (StringUtils.isNotEmpty(proxyHost) && proxyPort != null) {
ProxyOptions proxyOptions = new ProxyOptions(ProxyOptions.Type.HTTP,
new InetSocketAddress(proxyHost, proxyPort));
gatewayConnectionConfig.setProxy(proxyOptions);
gatewayConnectionConfig.setMaxConnectionPoolSize(MAX_CONNECTION_POOL_SIZE);
gatewayConnectionConfig.setIdleConnectionTimeout(Duration.of(1,
ChronoUnit.SECONDS));
}
return gatewayConnectionConfig;
}

Exception stacktrace:

java.net.UnknownHostException: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server (helx-dev-storefront-westus-cdb.documents.azure.com)
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_221]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[na:1.8.0_221]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[na:1.8.0_221]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[na:1.8.0_221]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[na:1.8.0_221]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[na:1.8.0_221]
at io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:169) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:166) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_221]
at io.netty.util.internal.SocketUtils.allAddressesByName(SocketUtils.java:166) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.resolver.DefaultNameResolver.doResolveAll(DefaultNameResolver.java:50) ~[netty-resolver-4.1.65.Final.jar:4.1.65.Final]
at io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:79) ~[netty-resolver-4.1.65.Final.jar:4.1.65.Final]
at io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:71) ~[netty-resolver-4.1.65.Final.jar:4.1.65.Final]
at io.netty.resolver.InetSocketAddressResolver.doResolveAll(InetSocketAddressResolver.java:73) ~[netty-resolver-4.1.65.Final.jar:4.1.65.Final]
at io.netty.resolver.InetSocketAddressResolver.doResolveAll(InetSocketAddressResolver.java:31) ~[netty-resolver-4.1.65.Final.jar:4.1.65.Final]
at io.netty.resolver.AbstractAddressResolver.resolveAll(AbstractAddressResolver.java:158) ~[netty-resolver-4.1.65.Final.jar:4.1.65.Final]
at reactor.netty.transport.TransportConnector.doResolveAndConnect(TransportConnector.java:283) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.netty.transport.TransportConnector.lambda$connect$6(TransportConnector.java:110) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.netty.transport.TransportConnector$MonoChannelPromise._subscribe(TransportConnector.java:579) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.netty.transport.TransportConnector$MonoChannelPromise.lambda$subscribe$0(TransportConnector.java:499) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_221]

2021-10-13 12:16:48.891 WARN 24244 --- [ctor-http-nio-2] c.a.c.i.WebExceptionRetryPolicy : Received retriable web exception, will retry

com.azure.cosmos.CosmosException: {"innerErrorMessage":null,"cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.16.0 Windows10/10.0 JRE/1.8.0_221","requestLatencyInMs":101,"requestStartTimeUTC":"2021-10-13T19:16:48.726Z","requestEndTimeUTC":"2021-10-13T19:16:48.827Z","responseStatisticsList":[],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":[],"retryContext":{"statusAndSubStatusCodes":null,"retryCount":0,"retryLatency":0},"metadataDiagnosticsContext":{"metadataDiagnosticList":null},"serializationDiagnosticsContext":{"serializationDiagnosticsList":null},"gatewayStatistics":{"sessionToken":null,"operationType":"Read","resourceType":"DatabaseAccount","statusCode":0,"subStatusCode":10001,"requestCharge":null,"requestTimeline":null},"systemInformation":{"usedMemory":"527921 KB","availableMemory":"6896079 KB","systemCpuLoad":"empty","availableProcessors":8},"clientCfgs":{"id":0,"connectionMode":"GATEWAY","numberOfClients":1,"connCfg":{"rntbd":null,"gw":"(cps:1, rto:PT5S, icto:PT1S, p:true)","other":"(ed: true, cs: false)"},"consistencyCfg":"(consistency: Session, mm: false, prgns: [westus,eastus])"}}}
at com.azure.cosmos.BridgeInternal.createCosmosException(BridgeInternal.java:454) ~[azure-cosmos-4.16.0.jar:na]
at com.azure.cosmos.implementation.RxGatewayStoreModel.lambda$toDocumentServiceResponse$3(RxGatewayStoreModel.java:323) ~[azure-cosmos-4.16.0.jar:na]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) [reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoSingle$SingleSubscriber.onError(MonoSingle.java:150) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) [reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.Operators.error(Operators.java:197) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.Mono.subscribe(Mono.java:4150) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) [reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:224) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:273) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:413) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.EmitterProcessor.drain(EmitterProcessor.java:491) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.EmitterProcessor.tryEmitNext(EmitterProcessor.java:299) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:97) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:189) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect$ClientTransportSubscriber.onError(HttpClientConnect.java:305) ~[reactor-netty-http-1.0.8.jar:1.0.8]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:173) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:444) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:399) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.onError(DefaultPooledConnectionProvider.java:555) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) [reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.Operators.error(Operators.java:197) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.Mono.subscribe(Mono.java:4150) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) [reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.Operators.error(Operators.java:197) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) ~[reactor-core-3.4.7.jar:3.4.7]
at reactor.netty.transport.TransportConnector$MonoChannelPromise._subscribe(TransportConnector.java:579) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at reactor.netty.transport.TransportConnector$MonoChannelPromise.lambda$subscribe$0(TransportConnector.java:499) ~[reactor-netty-core-1.0.8.jar:1.0.8]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_221]

PCAP dump for not working scenario:

PCAP dump for working scenario showing HTTP connection in line 1458

[joshfree]

@alzimmermsft could you assist with proxy configuration?

[alzimmermsft]

Thank you for reporting this @yamandeepsingh. You're configurations for ProxyOptions look correct to me.

@kushagraThapar could we find someone on the Cosmos SDK team to take a deeper look given that azure-core and azure-cosmos handle ProxyOptions slightly different internally.

[kushagraThapar]

@alzimmermsft sure, I will take a look at it, thanks!

[kushagraThapar]

This has been fixed ,thanks!