Intermittent s3 downloads failing with caused by "Unable to execute http error"

[ajinkyakadam]

Description

Downloads from s3 are intermittently failing. Note downloading files directly using awscli in the same setting (network, vpc, endpoints) works perfectly fine.

java.net.UnknownHostException: bucketxyz.s3.us-west-2.amazonaws.com
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at com.amazonaws.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:27)
	at com.amazonaws.http.DelegatingDnsResolver.resolve(DelegatingDnsResolver.java:38)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
	at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
	at com.amazonaws.http.conn.$Proxy18.connect(Unknown Source)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to Base_Test_Docker_Worker (i-0933c76e295dfe92f)
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1036)
		at hudson.FilePath.act(FilePath.java:1025)
		at de.taimos.pipeline.aws.S3DownloadStep$Execution$1.run(S3DownloadStep.java:159)
Caused: com.amazonaws.SdkClientException: Unable to execute HTTP request: bucketxyz.s3.us-west-2.amazonaws.com
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1264)
	at com.amazonaws.services.s3.transfer.TransferManager.doDownload(TransferManager.java:1050)
	at com.amazonaws.services.s3.transfer.TransferManager.doDownload(TransferManager.java:1008)
	at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:846)
	at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:764)
	at com.amazonaws.services.s3.transfer.TransferManager.download(TransferManager.java:718)
	at de.taimos.pipeline.aws.S3DownloadStep$RemoteDownloader.invoke(S3DownloadStep.java:207)
	at de.taimos.pipeline.aws.S3DownloadStep$RemoteDownloader.invoke(S3DownloadStep.java:177)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2913)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

Steps to Reproduce

Using the following snippet of jenkins pipeline aws plugin to download the file

        s3Download(
          bucket: "bucketxyz",
          file: "destination_filepath",
          path: "s3_objectpath",
          force: true
        )

Environment

AWS Java SDK Version : 1.11.264

Jenkins-Version: 2.121.2

Plugin-Version: 1.24

Master/Slave Setup: yes

[debora-ito]

Hi @ajinkyakadam, I see you're using jenkins aws plugin, and unfortunately we cannot guarantee the SDK will work with every third-party library, specially if they're using Java SDK version 1.11.264, which was released almost 2 years ago. I see you already created an issue on their Github repository, and that's the support channel I was going to suggest.

Check issue #1503, where the stack trace is almost identical to yours and you may find relevant info in the discussion there.

I'll go ahead and close this, but feel free to reopen if you have further questions.

[Beauclair]

@ajinkyakadam any update for this issue? I think I have similar issue :

com.amazonaws.SdkClientException: Unable to execute HTTP request: mybucket.s3.us-east-2.amazonaws.com
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException (AmazonHttpClient.java:1175)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper (AmazonHttpClient.java:1121)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute (AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer (AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute (AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500 (AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute (AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute (AmazonHttpClient.java:512)
at com.amazonaws.services.s3.AmazonS3Client.invoke (AmazonS3Client.java:4921)
at com.amazonaws.services.s3.AmazonS3Client.invoke (AmazonS3Client.java:4867)
at com.amazonaws.services.s3.AmazonS3Client.invoke (AmazonS3Client.java:4861)
at com.amazonaws.services.s3.AmazonS3Client.listObjects (AmazonS3Client.java:880)
at com.amazonaws.services.s3.AmazonS3Client.listObjects (AmazonS3Client.java:848)
at com.mybucket.training.services.AmazonS3Services.getObjects (AmazonS3Services.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at com.google.api.server.spi.SystemService.invokeServiceMethod (SystemService.java:351)
at com.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler.handle (EndpointsMethodHandler.java:119)
at com.google.api.server.spi.handlers.EndpointsMethodHandler$RestHandler.handle (EndpointsMethodHandler.java:102)
at com.google.api.server.spi.dispatcher.PathDispatcher.dispatch (PathDispatcher.java:50)
at com.google.api.server.spi.EndpointsServlet.service (EndpointsServlet.java:71)
at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:848)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1772)
at com.google.api.control.ControlFilter.doFilter (ControlFilter.java:225)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1759)
at com.google.api.control.ConfigFilter.doFilter (ConfigFilter.java:120)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1759)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter (JdbcMySqlConnectionCleanupFilter.java:60)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:524)
at org.eclipse.jetty.server.session.SessionHandler.doHandle (SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:134)
at com.google.apphosting.runtime.jetty9.ParseBlobUploadHandler.handle (ParseBlobUploadHandler.java:119)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle (ContextHandler.java:1182)
at com.google.apphosting.runtime.jetty9.AppEngineWebAppContext.doHandle (AppEngineWebAppContext.java:187)
at org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope (SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope (ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141)
at com.google.apphosting.runtime.jetty9.AppVersionHandlerMap.handle (AppVersionHandlerMap.java:293)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle (Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:333)
at com.google.apphosting.runtime.jetty9.RpcConnection.handle (RpcConnection.java:213)
at com.google.apphosting.runtime.jetty9.RpcConnector.serviceRequest (RpcConnector.java:81)
at com.google.apphosting.runtime.jetty9.JettyServletEngineAdapter.serviceRequest (JettyServletEngineAdapter.java:134)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest (JavaRuntime.java:728)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest (JavaRuntime.java:691)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run (JavaRuntime.java:661)
at com.google.apphosting.runtime.JavaRuntime$NullSandboxRequestRunnable.run (JavaRuntime.java:853)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run (ThreadGroupPool.java:270)
at java.lang.Thread.run (Thread.java:748)
Caused by: java.net.UnknownHostException: mybucket.s3.us-east-2.amazonaws.com
at java.net.InetAddress.getAllByName0 (InetAddress.java:1282)
at java.net.InetAddress.getAllByName (InetAddress.java:1194)
at java.net.InetAddress.getAllByName (InetAddress.java:1128)
at com.amazonaws.SystemDefaultDnsResolver.resolve (SystemDefaultDnsResolver.java:27)
at com.amazonaws.http.DelegatingDnsResolver.resolve (DelegatingDnsResolver.java:38)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect (DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect (PoolingHttpClientConnectionManager.java:373)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke (ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy26.connect (Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute (MainClientExec.java:394)
at org.apache.http.impl.execchain.MainClientExec.execute (MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute (ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute (InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute (SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest (AmazonHttpClient.java:1297)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper (AmazonHttpClient.java:1113)