Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

failed to create S3 Repository using dockerized ES 6.2.2 due to SDK connect timed out #28893

Closed
jlamillan opened this issue Mar 4, 2018 · 5 comments
Closed

failed to create S3 Repository using dockerized ES 6.2.2 due to SDK connect timed out #28893

jlamillan opened this issue Mar 4, 2018 · 5 comments
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs

Comments

@jlamillan
Copy link

jlamillan commented Mar 4, 2018 •
edited
Loading

I've been getting a number of errors when trying to create an S3 repository on my local box using Docker. Note, the same steps appear to work when elasticsearch is run outside of Docker (using brew).

Here is the latest:

Elasticsearch version: 6.2.2

Plugins installed: [ingest-geoip, ingest-user-agent, repository-s3, discovery-file, x-pack]

JVM version: openjdk version "1.8.0_161"

OS version: Docker / CentOS Linux release 7.4.1708 (Core)

Description of the problem including expected versus actual behavior:
Getting an error when trying to create a S3 Repository.

Steps to reproduce:

  1. Install the repository-s3 plugin, start the elasticsearch container:

mac$ docker run -e AWS_ACCESS_KEY_ID=XXXXXXXXXXX -e AWS_SECRET_KEY=XXXXXXXXXXX docker.elastic.co/elasticsearch/elasticsearch:6.2.2 /bin/sh -c "elasticsearch-plugin install --batch repository-s3 && /usr/local/bin/docker-entrypoint.sh"

mac$ curl http://127.0.0.1:9200/_cat/health
1520277749 19:22:29 docker-cluster green 1 1 1 1 0 0 0 0 - 100.0%
  1. Set the s3 credentials in the elasticsearch keystore (default client):

container# elasticsearch-keystore add s3.client.default.access_key XXXXXXXXXXX
container# elasticsearch-keystore add s3.client.default.secret_key XXXXXXXXXXX

container# elasticsearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key
  1. Attempt to create an S3 Repository (default client):
container# curl -XPUT "http://localhost:9200/_snapshot/my_s3_repository1" -H'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-index-storage-amazon"
  }
}'

Provide logs (if relevant):

Error from create S3 Repository PUT (i.e. step 3 from above):

{"error":{"root_cause":[{"type":"repository_exception","reason":"[my_s3_repository1] failed to create repository"}],"type":"repository_exception","reason":"[my_s3_repository1] failed to create repository","caused_by":{"type":"sdk_client_exception","reason":"Unable to load credentials from service endpoint","caused_by":{"type":"socket_timeout_exception","reason":"connect timed out"}}},"status":500}[root@cb5e3dd9d478

From server:

[2018-03-03T23:58:48,611][INFO ][o.e.c.m.MetaDataCreateIndexService] [hrK-wmC] [.monitoring-es-6-2018.03.03] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[2018-03-03T23:58:48,946][INFO ][o.e.c.r.a.AllocationService] [hrK-wmC] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.03.03][0]] ...]).
[2018-03-04T00:00:08,439][INFO ][o.e.c.m.MetaDataCreateIndexService] [hrK-wmC] [.monitoring-es-6-2018.03.04] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[2018-03-04T00:00:08,475][INFO ][o.e.c.r.a.AllocationService] [hrK-wmC] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.03.04][0]] ...]).
[2018-03-04T00:06:43,320][WARN ][o.e.r.RepositoriesService] [hrK-wmC] failed to create repository [s3][my_s3_repository1]
com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:108) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ~[?:?]
	... 44 more
[2018-03-04T00:06:43,345][WARN ][o.e.r.RepositoriesService] [hrK-wmC] failed to create repository [my_s3_repository1]
org.elasticsearch.repositories.RepositoryException: [my_s3_repository1] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:389) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:108) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
[2018-03-04T00:06:43,383][WARN ][r.suppressed             ] path: /_snapshot/my_s3_repository1, params: {repository=my_s3_repository1}
org.elasticsearch.repositories.RepositoryException: [my_s3_repository1] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:389) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
	at com.amazonaws.auth.EC2CredentialsFetcher.handleError(EC2CredentialsFetcher.java:183) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:162) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_161]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_161]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_161]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_161]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:?]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1199) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:?]
	at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:54) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:108) ~[?:?]
	at com.amazonaws.internal.EC2CredentialsUtils.readResource(EC2CredentialsUtils.java:79) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider$InstanceMetadataCredentialsEndpointProvider.getCredentialsEndpoint(InstanceProfileCredentialsProvider.java:174) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.fetchCredentials(EC2CredentialsFetcher.java:122) ~[?:?]
	at com.amazonaws.auth.EC2CredentialsFetcher.getCredentials(EC2CredentialsFetcher.java:82) ~[?:?]
	at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:151) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.InternalAwsS3Service$PrivilegedInstanceProfileCredentialsProvider.getCredentials(InternalAwsS3Service.java:177) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:5008) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:4982) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4231) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 16 more
@javanna javanna added the :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs label Mar 6, 2018
@javanna
Copy link
Member

javanna commented Mar 6, 2018

cc @elastic/es-distributed

@ywelsch
Copy link
Contributor

ywelsch commented Mar 6, 2018

Problem and solution should be explained in https://discuss.elastic.co/t/failed-to-create-s3-repository-with-es-6-0/108224/
Gist of it: All the modifications to the keystore take affect only after restarting Elasticsearch.

@ywelsch ywelsch closed this as completed Mar 6, 2018
@jlamillan
Copy link
Author

Sounds like a bug.

@windowsrefund
Copy link

Can anyone confirm the following link as being the fix for this? #30652

@ebuildy
Copy link
Contributor

ebuildy commented Feb 16, 2022

Using version 7.15.X, still see java.net.SocketTimeoutException: connect timed out wrong error message, whereas this was a credentials missing issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@windowsrefund @javanna @ebuildy @ywelsch @jlamillan