Timeout waiting for connection from pool while calling S3client.getObject

[sushilkumarchaudhary]

All,
I see issue while trying to read a file from S3. The behavior is totally random and some time it does appear and sometime it does not. What could I missing here. its using all the default connection property.

code which is calling get method :

S3Object object = amazonS3Client.getObject(new GetObjectRequest(s3Bucket, fileName));
try {
  bytes = IOUtils.toByteArray(object. getObjectContent());
} catch (IOException e) {
  AmazonS3TemplateException exception = new AmazonS3TemplateException(
      "Exception while downloading object from S3 bucket '" + s3Bucket + "'", e);
  LOGGER.error("AmazonS3Template.downloadObject", exception.getMessage(), exception);
  throw exception;
}finally{
	if(object!=null){
		object.close();
	}
}

Here are the Exception stack track :

Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4169) ~[aws-java-sdk-s3-1.11.125.jar!/:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4116) ~[aws-java-sdk-s3-1.11.125.jar!/:?]
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1365) ~[aws-java-sdk-s3-1.11.125.jar!/:?]
at com.capitalone.cerberus.amazon.s3.template.AmazonS3Template.downloadObject(AmazonS3Template.java:82) ~[classes!/:01.00.00.22-SNAPSHOT]
at
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:292) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.3.jar!/:4.5.3]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.conn.$Proxy98.get(Unknown Source) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar!/:4.5.3]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar!/:4.5.3]
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) ~[aws-java-sdk-core-1.11.125.jar!/:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) ~[aws-java-sdk-core-1.11.125.jar!/:?]
... 15 more
2017-11-30 21:29:48:

[dagnir]

Normally this error is caused by the object content stream being left open but it looks like you're closing them here.

Is there any chance you're doing many of these downloads concurrently in a threadpool?

[sushilkumarchaudhary]

Yes, I am using a thread pool of 50 threads but each of threads are trying to download different file.

Do we need separate s3 connection for each thread?

[dagnir]

@sushilkumarchaudhary Yes, the number of concurrent requests should be <= the number of max connections in the connection pool; otherwise the requests can get blocked trying to lease a connection from the pool and then eventually timeout.

[sushilkumarchaudhary]

I see the default max connection is also 50, which is mostly less or equal to number of threads running. I will increase max connection and will test it.Any other things which could be causing this.

[dagnir]

@sushilkumarchaudhary Hmm those are the two situations I'm aware of. Please let me know what the results are.

[sushilkumarchaudhary]

@dagnir it looks like it working for me so far. I will report if found any issue.

[varunnvs92]

Happy to hear the code is working fine. Feel free to reopen if you face the issue again.

[nidhiwalkover]

I am also facing the same issue. I am facing issues while resolving this.

Can you help me how to check no of connections in pool and How can i limit them and how to close them.

[rohithn]

@nidhiwalkover I was having the same issue. It turned out I hadn't closed all the S3Objects properly across the application. There was one piece of code that had a few objects open, which eventually lead to this error. Scan through your code and make sure all instances of S3Objects are closed.

Sample code below that I use to get an InputStream of the file.

public InputStream getObject(String path) throws IOException {
  
  InputStream is = null;
  S3Object s3Obj = null;
  
  try {
    s3Obj = s3.getObject(bucket, path);
    // There might be a better way to do this.
    try(S3ObjectInputStream stream = s3Obj.getObjectContent()) {
      ByteArrayOutputStream temp = new ByteArrayOutputStream();
      IOUtils.copy(stream, temp);
      is = new ByteArrayInputStream(temp.toByteArray());
    }
  } catch (AmazonServiceException ase) {
    logger.error(ase.getMessage(), ase);
  } catch (AmazonClientException ace) {
    logger.error(ace.getMessage(), ace);
  } finally {
    if (s3Obj != null) {
      try {
        // Close the object
        s3Obj.close(); 
      } catch (IOException e) {
        logger.error("Unable to close S3 object: {}", e.getMessage(), e);
      }
    }
  }
  return is;
}

[nidhiwalkover]

Hello, Thank You so much for updating me. In my case, I am using AwsS3Class private static final AmazonS3 *S3* = new AmazonS3Client(); And I am performing all operations with this object. For Ex: *S3Object o = S3.getObject(BUCKET, filePath);* *S3.deleteObject(new DeleteObjectRequest(BUCKET, filePath));* And with this s3 object, I am unable to find any method to release the object. can you send me any sample code to delete this s3 object??? Regards, Nidhi

[rohithn]

You need to close only the "S3Object" not the "AmazonS3" object.
So add o.close(); after you finish using o.

[nidhiwalkover]

Okay. Whenever i do S3Object o = S3.getObject(BUCKET, filePath); I have to close o.close. My problem is resolved with this. *Thank You* Regards, Nidhi

…

On Thu, Feb 1, 2018 at 1:36 PM, Rohith Nandakumar ***@***.***> wrote: You need to close only the "S3Object" not the "AmazonS3" object. So add o.close(); after you finish using o. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1405 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHZPQkxwJLb5vyAsyHb-y_bY12DEYHQ2ks5tQXBrgaJpZM4QyoLQ> .

[MarounMaroun]

How exactly does this create a timeout? Does the thread keep waiting until its input stream gets closed in order to release the connection?

[VishalChavda2423]

Can anyone help me for same connection timeout from pool error?

[diegobm23]

@nidhiwalkover I was having the same issue. It turned out I hadn't closed all the S3Objects properly across the application. There was one piece of code that had a few objects open, which eventually lead to this error. Scan through your code and make sure all instances of S3Objects are closed.

Sample code below that I use to get an InputStream of the file.

public InputStream getObject(String path) throws IOException {
  
  InputStream is = null;
  S3Object s3Obj = null;
  
  try {
    s3Obj = s3.getObject(bucket, path);
    // There might be a better way to do this.
    try(S3ObjectInputStream stream = s3Obj.getObjectContent()) {
      ByteArrayOutputStream temp = new ByteArrayOutputStream();
      IOUtils.copy(stream, temp);
      is = new ByteArrayInputStream(temp.toByteArray());
    }
  } catch (AmazonServiceException ase) {
    logger.error(ase.getMessage(), ase);
  } catch (AmazonClientException ace) {
    logger.error(ace.getMessage(), ace);
  } finally {
    if (s3Obj != null) {
      try {
        // Close the object
        s3Obj.close(); 
      } catch (IOException e) {
        logger.error("Unable to close S3 object: {}", e.getMessage(), e);
      }
    }
  }
  return is;
}

Boy, this save my life, thank you.

[jeyhun-dev]

public S3Object downloadFileFromS3bucket(String bucketName, String fileName) {
    S3Object s3Object = null;
    try {
        log.info("Downloading {} from bucket: {}", fileName, bucketName);
        s3Object = amazonS3Client.getObject(bucketName, fileName);
        return s3Object;
    } catch (Exception e) {
        log.error("Error downloading file {} from bucket {}", fileName, bucketName, e);
        throw new RuntimeException("Error downloading file from S3", e);
    } finally {
        if (s3Object != null) {
            try {
                s3Object.close();
            } catch (IOException e) {
                log.warn("Error closing S3Object", e);
            }
        }
    }
}