Rapid concurrent S3 uploads sometimes produce a java.io.IOException: Server failed to send complete response #452
Labels
bug
This issue is a bug.
Netty NIO Client
pending-release
This issue will be fixed by an approved PR that hasn't been released yet.
Comments
dagnir
added
investigating
|
This was a quirk we noticed when we originally debugged #202. Basically what would happen is that when connecting to S3 over TLS, on very rare occasions, S3 will suddenly close the connection after the request is sent rather than sending a response. Curiously, it does not seem to happen when connected over unencrypted HTTP. There are a couple issues that look quite similar that have been reported to the PHP SDK: aws/aws-sdk-php#983 |
|
I think it makes sense to treat this as a retryable exception. |
dagnir
added
bug
|
We're also getting the same exception every once in a while even with AWS SDK "2.0.0-preview-11". We're using the following asynchronous s3 client in scala: val amazonS3 = S3AsyncClient.builder()
.httpClient(NettyNioAsyncHttpClient)
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(settings.s3AccessId, settings.s3SecretAccessKey)))
.region(Region.of(settings.s3Region))
.build()And making the following requests: val request = GetObjectTaggingRequest.builder()
.bucket(bucketName)
.key(keyName)
.build()
val futureResponse = amazonS3.getObjectTagging(request)And every once in a while, If anyone knows why this may be, I would highly appreciate it! :-) Note: we're using scala, version "2.11.11" (on top of the openjdk version "1.8.0_171"). And we're running on ubuntu 16.04. |
|
Debugging (my) above issue further, it seems to have to do with SSL and the fact that channels are reused. To give you a little bit of context: using the above Here are the exact logs: |
|
To reproduce the error, I created a minimal code snippet: import java.util.function.BiConsumer
import com.typesafe.scalalogging.LazyLogging
import software.amazon.awssdk.auth.credentials.{AwsBasicCredentials, StaticCredentialsProvider}
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3AsyncClient
import software.amazon.awssdk.services.s3.model.{GetObjectTaggingRequest, GetObjectTaggingResponse}
object AwsBug extends LazyLogging {
def main(args: Array[String]): Unit = {
val s3AccessId = "your access id" // set me!
val s3SecretAccessKey = "your secret access key" // set me!
val bucketName = "specify an existing bucket here!" // set me!
val keyName = "specify an existing key inside your bucket here!" // set me!
val amazonS3 = S3AsyncClient.builder()
.httpClient(NettyNioAsyncHttpClient.builder().build())
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(s3AccessId, s3SecretAccessKey)))
.region(Region.of("us-east-1"))
.build()
val request = GetObjectTaggingRequest.builder()
.bucket(bucketName)
.key(keyName)
.build()
val responseHandler = new BiConsumer[GetObjectTaggingResponse, Throwable] {
override def accept(resp: GetObjectTaggingResponse, ex: Throwable): Unit = {
if (ex == null)
logger.info("responseHandler: SUCCESS")
else
logger.error("responseHandler: FAILURE!", ex)
}
}
while (true) {
val futureResponse = amazonS3.getObjectTagging(request)
futureResponse whenComplete responseHandler
Thread.sleep(60 * 1000L)
}
}
}Leaving that code running for at least some hours causes the above exception with very high likelyhood to occur on my machine. It should equally well be reproducible in Java. |
|
The same behavior while using SQS service when the pressure gets about 400 messages per second, some of requests (not many) cause the same
I am using 2.0.0-preview-13 version |
|
Thank all of you for the thorough reports. The issue has been fixed in #834. |
|
The fix has been released as part of |
|
I still see this issue in 2.3.9... did this fix it for you @ochrons ? |
|
@ShibaBandit, #1041, which will go out on Monday will also help to significantly reduce the chances of this occurring. |
aws-sdk-java-automation
added a commit
that referenced
this issue
Mar 19, 2019
…24d8d0b4 Pull request: release <- staging/97244113-58b1-42aa-b196-2bd624d8d0b4
justnance
added
pending-release
|
I'm still seeing this in 2.7.33 - is anyone else? We are managing our own Thread Pool, could this be contributing to the issue? Using JDK 8. Trying to see if #1122 (comment) will fix it |
|
I am using |
|
Same issue here with |
@zoewangg This is not fixed as I Noticed. I Have read same response from your side and closing unresolved issues on other posts. I think You shouldn't proceed to close posts if you are not sure if releases fix it. It is still happening in 2024 I have same issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
Uploading thousands of files concurrently to S3
Current Behavior
Previously we suffered from #202 but that seems to have been fixed in SDK preview-9. The new problem is similar, except it happens quite rarely, once per tens of thousands of uploads. We use multipart uploading for all uploads.
In one test case I was uploading batches of 1000 files each, running 32 uploads concurrently. The error happened on the first file of the first batch and also on the fifth file on the first batch. All other batches uploaded without errors.
Internally our service extracts individual files from the batches and uploads them to S3 using the new async API, so the total number of concurrent requests to S3 is probably quite high.
The error originates from
aws-sdk-java-v2/http-clients/netty-nio-client/src/main/java/software/amazon/awssdk/http/nio/netty/internal/ResponseHandler.java
Lines 124 to 133 in 1c8ae14
Steps to Reproduce (for bugs)
Upload lots of files to S3 concurrently.
Your Environment
AWS SDK preview-9
JDK8
AWS Linux
i3.xlarge EC2 instance
The text was updated successfully, but these errors were encountered: