SNS/SQS SdkClientException Server failed to send complete response #1520

alejandrod · 2019-11-21T14:19:19Z

This issue has been reported before for S3 and rapid uploads as described here:
#1122
#452

This is for SNS/SQS

Expected Behavior

No error should be thrown or a retry should be performed

Current Behavior

We randomly get this exception:

software.amazon.awssdk.core.exception.SdkClientException: Server failed to send complete response
java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Server failed to send complete response
	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:61)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:75)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryErrorIfNeeded(AsyncRetryableStage.java:175)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:126)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.lambda$notifyIfResponseNotCompleted$2(ResponseHandler.java:391)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:171)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:391)
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelInactive(ResponseHandler.java:143)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
	at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:393)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:358)
	at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:285)
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:393)
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:358)
	at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1075)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.lang.Thread.run(Thread.java:748)
	Caused by: software.amazon.awssdk.core.exception.SdkClientException: Server failed to send complete response
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
	at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125)
	... 52 common frames omitted
	Caused by: java.io.IOException: Server failed to send complete response
	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:390)
	... 43 common frames omitted

Possible Solution

Steps to Reproduce (for bugs)

val sdkHttpClient: SdkAsyncHttpClient = NettyNioAsyncHttpClient.builder()
    .maxConcurrency(100)
    .maxPendingConnectionAcquires(10000)
    .build()

val snsClient: SnsAsyncClient = SnsAsyncClient.builder()
    .credentialsProvider(credentialsProvider)
    .endpointOverride(URI.create(snsEndpoint))
    .region(Region.US_EAST_1)
    .httpClient(sdkHttpClient)
    .build()

Source.single(PublishRequest.builder().message("send some love").topicArn(topicArn).build())
        .runWith(SnsPublisher.publishSink())
        .map(_ => Right(true))
        .recover({
          case ex: Exception =>
            logger.error(s"Major error sending to topic $topicName $event", ex)
            Left(ex)
        })

Your Environment

AWS Java SDK version used: 2.10.18 but I've been having issues for a long time now. Since 2.5.x I think
JDK version used: Oracle 1.8 build 221
Operating System and version: Ubuntu 18.04
We use Akka Alpakka

The text was updated successfully, but these errors were encountered:

zoewangg · 2019-11-21T19:18:12Z

Hi @alejandrod can you try reducing connectionMaxIdleTimeout or upgrading to the latest sdk?

In 2.10.19 and later versions, the default connectionMaxIdleTimeout is 5 seconds for NettyNioAsyncClient.

https://github.com/aws/aws-sdk-java-v2/blob/master/CHANGELOG.md#21019-2019-11-18

bmaizels · 2019-11-25T18:13:30Z

Hi, hopefully the advice to change the connectionMaxIdleTimeout worked for you. Please feel free to re-open if you are still seeing this problem with the new setting.

alejandrod · 2019-12-09T20:19:33Z

Looks like it works ! Thanks! version 2.10.30

sarithareddy · 2020-06-12T14:47:18Z

The issue is still happening in versions 2.10.19 and latest version 2.13.34.

Context:
I have a unit test case that tries to uploads files to S3 and stores some information about the author and what the files are about to DynamoDB. The way the test are setup, there should not be more than 10 concurrent uploads. I have tried 2.10.19 version and the latest 2.13.34, observed error in both. Added the stacktraces as well to this ticket.

Also, we are using localstack (https://github.com/localstack/localstack), a local AWS cloud stack for unit testing. Let me know if any information is required, happy to provide.

Code that uploads files to S3:

	public Mono<FileUploadResponse> save(final FileUploadRequest request) {

		final String key = this.env + "/" + request.getKey();
		final Path path = Paths.get(request.getPath());

		return mono(() -> this.s3.putObject(PutObjectRequest.builder()
			.bucket(this.bucket)
			.key(key)
			.acl(ObjectCannedACL.PUBLIC_READ)
			.build(), path))
			.thenReturn(
				FileUploadResponse.newBuilder()
					.setUrl(this.s3DownloadUrl(key))
					.build()
			);

	}

S3AsyncClient builder:

	public static S3AsyncClient s3AsyncClient(final String s3Url) {
                // s3.url=http://localstack:4569
		log.info("Creating s3 client at endpoint {}", s3Url);
		final S3AsyncClientBuilder builder = S3AsyncClient.builder();
		if (!Strings.isNullOrEmpty(s3Url)) {
			builder.endpointOverride(URI.create(s3Url));
			builder.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("bogus", "bogus")));
		}
		return builder.build();
	}

The SDKException stacktrace is pretty much same in both versions, except exception message.

2.10.19 stacktrace:

software.amazon.awssdk.core.exception.SdkClientException: Server failed to send complete response
     	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
     	at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)
     	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125)
     	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107)
     	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
     	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
     	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
     	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
     	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249)
     	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.lambda$notifyIfResponseNotCompleted$2(ResponseHandler.java:391)
     	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:171)
     	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:391)
     	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelInactive(ResponseHandler.java:143)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
     	at com.typesafe.netty.HandlerSubscriber.channelInactive(HandlerSubscriber.java:146)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
     	at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
     	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
     	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
     	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:420)
     	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:391)
     	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:356)
     	at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:285)
     	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
     	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
     	at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)
     	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1417)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)
     	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)
     	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:913)
     	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
     	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
     	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
     	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
     	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
     	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
     	at java.base/java.lang.Thread.run(Thread.java:834)
     Caused by: java.io.IOException: Server failed to send complete response
     	at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:390)
     	... 41 common frames omitted

2.13.34 stacktrace:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Server failed to send complete response
      at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
      at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:199)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:143)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:125)
      at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
      at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
      at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
      at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
      at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$WrappedErrorForwardingResponseHandler.onError(MakeAsyncHttpRequestStage.java:122)

…8fa011d40 Pull request: release <- staging/84586be3-e69a-4e40-ab2c-1d98fa011d40

cgallom · 2024-05-02T19:23:49Z

sarithareddy

HI, have read this post and others like it and I notice that you never could check if the updates on aws-sdk-java-v2 worked to fix your bug. I am currently using version 2.10.41 and the error persists in this version. Did you find out how to fix this bug ?

alejandrod mentioned this issue Nov 21, 2019

Rapid S3 downloads causes failure #1122

Closed

alejandrod changed the title ~~SNS SdkClientException Server failed to send complete response~~ SNS/SQS SdkClientException Server failed to send complete response Nov 21, 2019

zoewangg added the guidance Question that needs advice or information. label Nov 21, 2019

bmaizels closed this as completed Nov 25, 2019

sarithareddy mentioned this issue Jun 15, 2020

S3 Uploads: SdkClientException Server failed to send complete response #1899

Closed

aws-sdk-java-automation added a commit that referenced this issue Jul 7, 2021

Merge pull request #1520 from aws/staging/84586be3-e69a-4e40-ab2c-1d9…

a88d244

…8fa011d40 Pull request: release <- staging/84586be3-e69a-4e40-ab2c-1d98fa011d40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNS/SQS SdkClientException Server failed to send complete response #1520

SNS/SQS SdkClientException Server failed to send complete response #1520

alejandrod commented Nov 21, 2019 •

edited

Loading

zoewangg commented Nov 21, 2019 •

edited

Loading

bmaizels commented Nov 25, 2019

alejandrod commented Dec 9, 2019

sarithareddy commented Jun 12, 2020

cgallom commented May 2, 2024

SNS/SQS SdkClientException Server failed to send complete response #1520

SNS/SQS SdkClientException Server failed to send complete response #1520

Comments

alejandrod commented Nov 21, 2019 • edited Loading

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Your Environment

zoewangg commented Nov 21, 2019 • edited Loading

bmaizels commented Nov 25, 2019

alejandrod commented Dec 9, 2019

sarithareddy commented Jun 12, 2020

cgallom commented May 2, 2024

alejandrod commented Nov 21, 2019 •

edited

Loading

zoewangg commented Nov 21, 2019 •

edited

Loading