Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3AsyncClient fails with EOFException when executing HeadObjectRequest for gzip encoded object #1216

Closed
richardtitze opened this issue Apr 19, 2019 · 14 comments
Labels
bug This issue is a bug.

Comments

@richardtitze
Copy link

I am trying to execute a HeadObjectRequest for objects stored in S3. All objects are compressed and do have Content-Encoding: gzip.

Expected Behavior

Just head info's are returned - and no exception is thrown.

Current Behavior

This stacktrace is logged:
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:816) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:797) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at com.example.demos3reactive.DemoS3ReactiveApplication.main(DemoS3ReactiveApplication.java:13) [classes/:na]
Caused by: java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_202]
at com.example.demos3reactive.DemoS3ReactiveApplication.run(DemoS3ReactiveApplication.java:25) [classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:813) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
... 5 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkClientException: null
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:123) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:105) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_202]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$1(MakeAsyncHttpRequestStage.java:137) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_202]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_202]
Caused by: java.io.UncheckedIOException: java.io.EOFException
at software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException(FunctionalUtils.java:180) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:110) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.decompressing(Crc32Validation.java:85) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.process(Crc32Validation.java:62) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.validate(Crc32Validation.java:44) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:52) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_202]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:180) ~[netty-nio-client-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:67) ~[netty-nio-client-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:298) ~[netty-nio-client-2.5.28.jar:na]
at com.typesafe.netty.HandlerPublisher.publishMessage(HandlerPublisher.java:362) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.flushBuffer(HandlerPublisher.java:304) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.receivedDemand(HandlerPublisher.java:258) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.access$200(HandlerPublisher.java:41) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:452) ~[netty-reactive-streams-2.0.0.jar:na]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
... 1 common frames omitted
Caused by: java.io.EOFException: null
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) ~[na:1.8.0_202]
at software.amazon.awssdk.core.http.Crc32Validation.lambda$decompressing$2(Crc32Validation.java:85) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108) ~[utils-2.5.28.jar:na]
... 24 common frames omitted

Steps to Reproduce

package com.example.demos3reactive;

import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;

@SpringBootApplication
public class DemoS3ReactiveApplication implements CommandLineRunner {

public static void main(String[] args) {
	SpringApplication.run(DemoS3ReactiveApplication.class, args);
}

@Override
public void run(String... args) throws Exception {
	S3AsyncClient client = S3AsyncClient.builder()
			.serviceConfiguration(builder -> builder.checksumValidationEnabled(false))
			.build();
	HeadObjectRequest hor = HeadObjectRequest.builder()
			.bucket("a_valid_bucket")
			.key("a_key") // object stored has Content-Encoding: gzip
			.build();
	client.headObject(hor).get();
}

}

Context

I'm trying to create a reactive spring microservice in front of S3.

Your Environment

SDK used: aws-sdk-java-v2 2.5.28
Java: Amazon Corretto 8
OS: MacOS

@varunnvs92
Copy link
Contributor

I could not repo the issue. I used the exact same code except I didn't use spring boot.

  1. Is there anything special about the object? I tried with empty and non-empty objects but can't reproduce.
  2. Did you try with the latest version of the SDK?
  3. Can you try with plain Java code and check if you still see the issue

@richardtitze
Copy link
Author

I have created a self-contained example of what i'm trying to achieve. You can find it here: https://github.com/richardtitze/demos/tree/master/demo-s3-reactive

AsyncClientTest starts up a minio server, creates a bucket, puts a gzip encoded object in the bucket, and then executes a head request. If you run "mvn test" you'll see that the one test fails with the same error i reported before. I also upgraded the v2 sdk dependency to 2.5.30

@dagnir dagnir added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Apr 29, 2019
@alexklibisz
Copy link

@richardtitze I'm getting the same thing. We're basically just calling headObject on a gzipped json file, and that yields a chain of CompletionException -> SdkException -> EofException. I don't quite have the code in a state where I can offer a reproducible example.

@jkeillor
Copy link

I work with @alexklibisz and have looked into this issue some as well.

This issue only happens when the object metadata indicates content-encoding: gzip. CR32Validation attempts to unzip the content body to do the validation which fails because for http HEAD the response content has been set to a zero-length byte array.

One solution would be to update AsyncResponseHandler to only set the content on SdkHttpFullResonse when a non-empty response body exists.

@debora-ito debora-ito added bug This issue is a bug. needs-reproduction This issue needs reproduction. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Oct 12, 2020
@oripwk
Copy link

oripwk commented Dec 2, 2020

Hitting this issue too. I have to idea what to do.

Is there a fix in the pipeline?

I also do not understand to workaround, if you could provide a code example that would be fantastic

@jkeillor
Copy link

jkeillor commented Dec 2, 2020

@oripwk we were using HEAD to check whether an object exists and were able to work around this issue by using the listObjects operation for that purpose instead.

Here's a gist with a diff showing the change to AsyncResponseHandler that I mentioned in my earlier comment: https://gist.github.com/jkeillor/9b41a4b36ba0ff1613421ccf5c0f0566

@oripwk
Copy link

oripwk commented Dec 2, 2020

@jkeillor So I guess I cannot use list-objects since I use head-object to inspect metadata.

Do you know of any way of changing the AsyncResponseHandler without forking?

@debora-ito
Copy link
Member

Apologies for the long delay in response in this issue, thank you @oripwk for calling our attention to this.
I can reproduce the issue with the latest version of the SDK.

@jkeillor would you like to submit a PR?

@debora-ito debora-ito removed the needs-reproduction This issue needs reproduction. label Dec 3, 2020
jkeillor added a commit to jkeillor/aws-sdk-java-v2 that referenced this issue Dec 3, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
This change addresses issue aws#1216. HEAD requests would fail against gzip'd resources the crc32Validator attempts to unzip the empty content and fails.
@oripwk
Copy link

oripwk commented Dec 4, 2020

Hi guys,

FWIW, I've put together a workaround which is quite effective.

The solution is to use getObject with a custom AsyncResponseTransformer. Then, you have access to all the fields as in headObject:

object Main {
  def main(args: Array[String]): Unit = {
    …
    val metadata = s3.getObject(GetObjectRequest.builder()
      .bucket("bucket")
      .key("key")
      .build(), new WorkaroundAsyncTransformer
    ).get().metadata()
    …
  }
}

class WorkaroundAsyncTransformer extends AsyncResponseTransformer[GetObjectResponse, GetObjectResponse] {
  @volatile private var future: CompletableFuture[GetObjectResponse] = _
    override def prepare(): CompletableFuture[GetObjectResponse] = {
      future = new CompletableFuture[GetObjectResponse]()
      future
    }
    override def onResponse(response: GetObjectResponse): Unit = future.complete(response)
    override def onStream(publisher: SdkPublisher[ByteBuffer]): Unit = publisher.subscribe(new DrainingSubscriber[ByteBuffer])
    override def exceptionOccurred(error: Throwable): Unit = future.completeExceptionally(error)
}

millems added a commit that referenced this issue Dec 11, 2020
…n it comes to creating an SDK HTTP full request.

This fixes issues like #1216 where HEAD request responses have a content-length but no payload, confusing the SDK.
millems added a commit that referenced this issue Dec 14, 2020
…n it comes to creating an SDK HTTP full request.

This fixes issues like #1216 where HEAD request responses have a content-length but no payload, confusing the SDK.
@productivityindustries
Copy link

sync client is still affected by this issue

@debora-ito
Copy link
Member

debora-ito commented Jan 22, 2021

Fix for the S3AsyncClient was released in SDK 2.15.46 2.15.51.
Since the original issue was addressed I'll go ahead and close this.

@productivityindustries: please open a new issue describing the problems you're seeing.

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@oripwk
Copy link

oripwk commented Jan 24, 2021

Fix for the S3AsyncClient was released in SDK 2.15.46.
Since the original issue was addressed I'll go ahead and close this.

@productivityindustries: please open a new issue describing the problems you're seeing.

@debora-ito For me it works only for versions starting from 2.15.51

@debora-ito
Copy link
Member

@oripwk you're right, I corrected my previous comment, thank you!

aws-sdk-java-automation pushed a commit that referenced this issue Mar 8, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…cb9a99095

Pull request: release <- staging/435b60ea-bf40-450e-9b59-bb6cb9a99095
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

8 participants