-
Notifications
You must be signed in to change notification settings - Fork 859
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3AsyncClient fails with EOFException when executing HeadObjectRequest for gzip encoded object #1216
Comments
I could not repo the issue. I used the exact same code except I didn't use spring boot.
|
I have created a self-contained example of what i'm trying to achieve. You can find it here: https://github.com/richardtitze/demos/tree/master/demo-s3-reactive AsyncClientTest starts up a minio server, creates a bucket, puts a gzip encoded object in the bucket, and then executes a head request. If you run "mvn test" you'll see that the one test fails with the same error i reported before. I also upgraded the v2 sdk dependency to 2.5.30 |
@richardtitze I'm getting the same thing. We're basically just calling headObject on a gzipped json file, and that yields a chain of CompletionException -> SdkException -> EofException. I don't quite have the code in a state where I can offer a reproducible example. |
I work with @alexklibisz and have looked into this issue some as well. This issue only happens when the object metadata indicates One solution would be to update |
Hitting this issue too. I have to idea what to do. Is there a fix in the pipeline? I also do not understand to workaround, if you could provide a code example that would be fantastic |
@oripwk we were using HEAD to check whether an object exists and were able to work around this issue by using the listObjects operation for that purpose instead. Here's a gist with a diff showing the change to |
@jkeillor So I guess I cannot use list-objects since I use head-object to inspect metadata. Do you know of any way of changing the |
This change addresses issue aws#1216. HEAD requests would fail against gzip'd resources the crc32Validator attempts to unzip the empty content and fails.
Hi guys, FWIW, I've put together a workaround which is quite effective. The solution is to use object Main {
def main(args: Array[String]): Unit = {
…
val metadata = s3.getObject(GetObjectRequest.builder()
.bucket("bucket")
.key("key")
.build(), new WorkaroundAsyncTransformer
).get().metadata()
…
}
}
class WorkaroundAsyncTransformer extends AsyncResponseTransformer[GetObjectResponse, GetObjectResponse] {
@volatile private var future: CompletableFuture[GetObjectResponse] = _
override def prepare(): CompletableFuture[GetObjectResponse] = {
future = new CompletableFuture[GetObjectResponse]()
future
}
override def onResponse(response: GetObjectResponse): Unit = future.complete(response)
override def onStream(publisher: SdkPublisher[ByteBuffer]): Unit = publisher.subscribe(new DrainingSubscriber[ByteBuffer])
override def exceptionOccurred(error: Throwable): Unit = future.completeExceptionally(error)
} |
…n it comes to creating an SDK HTTP full request. This fixes issues like #1216 where HEAD request responses have a content-length but no payload, confusing the SDK.
…n it comes to creating an SDK HTTP full request. This fixes issues like #1216 where HEAD request responses have a content-length but no payload, confusing the SDK.
sync client is still affected by this issue |
Fix for the S3AsyncClient was released in SDK @productivityindustries: please open a new issue describing the problems you're seeing. |
|
@debora-ito For me it works only for versions starting from |
@oripwk you're right, I corrected my previous comment, thank you! |
…cb9a99095 Pull request: release <- staging/435b60ea-bf40-450e-9b59-bb6cb9a99095
I am trying to execute a HeadObjectRequest for objects stored in S3. All objects are compressed and do have Content-Encoding: gzip.
Expected Behavior
Just head info's are returned - and no exception is thrown.
Current Behavior
This stacktrace is logged:
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:816) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:797) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:324) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at com.example.demos3reactive.DemoS3ReactiveApplication.main(DemoS3ReactiveApplication.java:13) [classes/:na]
Caused by: java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_202]
at com.example.demos3reactive.DemoS3ReactiveApplication.run(DemoS3ReactiveApplication.java:25) [classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:813) [spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
... 5 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkClientException: null
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:123) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:105) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_202]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$1(MakeAsyncHttpRequestStage.java:137) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) ~[na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_202]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_202]
Caused by: java.io.UncheckedIOException: java.io.EOFException
at software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException(FunctionalUtils.java:180) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:110) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136) ~[utils-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.decompressing(Crc32Validation.java:85) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.process(Crc32Validation.java:62) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.http.Crc32Validation.validate(Crc32Validation.java:44) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler.lambda$new$0(BaseAsyncClientHandler.java:52) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:88) ~[sdk-core-2.5.28.jar:na]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_202]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[na:1.8.0_202]
at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:129) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:180) ~[netty-nio-client-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:67) ~[netty-nio-client-2.5.28.jar:na]
at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:298) ~[netty-nio-client-2.5.28.jar:na]
at com.typesafe.netty.HandlerPublisher.publishMessage(HandlerPublisher.java:362) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.flushBuffer(HandlerPublisher.java:304) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.receivedDemand(HandlerPublisher.java:258) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher.access$200(HandlerPublisher.java:41) ~[netty-reactive-streams-2.0.0.jar:na]
at com.typesafe.netty.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:452) ~[netty-reactive-streams-2.0.0.jar:na]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495) ~[netty-transport-4.1.34.Final.jar:4.1.34.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.34.Final.jar:4.1.34.Final]
... 1 common frames omitted
Caused by: java.io.EOFException: null
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:268) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:258) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:164) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) ~[na:1.8.0_202]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) ~[na:1.8.0_202]
at software.amazon.awssdk.core.http.Crc32Validation.lambda$decompressing$2(Crc32Validation.java:85) ~[sdk-core-2.5.28.jar:na]
at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108) ~[utils-2.5.28.jar:na]
... 24 common frames omitted
Steps to Reproduce
package com.example.demos3reactive;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
@SpringBootApplication
public class DemoS3ReactiveApplication implements CommandLineRunner {
}
Context
I'm trying to create a reactive spring microservice in front of S3.
Your Environment
SDK used: aws-sdk-java-v2 2.5.28
Java: Amazon Corretto 8
OS: MacOS
The text was updated successfully, but these errors were encountered: