Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Downloading a big Multipart file on resteasy-reactive is throwing an OutOfMemoryError on Upstream #28920

Closed
pjgg opened this issue Oct 29, 2022 · 2 comments · Fixed by #28934
Closed

Downloading a big Multipart file on resteasy-reactive is throwing an OutOfMemoryError on Upstream #28920

pjgg opened this issue Oct 29, 2022 · 2 comments · Fixed by #28934
Labels
area/rest kind/bug Something isn't working
Milestone
2.13.4.Final

Comments

@pjgg
Copy link
Contributor

pjgg commented Oct 29, 2022

Describe the bug

Quarkus version: upstream
Extension: quarkus-rest-client-reactive-jackson

When I try to download a ~2Gb multipart file I am getting the following error

18:23:06,870 INFO  [app] Caused by: java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
18:23:06,871 INFO  [app]        at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
18:23:06,872 INFO  [app]        at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
18:23:06,872 INFO  [app]        at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
18:23:06,873 INFO  [app]        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
18:23:06,874 INFO  [app]        at org.jboss.resteasy.reactive.common.providers.serialisers.FileBodyHandler.doWrite(FileBodyHandler.java:64)
18:23:06,874 INFO  [app]        at org.jboss.resteasy.reactive.common.providers.serialisers.FileBodyHandler.writeTo(FileBodyHandler.java:56)
18:23:06,875 INFO  [app]        at org.jboss.resteasy.reactive.common.providers.serialisers.FileBodyHandler.writeTo(FileBodyHandler.java:22)
18:23:06,876 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writeEntity(MultipartMessageBodyWriter.java:189)
18:23:06,876 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writePart(MultipartMessageBodyWriter.java:130)
18:23:06,877 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.write(MultipartMessageBodyWriter.java:94)
18:23:06,878 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writeMultiformData(MultipartMessageBodyWriter.java:68)
18:23:06,878 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writeResponse(MultipartMessageBodyWriter.java:50)
18:23:06,879 INFO  [app]        at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:220)
18:23:06,880 INFO  [app]        ... 12 more
18:23:06,880 INFO  [app] 18:23:05,789 Request failed: java.lang.RuntimeException: java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
18:23:06,881 INFO  [app]        at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:258)
18:23:06,882 INFO  [app]        at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:191)
18:23:06,882 INFO  [app]        at org.jboss.resteasy.reactive.server.core.serialization.FixedEntityWriter.write(FixedEntityWriter.java:28)
18:23:06,883 INFO  [app]        at org.jboss.resteasy.reactive.server.handlers.ResponseWriterHandler.handle(ResponseWriterHandler.java:34)
18:23:06,884 INFO  [app]        at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:120)
18:23:06,884 INFO  [app]        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:145)
18:23:06,885 INFO  [app]        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:564)
18:23:06,885 INFO  [app]        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
18:23:06,886 INFO  [app]        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
18:23:06,886 INFO  [app]        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
18:23:06,887 INFO  [app]        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
18:23:06,888 INFO  [app]        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
18:23:06,888 INFO  [app]        at java.base/java.lang.Thread.run(Thread.java:833)
18:23:06,889 INFO  [app] Caused by: java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
18:23:06,889 INFO  [app]        at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
18:23:06,890 INFO  [app]        at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
18:23:06,890 INFO  [app]        at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
18:23:06,890 INFO  [app]        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
18:23:06,891 INFO  [app]        at org.jboss.resteasy.reactive.common.providers.serialisers.FileBodyHandler.doWrite(FileBodyHandler.java:64)
18:23:06,891 INFO  [app]        at org.jboss.resteasy.reactive.common.providers.serialisers.FileBodyHandler.writeTo(FileBodyHandler.java:56)
18:23:06,891 INFO  [app]        at org.jboss.resteasy.reactive.common.providers.serialisers.FileBodyHandler.writeTo(FileBodyHandler.java:22)
18:23:06,892 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writeEntity(MultipartMessageBodyWriter.java:189)
18:23:06,892 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writePart(MultipartMessageBodyWriter.java:130)
18:23:06,892 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.write(MultipartMessageBodyWriter.java:94)
18:23:06,893 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writeMultiformData(MultipartMessageBodyWriter.java:68)
18:23:06,893 INFO  [app]        at org.jboss.resteasy.reactive.server.core.multipart.MultipartMessageBodyWriter.writeResponse(MultipartMessageBodyWriter.java:50)
18:23:06,893 INFO  [app]        at org.jboss.resteasy.reactive.server.core.ServerSerialisers.invokeWriter(ServerSerialisers.java:220)
18:23:06,893 INFO  [app]        ... 12 more

Multipart pojo:

public class FileWrapper {
    @RestForm("file")
    @PartType(MediaType.APPLICATION_OCTET_STREAM)
    public File file;

    @RestForm("name")
    @PartType(MediaType.TEXT_PLAIN)
    public String name;
}

Rest API (that is failing)

    @GET
    @Produces(MediaType.MULTIPART_FORM_DATA)
    @Path("/download-multipart")
    public FileWrapper downloadMultipart() {
        FileWrapper wrapper = new FileWrapper();
        wrapper.file = file;
        wrapper.name = file.getName();
        return wrapper;
    }

Note: File is a 2Gb file. Quarkus 2.13.3.Final works as expected!

Expected behavior

No error

Actual behavior

outofMemory exception

How to Reproduce?

git clone [email protected]:quarkus-qe/quarkus-test-suite.git
cd http/rest-client-reactive
mvn clean verify -Dit.test=LargeFileHandlingIT#downloadMultipart

Note: double-check that downloadMultipart is not disabled.

Output of uname -a or ver

No response

Output of java -version

openjdk version "17.0.4" 2022-07-19 OpenJDK Runtime Environment Temurin-17.0.4+8 (build 17.0.4+8) OpenJDK 64-Bit Server VM Temurin-17.0.4+8 (build 17.0.4+8, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response
@pjgg pjgg added the kind/bug Something isn't working label Oct 29, 2022
@quarkus-bot
Copy link

quarkus-bot bot commented Oct 29, 2022

/cc @FroMage, @geoand, @stuartwdouglas

@pjgg pjgg changed the title Downloading a big Multipart file on resteasy-reactive is throwing a OutOfMemoryError on Upstream Downloading a big Multipart file on resteasy-reactive is throwing an OutOfMemoryError on Upstream Oct 29, 2022
@pjgg pjgg mentioned this issue Oct 29, 2022
Merged
3 tasks
@geoand
Copy link
Contributor

geoand commented Oct 29, 2022

cc @Sgitario

Sgitario added a commit to Sgitario/quarkus that referenced this issue Oct 31, 2022
@Sgitario
Use non closeable output stream for writting the multipart entities
25722b7 
Before, we were using a ByteArrayOutputStream because some message writers are closing the original output stream when writing the entity (like JSON). 
The problem is that this approach needs the double of space (keeping the data in the bytearray output stream and then copy the data to the original output stream).
With these changes, we are proxying the original output stream and preventing the message writers to not close the output stream (this was already being used in the JsonbMessageBodyWriter).
Fix quarkusio#28920
@Sgitario Sgitario mentioned this issue Oct 31, 2022
Merged
@quarkus-bot quarkus-bot bot added this to the 2.15 - main milestone Oct 31, 2022
@gsmet gsmet modified the milestones: 2.15 - main, 2.14.0.Final Oct 31, 2022
gsmet pushed a commit to gsmet/quarkus that referenced this issue Oct 31, 2022
@Sgitario @gsmet
Use non closeable output stream for writting the multipart entities
9ddc51c 
Before, we were using a ByteArrayOutputStream because some message writers are closing the original output stream when writing the entity (like JSON).
The problem is that this approach needs the double of space (keeping the data in the bytearray output stream and then copy the data to the original output stream).
With these changes, we are proxying the original output stream and preventing the message writers to not close the output stream (this was already being used in the JsonbMessageBodyWriter).
Fix quarkusio#28920

(cherry picked from commit 25722b7)
@gsmet gsmet modified the milestones: 2.14.0.Final, 2.13.4.Final Oct 31, 2022
gsmet pushed a commit to gsmet/quarkus that referenced this issue Oct 31, 2022
@Sgitario @gsmet
Use non closeable output stream for writting the multipart entities
081608e 
Before, we were using a ByteArrayOutputStream because some message writers are closing the original output stream when writing the entity (like JSON).
The problem is that this approach needs the double of space (keeping the data in the bytearray output stream and then copy the data to the original output stream).
With these changes, we are proxying the original output stream and preventing the message writers to not close the output stream (this was already being used in the JsonbMessageBodyWriter).
Fix quarkusio#28920

(cherry picked from commit 25722b7)
zakkak pushed a commit to zakkak/quarkus that referenced this issue Nov 15, 2022
@Sgitario @zakkak
Use non closeable output stream for writting the multipart entities
ead8c38 
Before, we were using a ByteArrayOutputStream because some message writers are closing the original output stream when writing the entity (like JSON).
The problem is that this approach needs the double of space (keeping the data in the bytearray output stream and then copy the data to the original output stream).
With these changes, we are proxying the original output stream and preventing the message writers to not close the output stream (this was already being used in the JsonbMessageBodyWriter).
Fix quarkusio#28920

(cherry picked from commit 25722b7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/rest kind/bug Something isn't working
Projects
None yet
Milestone
2.13.4.Final
Development

Successfully merging a pull request may close this issue.

Use non closeable output stream for writting the multipart entities Sgitario/quarkus
3 participants
@gsmet @pjgg @geoand