Resteasy "connection was closed" error after upgrading to 2.4.x #21762

pedroigor · 2021-11-28T21:08:07Z

Describe the bug

After bumping the Quarkus version to 2.4.x (tested 2.4.0.Final, 2.4.1.Final, and 2.4.2.Final) in Keycloak, the following error is happening every time you reload a page multiple times during a short period (e.g.: holding F5 when at the browser):

2021-11-28 17:51:03,913 ERROR [org.jbo.thr.errors] (executor-thread-4) Thread Thread[executor-thread-4,5,main] threw an uncaught exception: java.lang.RuntimeException: java.io.IOException: j
ava.io.IOException: io.vertx.core.VertxException: Connection was closed
        at io.quarkus.resteasy.runtime.standalone.VertxHttpRequest$VertxExecutionContext$VertxHttpAsyncResponse.vertxFlush(VertxHttpRequest.java:396)
        at io.quarkus.resteasy.runtime.standalone.VertxHttpRequest$VertxExecutionContext$VertxHttpAsyncResponse$1.run(VertxHttpRequest.java:326)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:543)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: java.io.IOException: io.vertx.core.VertxException: Connection was closed
        at io.quarkus.resteasy.runtime.standalone.VertxOutputStream.close(VertxOutputStream.java:124)
        at io.quarkus.resteasy.runtime.standalone.VertxHttpResponse.finish(VertxHttpResponse.java:145)
        at io.quarkus.resteasy.runtime.standalone.VertxHttpRequest$VertxExecutionContext$VertxHttpAsyncResponse.vertxFlush(VertxHttpRequest.java:394)
        ... 8 more
Caused by: java.io.IOException: io.vertx.core.VertxException: Connection was closed
        at io.quarkus.resteasy.runtime.standalone.VertxBlockingOutput.write(VertxBlockingOutput.java:79)
        at io.quarkus.resteasy.runtime.standalone.VertxHttpResponse.writeBlocking(VertxHttpResponse.java:172)
        at io.quarkus.resteasy.runtime.standalone.VertxOutputStream.close(VertxOutputStream.java:122)
        ... 10 more
Caused by: io.vertx.core.VertxException: Connection was closed

The error does not happen if we downgrade to version 2.3.1.

Expected behavior

No errors in logs if loading a page multiple times.

Actual behavior

Errors are shown in logs as per description.

How to Reproduce?

I'm did not work yet a reproducer but the error is very easy to reproduce by following these steps:

Build Keycloak.X Distribution with pom.xml in quarkus package set to 2.5.0.Final (and postgres to 42.3.1), use e.g. mvn clean install -DskipTests -Pdistribution in root directory (or see readme here: https://github.com/keycloak/keycloak/tree/main/quarkus ).
unpack zip from distribution/server-x-dist/target
cd /bin
run ./kc.sh start-dev
Using a browser, reload the page multiple times during a very short period of time

Output of `uname -a` or `ver`

Linux fedora 5.14.18-200.fc34.x86_64 #1 SMP Fri Nov 12 16:48:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Output of `java -version`

openjdk version "11.0.13" 2021-10-19 OpenJDK Runtime Environment 18.9 (build 11.0.13+8) OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.4.2.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

pedroigor · 2021-11-28T21:09:56Z

@geoand Any chance this could be related to #21602? The pages being loaded are obtained from the classpath, similarly to that issue.

I'm also going to check if the error is happening when running the latest upstream/main.

pedroigor · 2021-11-28T21:23:59Z

@geoand I was able to reproduce the issue using the reproducer from #21602. By reloading the following URL http://localhost:8080/hello/res/styles.css multiple times in a short period of time, you should see the exception aforementioned in the logs.

The good news is that I was not able to reproduce the issue using 2.5.0.CR1.

geoand · 2021-11-29T08:19:25Z

@geoand Any chance this could be related to #21602? The pages being loaded are obtained from the classpath, similarly to that issue.

It could be, but it seems unlikeky.

The good news is that I was not able to reproduce the issue using 2.5.0.CR1.

How about with 2.5.0.Final? If you can't reproduce with that version either, we can go ahead and close this.

pedroigor · 2021-11-29T11:01:40Z

@geoand Works with 2.5.0.Final and also with the latest 999-SNAPSHOT. I'm still running tests though but I could not reproduce the error so far using those versions.

I'm afraid we are going to need to wait for 2.5.1 before we release KC v16.

Thanks.

pedroigor · 2021-11-30T14:49:01Z

@geoand Does it make sense to end the response here https://github.com/quarkusio/quarkus/blob/main/extensions/resteasy-classic/resteasy/runtime/src/main/java/io/quarkus/resteasy/runtime/standalone/VertxBlockingOutput.java#L79?

Similarly to the lines right above where the response is ended prior to throwing the exception or just returning.

I'm asking because if the vert.x response is not properly ended handlers added via RoutingContext.addHeadersEndHandler are never called. Probably indicating that the Vert.X response was not properly ended/closed?

geoand · 2021-11-30T15:03:34Z

I assume that throwing at that point ensure the exception is picked and handled up elsewhere, although I am not sure.

Have you tried patching that part to see if it affect your use case somehow?

pedroigor · 2021-11-30T15:20:58Z

Not yet. Just found this and solved it on our side by checking if headers were written after invoking downstream handlers from our filter to then make sure our resources are properly cleaned up.

I'll do that and get back to you. Thanks.

pedroigor · 2021-11-30T16:03:10Z

@geoand Yes, it does help. And it kind of makes sense because until 2.3.1.Final things were working as expected.

After bumping to 2.4.x we started noticing the server hanging from time to time. And the change was included in 2.4.

If you want I can check how to write a test for it and send a PR.

geoand · 2021-11-30T16:05:51Z

That would be great

pedroigor · 2021-12-01T13:00:19Z

@geoand Created #21842. Sorry, I failed to find a way to write a test. Any help is appreciated.

geoand · 2021-12-01T13:54:55Z

No idea honestly since I don't know how you reproduced the error in the first place.

mschorsch · 2021-12-04T09:10:33Z

@geoand @pedroigor

Here is a simple reproducer with Quarkus 2.5.1: reproducer.zip

Start and open a browser tab on http://localhost:8080
Close the browser tab
The exception is thrown after 5 seconds

@geoand reopen?

Relates to: quarkusio#21762 (comment)

geoand · 2021-12-06T07:34:52Z

Thanks for the reproducer.

I opened #21942 to handle it

Relates to: quarkusio#21762 (comment)

Relates to: quarkusio#21762 (comment) (cherry picked from commit 4783958)

pedroigor added the kind/bug Something isn't working label Nov 28, 2021

quarkus-bot bot added the triage/needs-triage label Nov 28, 2021

pedroigor closed this as completed Nov 29, 2021

quarkus-bot bot removed the triage/needs-triage label Nov 29, 2021

geoand added the triage/out-of-date This issue/PR is no longer valid or relevant label Nov 29, 2021

pedroigor mentioned this issue Dec 1, 2021

End Vert.x response before propagating mapped exception in VertxBlockingoutput #21841

Closed

geoand added a commit to geoand/quarkus that referenced this issue Dec 6, 2021

Don't throw an exception when finishing a closed RESTEasy response

4783958

Relates to: quarkusio#21762 (comment)

geoand mentioned this issue Dec 6, 2021

Don't throw an exception when finishing a closed RESTEasy response #21942

Merged

jeanphi-baconnais pushed a commit to jeanphi-baconnais/quarkus that referenced this issue Dec 7, 2021

Don't throw an exception when finishing a closed RESTEasy response

2c29721

Relates to: quarkusio#21762 (comment)

gsmet pushed a commit to gsmet/quarkus that referenced this issue Dec 9, 2021

Don't throw an exception when finishing a closed RESTEasy response

07501e3

Relates to: quarkusio#21762 (comment) (cherry picked from commit 4783958)

mschorsch mentioned this issue Jan 18, 2022

Unhandled VertxException when client closes http connection #22968

Closed

pedroigor mentioned this issue Jun 14, 2022

[KC17] Frequent "Unexpected server error" mostly without any log keycloak/keycloak#10892

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resteasy "connection was closed" error after upgrading to 2.4.x #21762

Resteasy "connection was closed" error after upgrading to 2.4.x #21762

pedroigor commented Nov 28, 2021 •

edited

Loading

pedroigor commented Nov 28, 2021 •

edited

Loading

pedroigor commented Nov 28, 2021

geoand commented Nov 29, 2021

pedroigor commented Nov 29, 2021

pedroigor commented Nov 30, 2021

geoand commented Nov 30, 2021

pedroigor commented Nov 30, 2021 •

edited

Loading

pedroigor commented Nov 30, 2021 •

edited

Loading

geoand commented Nov 30, 2021

pedroigor commented Dec 1, 2021

geoand commented Dec 1, 2021

mschorsch commented Dec 4, 2021 •

edited

Loading

geoand commented Dec 6, 2021

Resteasy "connection was closed" error after upgrading to 2.4.x #21762

Resteasy "connection was closed" error after upgrading to 2.4.x #21762

Comments

pedroigor commented Nov 28, 2021 • edited Loading

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

pedroigor commented Nov 28, 2021 • edited Loading

pedroigor commented Nov 28, 2021

geoand commented Nov 29, 2021

pedroigor commented Nov 29, 2021

pedroigor commented Nov 30, 2021

geoand commented Nov 30, 2021

pedroigor commented Nov 30, 2021 • edited Loading

pedroigor commented Nov 30, 2021 • edited Loading

geoand commented Nov 30, 2021

pedroigor commented Dec 1, 2021

geoand commented Dec 1, 2021

mschorsch commented Dec 4, 2021 • edited Loading

geoand commented Dec 6, 2021

pedroigor commented Nov 28, 2021 •

edited

Loading

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

pedroigor commented Nov 28, 2021 •

edited

Loading

pedroigor commented Nov 30, 2021 •

edited

Loading

pedroigor commented Nov 30, 2021 •

edited

Loading

mschorsch commented Dec 4, 2021 •

edited

Loading