Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rest Client exausts Threads. #22862

Closed
brunobat opened this issue Jan 13, 2022 · 8 comments · Fixed by #22942
Closed

Rest Client exausts Threads. #22862

brunobat opened this issue Jan 13, 2022 · 8 comments · Fixed by #22942
Labels
area/rest-client kind/bug Something isn't working
Milestone

Comments

@brunobat
Copy link
Contributor

Describe the bug

This is a major issue for us.
I have dozens of services in prod that from time to time they freeze due to thread exhaustion. The vertex event loop stalls.
his can be traced back to unconfigured timeouts on rest clients that hog connections.
We need to go to all rest clients and specify connection and read timeouts on configurations, like:

com.td.wfm.common.client.SomeRestClientClass/mp-rest/connectTimeout=500
com.td.wfm.common.client.SomeRestClientClass/mp-rest/readTimeout=1000

If you use programatic rest creation you must:

        return RestClientBuilder.newBuilder()
                                .baseUri(URI.create(uri))
                                .connectTimeout(connectionTimeout, TimeUnit.MILLISECONDS)
                                .readTimeout(readTimeout, TimeUnit.MILLISECONDS)
                                //...

There are many issues related with this. See:
#15056
#15383
#15191

This happens on 1.13.7.Final but I believe 2.* suffers from the same problem.

Expected behavior

If the REST Client connection is half closed in the other side, it will timeout after a default amount of time.

Actual behavior

If the REST Client connection is half closed in the other side it will never timeout.

How to Reproduce?

You need a rest client calling an endpoint in a loop, in different threads. The other side must hold the connection and not reply.

Output of uname -a or ver

No response

Output of java -version

11

GraalVM version (if different from Java)

No response

Quarkus version or git rev

1.13.7.Final

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

@brunobat brunobat added the kind/bug Something isn't working label Jan 13, 2022
@quarkus-bot
Copy link

quarkus-bot bot commented Jan 13, 2022

/cc @michalszynkiewicz

@geoand
Copy link
Contributor

geoand commented Jan 14, 2022

I assume you are using quarkus-rest-client. Have you tried with quarkus-rest-client-reactive?

@TomasHofman
Copy link
Contributor

I think I can add the same default timeouts that the reactive client uses as part of #22872.

@radcortez
Copy link
Member

I think it could be interesting to have a global timeout for REST clients (like we have other properties that set global configuration for clients).

@michalszynkiewicz
Copy link
Member

that's a cool idea

@geoand
Copy link
Contributor

geoand commented Jan 17, 2022

I think it could be interesting to have a global timeout for REST clients (like we have other properties that set global configuration for clients).

Definitely

@TomasHofman
Copy link
Contributor

Would something like #22942 be OK?

@quarkus-bot quarkus-bot bot added this to the 2.7 - main milestone Jan 18, 2022
@brunobat
Copy link
Contributor Author

brunobat commented Jan 18, 2022

This is ok. Thanks @gsmet !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/rest-client kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants