Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle client close/cancel on grpc mutiny streaming service #25076

Merged

Conversation

pcasaes
Copy link
Contributor

@pcasaes pcasaes commented Apr 21, 2022

When using the Mutiny GRPC interface for streaming as described here, we eventually see the following error whenever a client closes a connection before the service tries to push down new messages:

2022-04-21 13:17:10,859 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-0) Mutiny had to drop the following exception: io.grpc.StatusRuntimeException: CANCELLED: call already cancelled. Use ServerCallStreamObserver.setOnCancelHandler() to disable this exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:366)
        ...

What is happening is that the Multi subscription isn't being canceled whenever a client disconnects. Besides the log pollution this behavior can cause some user pain, particulary if the user adds a termination handler to the multi subscription.

.onTermination().invoke(handleTermination)

If the service pushes messages downstream infrequently we might also see a situation where long disconnected client's subscriptions are still being maintained, possibly causing observability and memory pressure concerns.

The solution here is to simply add a cancel and close handler as the error message suggests. On either on cancel or on close we'll cancel the multi subscription.

@quarkus-bot quarkus-bot bot added the area/grpc gRPC label Apr 21, 2022
@pcasaes pcasaes force-pushed the pc/grpc-handle-close-connection branch from ee94e06 to 26ea2b3 Compare April 21, 2022 20:08
Copy link
Member

@cescoffier cescoffier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, @michalszynkiewicz wdyt?

@michalszynkiewicz michalszynkiewicz merged commit b57ef3d into quarkusio:main Apr 22, 2022
@quarkus-bot quarkus-bot bot added this to the 2.9 - main milestone Apr 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants