-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: improve TopicClient connection resiliency #1400
Conversation
f89046f
to
9bdbcb1
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -32,6 +32,11 @@ export class Default extends TopicClientConfiguration { | |||
transportStrategy: new StaticTopicTransportStrategy({ | |||
grpcConfiguration: new StaticTopicGrpcConfiguration({ | |||
numClients: 1, | |||
// TODO: when we introduce lambda configurations, we do not want to enable keepalives, because they |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it be worth it to just add the lambda config now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah good call.
// restartedDueToError: false, | ||
// firstMessage: true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove?
At some point in time a refactor seems to have made the TopicClient less resilient to network interruptions. It may have happened when we made changes to disable keepalives by default (due to lambda issues), or during some refactor work to add topics support to the web SDK. This commit does the following: * Add configuration options for grpc keepalives and enable them in the default topicclient config * Adds a good deal of trace-level logging to help debug what is happening when subscriptions are interrupted * Re-work the error-handling and "end of stream" logic a bit to improve connection resiliency. I tested this on my laptop by disabling my wifi after the subscription was created. Prior to these changes, the network issues were not detected and the connection was left in a broken state. After these changes, the issues are detected and the connection is re-established successfully.
a013d8a
to
6d2907a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good and recommend deleting dead code vs commenting out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
Tested again and it was fine after 5min of being idle. I somehow didn't see the unsubscribe line in the topic-subscribe.ts
example 🤦🏻♀️
At some point in time a refactor seems to have made the TopicClient less resilient to network interruptions. It may have happened when we made changes to disable keepalives by default (due to lambda issues), or during some refactor work to add topics support to the web SDK.
This commit does the following:
I tested this on my laptop by disabling my wifi after the subscription was created. Prior to these changes, the network issues were not detected and the connection was left in a broken state. After these changes, the issues are detected and the connection is re-established successfully.