Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpc: grpc-gateway loopback conn mistakenly uses onlyOnceDialer and causes sticky permanent RPC errors #103762

Closed
knz opened this issue May 23, 2023 · 2 comments · Fixed by #103764
Closed

rpc: grpc-gateway loopback conn mistakenly uses onlyOnceDialer and causes sticky permanent RPC errors #103762

knz opened this issue May 23, 2023 · 2 comments · Fixed by #103764
Assignees
@knz
Labels
A-observability-inf A-server-networking Pertains to network addressing,routing,initialization backport-23.1.x Flags PRs that need to be backported to 23.1 branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs regression Regression from a release. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.

Comments

@knz
Copy link
Contributor

knz commented May 23, 2023 •
edited
Loading

Describe the problem

The rpc-gateway connection (used by our HTTP interfaces) uses a loopback connector. This is mistakenly configured in v23.1 to use "onlyOnceDialer", a mechanism through which a connection is not re-attempted if it fails.

The result is that when a cluster is overloaded, the loopback connection may fail once (due to a timeout) and then it will fail forever after, causing most of the HTTP interfaces to become unusable.

xref #103692 (comment)
xref #99261 (comment)

To Reproduce

Overload a v23.1 cluster and use the HTTP connection until it fails once.
Then the failure will persist forever until the node is restarted.

Expected behavior

The loopback connection should be retried if it fails (i.e. not use onlyOnceDialer)

Jira issue: CRDB-28178
Epic: CRDB-28893
@knz knz added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. regression Regression from a release. A-server-networking Pertains to network addressing,routing,initialization backport-23.1.x Flags PRs that need to be backported to 23.1 labels May 23, 2023
@erikgrinaker erikgrinaker added release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels May 23, 2023
@knz knz mentioned this issue May 23, 2023
Merged
@craig craig bot closed this as completed in 741c91b May 23, 2023
@blathers-crl blathers-crl bot mentioned this issue May 23, 2023
Merged
@erikgrinaker
Copy link
Contributor

Reopening until 23.1 backport is merged.

@erikgrinaker erikgrinaker reopened this May 23, 2023
@erikgrinaker
Copy link
Contributor

Backport merged.

@irfansharif irfansharif added the O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs label May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@knz knz

Labels
A-observability-inf A-server-networking Pertains to network addressing,routing,initialization backport-23.1.x Flags PRs that need to be backported to 23.1 branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-support Would prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docs regression Regression from a release. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rpc: use the loopback conn also for GRPCDialOptions knz/cockroach
3 participants
@knz @erikgrinaker @irfansharif