Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

after node is removed forcibly from cluster logs are spammed by side-transport messages #80156

Closed
aliher1911 opened this issue Apr 19, 2022 · 0 comments · Fixed by #80164
Closed
Assignees
Labels
A-kv Anything in KV that doesn't belong in a more specific category. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-kv KV Team

Comments

@aliher1911
Copy link
Contributor

aliher1911 commented Apr 19, 2022

Describe the problem

Whenever node crashes irrecoverably and removed from the cluster closed timestamp side transport will pick up node id's from range descriptors and will keep trying to talk to them. This will continue possibly forever or till each node restart after all range descriptors are fixed.
This would show up in the logs as:

I220419 10:56:23.265470 1461 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=3] 871  side-transport failed to connect to n3: failed to resolve n3: unable to look up descriptor for n3
I220419 10:56:23.271968 1460 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=2] 872  side-transport failed to connect to n2: failed to resolve n2: unable to look up descriptor for n2
...
I220419 10:56:33.271807 1461 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=3] 878  side-transport failed to connect to n3: failed to resolve n3: unable to look up descriptor for n3
I220419 10:56:33.277426 1460 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=2] 879  side-transport failed to connect to n2: failed to resolve n2: unable to look up descriptor for n2
...
I220419 10:56:43.277381 1461 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=3] 882  side-transport failed to connect to n3: failed to resolve n3: unable to look up descriptor for n3
I220419 10:56:43.281789 1460 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=2] 883  side-transport failed to connect to n2: failed to resolve n2: unable to look up descriptor for n2

To Reproduce

Run a cluster with 5 nodes. Kill all of the nodes and perform recovery from quorum loss using nodes 1, 4, 5. Restart "fixed" nodes and observe the messages in logs.

Jira issue: CRDB-15826

@aliher1911 aliher1911 added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-kv Anything in KV that doesn't belong in a more specific category. T-kv KV Team labels Apr 19, 2022
@aliher1911 aliher1911 changed the title after node is removed forcibly from cluster logs are spammed by side-transport after node is removed forcibly from cluster logs are spammed by side-transport messages Apr 19, 2022
@aliher1911 aliher1911 self-assigned this Apr 19, 2022
@craig craig bot closed this as completed in 8e48bc1 May 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv Anything in KV that doesn't belong in a more specific category. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-kv KV Team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants