You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
aliher1911 opened this issue
Apr 19, 2022
· 0 comments
· Fixed by #80164
Assignees
Labels
A-kvAnything in KV that doesn't belong in a more specific category.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-kvKV Team
Whenever node crashes irrecoverably and removed from the cluster closed timestamp side transport will pick up node id's from range descriptors and will keep trying to talk to them. This will continue possibly forever or till each node restart after all range descriptors are fixed.
This would show up in the logs as:
I220419 10:56:23.265470 1461 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=3] 871 side-transport failed to connect to n3: failed to resolve n3: unable to look up descriptor for n3
I220419 10:56:23.271968 1460 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=2] 872 side-transport failed to connect to n2: failed to resolve n2: unable to look up descriptor for n2
...
I220419 10:56:33.271807 1461 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=3] 878 side-transport failed to connect to n3: failed to resolve n3: unable to look up descriptor for n3
I220419 10:56:33.277426 1460 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=2] 879 side-transport failed to connect to n2: failed to resolve n2: unable to look up descriptor for n2
...
I220419 10:56:43.277381 1461 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=3] 882 side-transport failed to connect to n3: failed to resolve n3: unable to look up descriptor for n3
I220419 10:56:43.281789 1460 kv/kvserver/closedts/sidetransport/sender.go:785 ⋮ [n1,ctstream=2] 883 side-transport failed to connect to n2: failed to resolve n2: unable to look up descriptor for n2
To Reproduce
Run a cluster with 5 nodes. Kill all of the nodes and perform recovery from quorum loss using nodes 1, 4, 5. Restart "fixed" nodes and observe the messages in logs.
The text was updated successfully, but these errors were encountered:
aliher1911
added
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
A-kv
Anything in KV that doesn't belong in a more specific category.
T-kv
KV Team
labels
Apr 19, 2022
aliher1911
changed the title
after node is removed forcibly from cluster logs are spammed by side-transport
after node is removed forcibly from cluster logs are spammed by side-transport messages
Apr 19, 2022
A-kvAnything in KV that doesn't belong in a more specific category.C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.T-kvKV Team
Describe the problem
Whenever node crashes irrecoverably and removed from the cluster closed timestamp side transport will pick up node id's from range descriptors and will keep trying to talk to them. This will continue possibly forever or till each node restart after all range descriptors are fixed.
This would show up in the logs as:
To Reproduce
Run a cluster with 5 nodes. Kill all of the nodes and perform recovery from quorum loss using nodes 1, 4, 5. Restart "fixed" nodes and observe the messages in logs.
Jira issue: CRDB-15826
The text was updated successfully, but these errors were encountered: