You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a node starts draining, it starts transferring its leases away. The drain process does this until there's no lease left to transfer.
However in the meantime the store rebalancer sees the QPS on the other stores, which are taking on the draining node's load, increase. So it then selects the now-draining node as a good candidate to transfer leases to, because it appears to be low on QPS (because it's draining)
So the leases are transferred back to the draining node.
So the graceful drain never completes.
And the node eventually is killed with active leases.
Which disrupts performance.
The text was updated successfully, but these errors were encountered:
knz
added
the
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
label
Oct 12, 2020
This is the underlying cause of issues #53760 #53725 #54088 #53811
When a node starts draining, it starts transferring its leases away. The drain process does this until there's no lease left to transfer.
However in the meantime the store rebalancer sees the QPS on the other stores, which are taking on the draining node's load, increase. So it then selects the now-draining node as a good candidate to transfer leases to, because it appears to be low on QPS (because it's draining)
So the leases are transferred back to the draining node.
So the graceful drain never completes.
And the node eventually is killed with active leases.
Which disrupts performance.
The text was updated successfully, but these errors were encountered: