Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kverserver: store rebalancer pushes leases back to draining nodes #55440

Closed
knz opened this issue Oct 12, 2020 · 1 comment · Fixed by #55808
Closed

kverserver: store rebalancer pushes leases back to draining nodes #55440

knz opened this issue Oct 12, 2020 · 1 comment · Fixed by #55808
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@knz
Copy link
Contributor

knz commented Oct 12, 2020

This is the underlying cause of issues #53760 #53725 #54088 #53811

When a node starts draining, it starts transferring its leases away. The drain process does this until there's no lease left to transfer.

However in the meantime the store rebalancer sees the QPS on the other stores, which are taking on the draining node's load, increase. So it then selects the now-draining node as a good candidate to transfer leases to, because it appears to be low on QPS (because it's draining)

So the leases are transferred back to the draining node.

So the graceful drain never completes.
And the node eventually is killed with active leases.
Which disrupts performance.

@knz knz added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Oct 12, 2020
@knz
Copy link
Contributor Author

knz commented Oct 12, 2020

The allocator does it too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant