Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: the --drain-wait arg for the drain command should check the server.shutdown.*_wait cluster settings #98388

Closed
rafiss opened this issue Mar 10, 2023 · 0 comments · Fixed by #98390
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-server-and-security DB Server & Security

Comments

@rafiss
Copy link
Collaborator

rafiss commented Mar 10, 2023

Is your feature request related to a problem? Please describe.

We have this guidance in our docs right now. https://www.cockroachlabs.com/docs/stable/node-shutdown.html#drain-timeout

--drain-wait sets the timeout for all draining phases and is not related to the server.shutdown.drain_wait cluster setting, which configures the "unready phase" of draining. The value of --drain-wait should be greater than the sum of server.shutdown.drain_wait, server.shutdown.connection_wait, server.shutdown.query_wait times two, and server.shutdown.lease_transfer_wait.

This is pretty easy for anyone to miss. The impact of not setting --drain-wait high enough is that the drain command will be cancelled, and node shutdown will proceed non-gracefully.

Describe the solution you'd like

The code that uses the --drain-wait argument should first check the values of server.shutdown.drain_wait, server.shutdown.connection_wait, server.shutdown.query_wait, and server.shutdown.lease_transfer_wait, and if needed, increase the value of --drain-wait to be larger than our recommendation.

Describe alternatives you've considered

Instead of automatically computing --drain-wait, we could fail loudly and tell the user to explicitly set drain-wait to an appropriate value.

Jira issue: CRDB-25243

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-server-and-security DB Server & Security
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant