Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: configurable clock offset verification threshold #94999

Closed
erikgrinaker opened this issue Jan 10, 2023 · 0 comments · Fixed by #96141
Closed

server: configurable clock offset verification threshold #94999

erikgrinaker opened this issue Jan 10, 2023 · 0 comments · Fixed by #96141
Assignees
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@erikgrinaker
Copy link
Contributor

erikgrinaker commented Jan 10, 2023

We use RPC pings to estimate the remote clock offset, and terminate the node if its offset is greater than MaxOffset away from the median clock reading.

Some clusters may run with fairly precise clocks, and want to reduce the MaxOffset to e.g. 10ms. At these scales, uncertainty in the clock offset estimate can lead to false positives, prematurely terminating the node. Furthermore, the latency measurements from the RPC heartbeats can be wildly inaccurate because it's vulnerable to RPC head-of-line blocking (#93397), which also affects the clock uncertainty measurements.

We should provide an option to set an explicit threshold for these uncertainty checks, rather that using MaxOffset, such that the nodes can run with e.g. a 10 ms max offset but only self-terminate when they see obviously-incorrect clock offsets by e.g. 1 seconds.

Jira issue: CRDB-23265

@erikgrinaker erikgrinaker added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team labels Jan 10, 2023
@erikgrinaker erikgrinaker self-assigned this Jan 28, 2023
@craig craig bot closed this as completed in 18c5a38 Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant