Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: selectively enqueue replicas into the replicate queue on span config updates #108795

Closed
kvoli opened this issue Aug 15, 2023 · 0 comments · Fixed by #108809
Closed

kvserver: selectively enqueue replicas into the replicate queue on span config updates #108795

kvoli opened this issue Aug 15, 2023 · 0 comments · Fixed by #108809
Assignees
@andrewbaptist
Labels
A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team

Comments

@kvoli
Copy link
Collaborator

kvoli commented Aug 15, 2023 •
edited by cockroach-jira-scripts
Loading

Is your feature request related to a problem? Please describe.
In #100349, we began enqueuing replicas (into the replicate queue) upon receiving span config updates. This came with an unreasonable overhead in clusters with many ranges, described in #108724.

Not all span config updates require replication changes. Only changes to lease_preferences, constraints, voter_constraints, num_replicas and num_voters could possibly require replication changes.

Describe the solution you'd like

Only enqueue into the replicate queue for span config updates when there's a change to one of the fields mentioned above, from the last span config seen.

Jira issue: CRDB-30641
@kvoli kvoli added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-distribution Relating to rebalancing and leasing. labels Aug 15, 2023
@kvoli kvoli added the T-kv KV Team label Aug 15, 2023
@andrewbaptist andrewbaptist self-assigned this Aug 15, 2023
kvoli added a commit to kvoli/cockroach that referenced this issue Aug 15, 2023
@kvoli
kvserver: enable replicate enqueue on span cfg updates by default
a285fcb 
`kv.enqueue_in_replicate_queue_on_span_config_update.enabled` controls
whether replicas are enqueued into the replicate queue, upon a span
config update. The setting was disabled due to overhead. Re-enable the
setting, as we anticipate significantly reducing the overhead in

Informs: cockroachdb#108795
Release note (ops change): Enable
`kv.enqueue_in_replicate_queue_on_span_config_update.enabled`
by default. This setting speeds up how quickly replication changes
begin, after changing a related configuration e.g. database primary
region or replication factor.
@kvoli kvoli mentioned this issue Aug 15, 2023
Closed
@andrewbaptist andrewbaptist mentioned this issue Aug 16, 2023
Merged
andrewbaptist added a commit to andrewbaptist/cockroach that referenced this issue Aug 16, 2023
@andrewbaptist
kvserver: queue replicas on significant span config changes
8b8703c 
Previously we would queue replicas for split, merge and replicate
changes on every change to the span configs including PTS changes. This
was inefficient and now we check if the change to the SpanConfig is
significant.

Informs: cockroachdb#107977
Fixes: cockroachdb#108795

Epic: none

Release note: None
andrewbaptist added a commit to andrewbaptist/cockroach that referenced this issue Sep 8, 2023
@andrewbaptist
kvserver: queue replicas on significant span config changes
dcaf488 
Previously we would queue replicas for split, merge and replicate
changes on every change to the span configs including PTS changes. This
was inefficient and now we check if the change to the SpanConfig is
significant.

Informs: cockroachdb#107977
Fixes: cockroachdb#108795

Epic: none

Release note: None
andrewbaptist added a commit to andrewbaptist/cockroach that referenced this issue Sep 8, 2023
@andrewbaptist
kvserver: queue replicas on significant span config changes
2de1db7 
Previously we would queue replicas for split, merge and replicate
changes on every change to the span configs including PTS changes. This
was inefficient and now we check if the change to the SpanConfig is
significant.

Informs: cockroachdb#107977
Fixes: cockroachdb#108795

Epic: none

Release note: None
craig bot pushed a commit that referenced this issue Sep 13, 2023
@andrewbaptist
Merge #108809
81313b8 
108809: kvserver: queue replicas on significant span config changes r=kvoli a=andrewbaptist

Previously we would queue replicas for split, merge and replicate
changes on every change to the span configs including PTS changes. This
was inefficient and now we check if the change to the SpanConfig is
significant.

Informs: #107977
Fixes: #108795

Epic: none

Release note: None

Co-authored-by: Andrew Baptist <[email protected]>
@craig craig bot closed this as completed in c09f1b9 Sep 13, 2023
This was referenced Jul 15, 2024
Open
Closed
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andrewbaptist andrewbaptist

Labels
A-kv-distribution Relating to rebalancing and leasing. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv KV Team
Projects
No open projects
KV
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: queue replicas on significant span config changes andrewbaptist/cockroach
2 participants
@kvoli @andrewbaptist