-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kvserver: use WithExplicitX
for snapshot rate limits
#75728
Comments
We've seen recent problems (cockroachdb#75728) related to mis-configured cluster settings. Preemptively tighten the range of allowable values for the `kv.replica_circuit_breaker.slow_replication_threshold` setting. Release note: None
Related to cockroachdb#75728. This commit enforces a minimum Raft snapshot rate limit of 1mb/s. We've seen recent problems (cockroachdb#75728) related to mis-configured cluster settings, so it's best to protect ourselves from this kind of misconfiguration. ``` [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8 B [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Kib'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8.0 KiB [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Mib'; SET CLUSTER SETTING ```
Related to cockroachdb#75728. This commit enforces a minimum Raft snapshot rate limit of 1mb/s. We've seen recent problems (cockroachdb#75728) related to mis-configured cluster settings, so it's best to protect ourselves from this kind of misconfiguration. ``` [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8 B [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Kib'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8.0 KiB [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Mib'; SET CLUSTER SETTING ```
75889: server: use constant for filtering internal stmt stats r=xinhaoz a=xinhaoz Previously, the internal app name prefix was hard-coded into the query used to fetch combined stmt/txn stats. This commit replaces that usage with the pre-existing internal app prefix defined in constants. Release note: None 76209: kv: enforce minimum Raft snapshot rate limit r=nvanbenschoten a=nvanbenschoten Related to #75728. This commit enforces a minimum Raft snapshot rate limit of 1mb/s. We've seen recent problems (#75728) related to mis-configured cluster settings, so it's best to protect ourselves from this kind of misconfiguration. ``` [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8 B [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Kib'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8.0 KiB [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Mib'; SET CLUSTER SETTING ``` 76236: tlp: respect test shutdown deadline r=jordanlewis a=jordanlewis Closes #74675 This commit fixes the TLP test shutdown routine to actually begin shutting down 5 minutes before the test timeout. Before, the code that was supposed to check the timeout was not reached reliably. Release note: None Co-authored-by: Xin Hao Zhang <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]> Co-authored-by: Jordan Lewis <[email protected]>
Related to cockroachdb#75728. This commit enforces a minimum Raft snapshot rate limit of 1mb/s. We've seen recent problems (cockroachdb#75728) related to mis-configured cluster settings, so it's best to protect ourselves from this kind of misconfiguration. ``` [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8 B [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Kib'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8.0 KiB [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Mib'; SET CLUSTER SETTING ```
76144: kvserver: enforce min duration for slow_replication_threshold r=erikgrinaker a=tbg We've seen recent problems (#75728) related to mis-configured cluster settings. Preemptively tighten the range of allowable values for the `kv.replica_circuit_breaker.slow_replication_threshold` setting. Touches #33007. Release note: None Co-authored-by: Tobias Grieger <[email protected]>
Related to cockroachdb#75728. This commit enforces a minimum Raft snapshot rate limit of 1mb/s. We've seen recent problems (cockroachdb#75728) related to mis-configured cluster settings, so it's best to protect ourselves from this kind of misconfiguration. ``` [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8 B [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Kib'; ERROR: invalid value for kv.snapshot_rebalance.max_rate: snapshot rate cannot be set to a value below 1.0 MiB: 8.0 KiB [email protected]:26257/movr> set cluster setting kv.snapshot_rebalance.max_rate = '8Mib'; SET CLUSTER SETTING ```
We've seen recent problems (cockroachdb#75728) related to mis-configured cluster settings. Preemptively tighten the range of allowable values for the `kv.replica_circuit_breaker.slow_replication_threshold` setting. Release note: None
This is somewhat awkward: for each setting type that we want this for, we need to implement a new type and special case it here: cockroach/pkg/sql/set_cluster_setting.go Lines 157 to 162 in 6700a17
I was hoping we could "just" have an Since we already set min values for these rate limits, I am not convinced we should be doing more this time around. It does seem desirable to have better settings infra though, where units can be requested for any setting without doing plumbing in the SQL layer. cc @mwang1026 not sure where to route this ask. |
I think given Nathan's min setting change we're OK to close this for now. re: cluster setting infra let's take that offline |
Is your feature request related to a problem? Please describe.
A customer (internal) accidentally set the snapshot rate limits to
8
, which is 8 bytes per second. They intended to set it to 8mb/s.Describe the solution you'd like
Use the equivalent of
cockroach/pkg/settings/duration.go
Lines 155 to 160 in 30afca0
for all of our cluster settings for which it makes sense.
Note that this isn't a backwards-compatible change, but we should make it regardless.
Jira issue: CRDB-12806
The text was updated successfully, but these errors were encountered: