rfcs: update the RFC on multi-tenant cluster settings

[knz]

Note: this is a RFC update, not a new RFC. We are evaluating the 'diff' between the previous and the new version.

Informs #85729

Text of RFC: https://github.com/knz/cockroach/blob/20220811-tenant-settings/docs/RFCS/20211106_multitenant_cluster_settings.md

Release note: None

[cockroach-teamcity]

This change is 

[knz]

@dt @stevendanna could you help here and provide some examples (1-2 would be sufficient) of cluster settings that would benefit from the new definition of system-ro?

[dt]

rangefeeds.enabled is the classic one, where it is a system-only settings (it only controls kvserver behavior) but we want to know its value in sql to provide a better error if it is off and we're about to attempt something that requires it be on.

[ajwerner]

That's my least favorite example because there's such an easy solution to it: make it a zone config instead of a cluster setting.

[dt]

@ajwerner sure, yes, but for now it is a cluster setting ,that (a) controls kv server behavior that (b) SQL behavior wants to know its value, so it fits the bill.

Some hypothetical ones:
kvserver has a cluster setting to limit the number of slots it provides to concurrently process some type of request; observing that limit allows SQL-side senders to adjust their sending concurrency accordingly, as there is no point in them spending more resources to create and send and wait for wildly more requests than the server will possibly handle.

Similar to rangefeeds, if a certain type of kv request has to be enabled/disabled, knowing that fact client-side can allow switching to a different implementation (e.g. poll if you won't be allowed to get a feed, use different backfiller if you can't backdate sstables, etc) or at least supplying a better error.

[ajwerner]

Another one which affects the kvclient and sql layer is the maximum raft command size.

[knz]

Added the examples in the text. Thanks!

[dt]

Looks good to me.

I still prefer to conceptually separate discussion of overrides -- a capability common to any tenant-read setting -- from the two types of tenant read setting and how they differ (reading the host's value vs their local value).

But even if we don't do that, the document is clear with this edits as they are now than it was before, so I'm not going to die on this hill.

[dt]

To me these aren't so much two distinct categories, but rather one category, plus the same override-capability we have for all tenant-* settings, ro or rw.

I'd describe this category as:

• Settings where exposing the system tenant's current value -- whether that is coming from its default or has been configured explicitly -- is desirable. For example, clients typically try to limit the size of batches they send to stay within the limits of what the host cluster is configured to process, making it useful to know the current values of kv.raft.command.max_size and kv.bulk_ingest.batch_size in the host cluster.

• As with all tenant-exposed setting, the value read by a specific or all tenants can be forcibly overridden from the host, which could be useful, for example, as a mitigation measure in an incident to force a tenant who is running several large imports to use a smaller batch size to stop OOMs.

As I sit here and think about it: Is there really any reason tenant_cpu_usage_allowance should actually be tenant-ro versus tenant-rw? Seems like it doesn't really matter that it be tenant-ro if we're just going to override it anyway.

[knz]

i've updated this as per our recent discussions and the work in #111378.

bors r=dt

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)