multi-tenant: Relax restrictions on supported zone configuration changes from secondary tenants

[ajstorm]

Originally we were of the opinion that zone configurations in secondary tenants would only be updated by the multi-region semantics. As a result, we restricted their use to only specify constraints which referenced the region or zone tiers:

cockroach/pkg/sql/set_zone_config.go

Lines 991 to 1020 in 81c447d

	for _, constraint := range toValidate {
	switch constraint.Key {
	case "zone":
	_, found := zones[constraint.Value]
	if !found {
	return pgerror.Newf(
	pgcode.CheckViolation,
	"zone %q not found",
	constraint.Value,
	)
	}
	case "region":
	_, found := regions[constraint.Value]
	if !found {
	return pgerror.Newf(
	pgcode.CheckViolation,
	"region %q not found",
	constraint.Value,
	)
	}
	default:
	return errors.WithHint(pgerror.Newf(
	pgcode.CheckViolation,
	"invalid constraint attribute: %q",
	constraint.Key,
	),
	`only "zone" and "region" are allowed`,
	)
	}
	}

The potential problem with this approach is that if we're going to make multi-tenant deployments the default deployment pattern for CRDB (something which is still being debated), this restriction will break some users. We'll need to relax this restriction if/when we go down that route.

Jira issue: CRDB-12730

Epic CRDB-26686

[arulajmani]

Originally we were of the opinion that zone configurations in secondary tenants would only be updated by the multi-region semantics. As a result, we restricted their use to only specify constraints which referenced the region or zone tiers:

It might be just me, but I found this phrasing slightly ambiguous. To clarify, we only restrict secondary tenants' to reference zone/region tiers in the constraints/voter constraints/lease preferences fields. However, they are allowed to alter all other fields, such as num_voters, gc.ttl etc.

The reason for this limitation is that secondary tenants do not have access to the NodeStatusServer, and in turn, the underlying store descriptors. So they can't validate constraints such as hardware or other locality tiers. It's an open question if secondary tenants should have a wider view than what the RegionServer offers today, but if we were to support this functionality, we'd need to expose more things on the RegionServer.

[yuzefovich]

@knz suggested that this issue has been effectively superseded by #100787 which is next on Jeff's plate. I'll make some minor adjustments to the code to highlight that issue in error messages.