server: make /data-distribution compatible with range coalescing

[dhartunian]

Current implementation returns an error.

Epic: CRDB-12100

Jira issue: CRDB-24980

[irfansharif]

We'd have to teach the underlying API to deal with coalesced ranges (#98820), and reflect it in the visualization as well.

[abarganier]

Some notes after discussing with @dhartunian:

• Alongside this, it would be nice to have a tenant capability that gates range-level debug information (e.g. ability to view advanced debug pages such as problem ranges, /ranges, etc.)
• /data-distribution can be gated behind the new capability as well (if it hasn't already been tenant scoped)
• Some range-level pages already exist at a tenant-scoped level, while others show range information across the entire cluster. Identify those that fall into the latter category, and gate behind the new capability.

[abarganier]

Given that this endpoint doesn't take coalesced ranges into account, and tenants' ranges are always coalesced, it doesn't make sense to implement it for secondary tenants.

Instead, we're going to make the endpoint system-tenant-only until KV updates it to take coalesced ranges into account (or nukes it entirely).

[abarganier]

Marking as closed - see #100620 which disables this endpoint for secondary tenants.

Context here: #97942 (comment)

[knz]

The feature will become broken for the system tenant as well in v23.2, because we will enable range coalescing by default.

I am reopening this issue now to decide what to do with this. We have only 2 options:

• either remove the feature entirely
• or, make it work with range coalescing

The current approach (ref) which keeps it but informs the user in DB console that "it is broken for coalesced ranges" is simply not acceptable. Tthe feature will break in 23.2 for all clusters if we don't take action.

[shubhamdhama]

So, if I understand correctly, the data_distribution endpoint returns the count of range replicas per node for each table (along with some zone config information, which is not broken).

The problem seems to be here:

cockroach/pkg/server/admin.go

Lines 3039 to 3094 in d527c4c

	// Get replica counts.
	if err := s.db.Txn(ctx, func(txnCtx context.Context, txn *kv.Txn) error {
	acct := s.memMonitor.MakeBoundAccount()
	defer acct.Close(txnCtx)
	kvs, err := kvclient.ScanMetaKVs(ctx, txn, roachpb.Span{
	Key: keys.SystemSQLCodec.TablePrefix(keys.MaxReservedDescID + 1),
	EndKey: keys.MaxKey,
	})
	if err != nil {
	return err
	}
	// Group replicas by table and node, accumulate counts.
	var rangeDesc roachpb.RangeDescriptor
	for _, kv := range kvs {
	if err := acct.Grow(txnCtx, int64(len(kv.Key)+len(kv.Value.RawBytes))); err != nil {
	return err
	}
	if err := kv.ValueProto(&rangeDesc); err != nil {
	return err
	}
	// TODO(embrown): Tables can use one codec since they
	// seem to all share the same id.
	_, tenID, err := keys.DecodeTenantPrefix(rangeDesc.StartKey.AsRawKey())
	if err != nil {
	return err
	}
	// A range descriptor for a secondary tenant may not contain
	// a table prefix. Often, the start key for a tenant will be just
	// the tenant prefix itself, e.g. `/Tenant/2`. Once the tenant prefix
	// is stripped inside `DecodeTablePrefix`, nothing (aka `/Min`) is left.
	keySansPrefix, _ := keys.MakeSQLCodec(tenID).StripTenantPrefix(rangeDesc.StartKey.AsRawKey())
	if keys.MinKey.Equal(keySansPrefix) {
	// There's no table prefix to be decoded.
	// Try the next descriptor.
	continue
	}
	_, tableID, err := keys.MakeSQLCodec(tenID).DecodeTablePrefix(rangeDesc.StartKey.AsRawKey())
	if err != nil {
	return err
	}
	for _, replicaDesc := range rangeDesc.Replicas().Descriptors() {
	tableInfo, found := tableInfosByTableID[tableID]
	if !found {
	// This is a database, skip.
	continue
	}
	tableInfo.ReplicaCountByNodeId[replicaDesc.NodeID]++
	}
	}
	return nil
	}); err != nil {
	return nil, err
	}

We retrieve all the meta KVs and use the start key to determine the table ID. However, for coalesced ranges, we can have data from multiple tables within the same range.

Initially, it seemed like the work would be easy since I could have simply used SHOW CLUSTER RANGES, but that command doesn't show details for a deleted table (that hasn't been garbage collected yet), which is currently supported by this endpoint for uncoalesced ranges.

So I explored the queries that `SHOW CLUSTER...` uses underneath

WITH ranges AS (
    SELECT
        r.range_id,
        r.start_key,
        r.end_key,
        r.replicas,
        r.replica_localities,
        r.voting_replicas,
        r.non_voting_replicas,
        r.learner_replicas,
        r.split_enforced_until,
        s.descriptor_id AS table_id,
        s.start_key AS object_start_key,
        s.end_key AS object_end_key,
        IF (s.start_key < r.start_key,
            r.start_key,
            s.start_key) AS per_range_start_key,
        IF (s.end_key < r.end_key,
            s.end_key,
            r.end_key) AS per_range_end_key
    FROM
        crdb_internal.ranges_no_leases r
        LEFT OUTER JOIN "".crdb_internal.table_spans s ON s.start_key < r.end_key AND s.end_key > r.start_key
),
named_ranges AS (
SELECT
    r.*,
    database_name,
    t.schema_name,
    t.name AS table_name
FROM
    ranges r
    LEFT OUTER JOIN "".crdb_internal.tables t ON r.table_id = t.table_id
),
intermediate AS (
SELECT
    r.*
FROM
    named_ranges r
)
SELECT
    crdb_internal.pretty_key(r.start_key, -1) AS start_key,
    crdb_internal.pretty_key(r.end_key, -1) AS end_key,
    range_id,
    database_name,
    schema_name,
    table_name,
    table_id,
    crdb_internal.pretty_key(per_range_start_key, -1) AS table_start_key,
    crdb_internal.pretty_key(per_range_end_key, -1) AS table_end_key,
    replicas,
    replica_localities,
    voting_replicas,
    non_voting_replicas,
    learner_replicas,
    split_enforced_until
FROM
    intermediate r
ORDER BY
    r.start_key,
    r.per_range_start_key

While select * from "".crdb_internal.tables contains such deleted tables, select * from "".crdb_internal.table_spans does not, which is bit weird because if you add a ... WHERE descriptor_id clause it also shows that particular deleted table.

Now, I wonder if there is a simpler way at the KV layer to achieve our goal instead of using SQL queries. If not, I can think of creating a new version of or modifying crdb_internal.table_spans so that I can use in JOIN.

[shubhamdhama]

While this endpoint is enabled for secondary tenants, there are still some follow-ups needed to make it user friendly and compatible with range coalescing: #128578 (comment). I'm removing my assignment as discussed (#128578 (review))