server: enable data_distribution API for secondary tenants.

[shubhamdhama]

Previously, this endpoint scanned over meta KVs and used their startKey to
determine the table they belong to. However, this approach does not work
for coalesced ranges.

The fix, inspired by the SHOW CLUSTER RANGES query, uses the
crdb_internal.table_spans table and joins it with the
crdb_internal.ranges_no_leases table.

There is still some work left that can be addressed in follow-up PR:

• We might want to add a summation of ranges at various levels because the
  sum of replicas of all tables in a database can be more than the actual
  number of ranges that belong to that database due to coalescing.

Overall, this PR also fixes the broken behavior to a large extent of this
endpoint since range coalescing is enabled.

Informs: #97942
Fixes: #109501, #106897
Epic: CRDB-12100
Release note: None

[cockroach-teamcity]

This change is 

[shubhamdhama]

• We might want to put this behind some capability.

[dhartunian]

• We might want to put this behind some capability.

I'm not sure this is necessary, can you say more about why you think a capability is needed? Are you accessing some information that's not already tenant-scoped? Since you're using the adminServer not the systemAdminServer, the APIs should already account for tenant scoping automatically.

[shubhamdhama]

I'm not sure this is necessary, can you say more about why you think a capability is needed? Are you accessing some information that's not already tenant-scoped? Since you're using the adminServer not the systemAdminServer, the APIs should already account for tenant scoping automatically.

Yes, I wasn't sure whether we would need it or not, but you are right – we don't. The information we are accessing is through SQL, so we are fine on that front, and since it's an admin endpoint, so we are good at the front too.

On an unrelated note, once this PR is merged, I will create an issue for the following:

We might want to add a summation of ranges at various levels because the
sum of replicas of all tables in a database can be more than the actual
number of ranges that belong to that database due to coalescing.

Essentially, this PR has enabled the endpoint for secondary tenants and improved the result too (i.e., they are not as misleading as before).
However, there are few aspects for which I believe observability team would be better equipped to lead, particularly how should this data distribution UI page should adapt to range coalescing. For instance, a single range might intersect with three tables, meaning all three tables would count that same range. This creates a challenge in accurately displaying the total number of ranges. I suggest adding some more fields to the result protos, but that's an implementation detail.

[dhartunian]

can you just file a small follow up to enable it on the tenant DB Console page? it's disabled here:

cockroach/pkg/ui/workspaces/db-console/src/views/reports/containers/debug/index.tsx

Lines 304 to 309 in 33a791d

	<DebugPanelLink
	name="Data Distribution and Zone Configs"
	url="#/data-distribution"
	note="View the distribution of table data across nodes and verify zone configuration."
	disabled={disable_kv_level_advanced_debug}
	/>

However, there are few aspects for which I believe observability team would be better equipped to lead, particularly how should this data distribution UI page should adapt to range coalescing. For instance, a single range might intersect with three tables, meaning all three tables would count that same range. This creates a challenge in accurately displaying the total number of ranges. I suggest adding some more fields to the result protos, but that's an implementation detail.

Yes, we've been discussing what to do about stuff like this as we need to figure out some improvements for the hot ranges page already. You can leave that followup with the obs team.

Reviewed 2 of 3 files at r1, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @kyle-a-wong and @stevendanna)

[stevendanna]

I've left some potential cleanups, but I'm OK with those being in follow-up PRs and/or put into someone's backlog.

[stevendanna]

👍 Thanks for doing that extra cleanup.

Not recommending a change, but just leaving this here as you might find it amusing. The query could be made to return 1 row per table with something like:

   WITH tables AS (
        SELECT
            t.schema_name, t.name AS table_name, t.database_name,
            t.table_id, t.drop_time, s.start_key, s.end_key
        FROM
            "".crdb_internal.tables t
            JOIN "".crdb_internal.table_spans s ON t.table_id = s.descriptor_id
        WHERE
            t.database_name IS NOT NULL
   )
    SELECT
        table_id, table_name, schema_name, database_name, drop_time, array_agg(replica) as replicas
    FROM
    (SELECT
        t.table_id, t.table_name, t.schema_name, t.database_name, t.drop_time, unnest(r.replicas) as replica
     FROM
        tables t
        JOIN "".crdb_internal.ranges_no_leases r ON t.start_key < r.end_key
        AND t.end_key > r.start_key
    ) GROUP BY table_id, table_name, schema_name, database_name, drop_time;

I have no idea how this would perform in a big cluster vs what you have.

[shubhamdhama]

Yes, it should make that code more fun. Thanks for playing around with it.
This reminds me of a book in my list: https://theartofpostgresql.com "Turn thousands of lines of code into simple queries".

[shubhamdhama]

Thanks everyone for the review ✌️

bors r=dhartunian,stevendanna

[craig]

Build succeeded:

• local_roachtest_fips
• docker_image_amd64
• unit_tests
• linux_amd64_build
• local_roachtest
• check_generated_code
• macos_arm64_build
• macos_amd64_build
• acceptance
• windows_build
• examples_orms
• lint
• linux_amd64_fips_build
• linux_arm64_build