Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: SHOW RANGES broken by coalesced ranges #93617

Closed
knz opened this issue Dec 14, 2022 · 1 comment · Fixed by #93644
Closed

sql: SHOW RANGES broken by coalesced ranges #93617

knz opened this issue Dec 14, 2022 · 1 comment · Fixed by #93644
Assignees
Labels
A-kv-observability A-sql-vtables Virtual tables - pg_catalog, information_schema etc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-kv KV Team

Comments

@knz
Copy link
Contributor

knz commented Dec 14, 2022

Describe the problem

The output of SHOW RANGES assumes that a range maps to at most one db, table, index.

Since recently, ranges coalesce across tables and even databases. This means that the db/table/index name fields are now incorrect; they merely describe the data at the beginning of the range but do not include data/tables that's in the middle or at the end.

This has hilarious effects:

> create table db1; create table db1.t as select 1 as x; 
> create table db2; create table db2.t as select 1 as x; 
> show ranges from database d1;
  table_name | start_key | end_key | range_id |       range_size_mb        | lease_holder | lease_holder_locality | replicas |    replica_localities
-------------+-----------+---------+----------+----------------------------+--------------+-----------------------+----------+---------------------------
  t          | /108      | NULL    |       56 | 0.000068000000000000000000 |            1 | region=us-east1,az=b  | {1}      | {"region=us-east1,az=b"}

> show ranges from database db2;

SHOW RANGES 0

oops!
(the problem is because the database_name field in the range is computed on the start key, and there's no separate range with db2 as its database_name)

Additionally, the start_key/end_key values are often broken, because they don't fit within the keyspace of the selected index (in show ranges from table / from index).

Expected behavior

Fundamentally the assumption that "1 range <-> at most one object" is broken so we can't repair this with the current schema of the show ranges output, nor the underlying crdb_internal.ranges. Both need to change.

We have two alternatives:

  • keep the guarantee "1 row per range" (range_id remains unique), but then include a list of designated objects elsewhere on the row, e.g. via a new column with type ARRAY.
  • keep the guarantee "at least 1 row per object" (db/table/index names remain unique), but then introduce synthetic range rows, 1 per SQL object included in the range, with synthetic start/end keys that map to just the object selected, and the range_id repeated across rows.

Given how we're using this facility in tests, it looks like both are desirable!

So what can happen is likely that we'll split the vtable in two: one with the former behavior and one with the latter.

Jira issue: CRDB-22439

Epic CRDB-22701

@knz knz added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-sql-vtables Virtual tables - pg_catalog, information_schema etc A-kv-observability T-kv-observability labels Dec 14, 2022
@arulajmani
Copy link
Collaborator

Relates to #93198

@exalate-issue-sync exalate-issue-sync bot added T-kv KV Team and removed T-kv-observability labels Dec 22, 2022
@craig craig bot closed this as completed in 6b72f70 Dec 22, 2022
irfansharif added a commit to irfansharif/cockroach that referenced this issue May 7, 2023
Fixes cockroachdb#81008.

We built the basic infrastructure to coalesce ranges across table
boundaries back in 22.2 as part of cockroachdb#66063. We've enabled this
optimization for secondary tenants since then, but hadn't for the system
tenant because of two primary blockers:

- cockroachdb#93617: SHOW RANGES was broken by coalesced ranges.
- cockroachdb#84105: APIs to compute sizes for schema objects (used in our UI) was
  broken by coalesced ranges.

In both these cases we baked in assumptions about there being a minimum
of one-{table,index,partition}-per-range. These blockers didn't apply to
secondary tenants at the time since they didn't have access to SHOW
RANGES, nor the UI pages where these schema statistics were displayed.

We've addressed both these blockers in the 23.1 cycle as part of
bridging the compatibility between secondary tenants and yesteryear's
system tenant.
- cockroachdb#93644 revised SHOW RANGES and crdb_internal.ranges{,_no_leases}, both
  internally and its external UX, to accommodate ranges straddling
  table/database boundaries.
- cockroachdb#96223 re-worked our SpanStats API to work in the face of coalesced
  ranges, addressing cockroachdb#84105.

Release note (general change): CockroachDB would previously use separate
ranges for each table, index, or partition. This is no longer true --
it's possible now to have multiple tables, indexes, and partitions to
get packed into the same range. For users with many such "schema
objects", this will reduce the total range count in their clusters. This
is especially true if individual tables, indexes, or partitions are
smaller than the default configured maximum range size (controlled using
zone configs, specifically the range_max_bytes parameter).
We made this change to improve scalability with respect to the number of
schema objects, since the underlying range count now no longer a
bottleneck. Users upgrading from 22.2, once finalizing their upgrade,
may observe a round of range merges and snapshot transfers (to power
said range merges) as a result of this change. If users want to opt-out
of this optimization, they can use the following cluster setting:

  SET CLUSTER SETTING spanconfig.storage_coalesce_adjacent.enabled = false;
irfansharif added a commit to irfansharif/cockroach that referenced this issue May 8, 2023
Fixes cockroachdb#81008.

We built the basic infrastructure to coalesce ranges across table
boundaries back in 22.2 as part of cockroachdb#66063. We've enabled this
optimization for secondary tenants since then, but hadn't for the system
tenant because of two primary blockers:

- cockroachdb#93617: SHOW RANGES was broken by coalesced ranges.
- cockroachdb#84105: APIs to compute sizes for schema objects (used in our UI) was
  broken by coalesced ranges.

In both these cases we baked in assumptions about there being a minimum
of one-{table,index,partition}-per-range. These blockers didn't apply to
secondary tenants at the time since they didn't have access to SHOW
RANGES, nor the UI pages where these schema statistics were displayed.

We've addressed both these blockers in the 23.1 cycle as part of
bridging the compatibility between secondary tenants and yesteryear's
system tenant.
- cockroachdb#93644 revised SHOW RANGES and crdb_internal.ranges{,_no_leases}, both
  internally and its external UX, to accommodate ranges straddling
  table/database boundaries.
- cockroachdb#96223 re-worked our SpanStats API to work in the face of coalesced
  ranges, addressing cockroachdb#84105.

Release note (general change): CockroachDB would previously use separate
ranges for each table, index, or partition. This is no longer true --
it's possible now to have multiple tables, indexes, and partitions to
get packed into the same range. For users with many such "schema
objects", this will reduce the total range count in their clusters. This
is especially true if individual tables, indexes, or partitions are
smaller than the default configured maximum range size (controlled using
zone configs, specifically the range_max_bytes parameter).
We made this change to improve scalability with respect to the number of
schema objects, since the underlying range count now no longer a
bottleneck. Users upgrading from 22.2, once finalizing their upgrade,
may observe a round of range merges and snapshot transfers (to power
said range merges) as a result of this change. If users want to opt-out
of this optimization, they can use the following cluster setting:

  SET CLUSTER SETTING spanconfig.storage_coalesce_adjacent.enabled = false;
craig bot pushed a commit that referenced this issue May 8, 2023
98820: spanconfig: enable range coalescing by default r=irfansharif a=irfansharif

Fixes #81008.

We built the basic infrastructure to coalesce ranges across table boundaries back in 22.2 as part of #66063. We've enabled this optimization for secondary tenants since then, but hadn't for the system tenant because of two primary blockers:

- #93617: SHOW RANGES was broken by coalesced ranges.
- #84105: APIs to compute sizes for schema objects (used in our UI) was broken by coalesced ranges.

In both these cases we baked in assumptions about there being a minimum of one-{table,index,partition}-per-range. These blockers didn't apply to secondary tenants at the time since they didn't have access to SHOW RANGES, nor the UI pages where these schema statistics were displayed.

We've addressed both these blockers in the 23.1 cycle as part of bridging the compatibility between secondary tenants and yesteryear's system tenant.
- #93644 revised SHOW RANGES and crdb_internal.ranges{,_no_leases}, both internally and its external UX, to accommodate ranges straddling table/database boundaries.
- #96223 re-worked our SpanStats API to work in the face of coalesced ranges, addressing #84105.

Release note (general change): CockroachDB would previously use separate ranges for each table, index, or partition. This is no longer true -- it's possible now to have multiple tables, indexes, and partitions to get packed into the same range. For users with many such "schema objects", this will reduce the total range count in their clusters. This is especially true if individual tables, indexes, or partitions are smaller than the default configured maximum range size (controlled using zone configs, specifically the range_max_bytes parameter). We made this change to improve scalability with respect to the number of schema objects, since the underlying range count now no longer a bottleneck. Users upgrading from 22.2, once finalizing their upgrade, may observe a round of range merges and snapshot transfers (to power said range merges) as a result of this change. If users want to opt-out of this optimization, they can use the following cluster setting:
```
  SET CLUSTER SETTING spanconfig.storage_coalesce_adjacent.enabled = false; 
```

Co-authored-by: irfan sharif <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv-observability A-sql-vtables Virtual tables - pg_catalog, information_schema etc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-kv KV Team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants