-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
release-23.1: sql: extend SHOW RANGES to include object size estimates #105306
Conversation
This commit adds object size estimates to `SHOW RANGES WITH DETAILS`. A new override of `crdb_internal.tenant_span_stats` is introduced to leverage batched span statistics requests to KV, so only 1 fan-out is required. Resolves: cockroachdb#97858 Epic: https://cockroachlabs.atlassian.net/browse/CRDB-24928 Release note (sql change): The SHOW RANGES command will now emit span statistics when the DETAILS option is specified. The statistics are included in a new column named 'span_stats', as a JSON object. The statistics are calculated for the identifier of each row: `SHOW RANGES WITH DETAILS` will compute span statistics for each range. `SHOW RANGES WITH TABLES, DETAILS` will compute span statistics for each table, etc. The 'span_stats' JSON object has the following keys: - approximate_disk_bytes - [key|val|sys|live|intent]_count - [key|val|sys|live|intent]_bytes 'approximate_disk_bytes' is an approximation of the total on-disk size of the given object. 'key_count' is the number of meta keys tracked under key_bytes. 'key_bytes' is the number of bytes stored in all non-system point keys, including live, meta, old, and deleted keys. Only meta keys really account for the "full" key; value keys only for the timestamp suffix. 'val_count' is the number of meta values tracked under val_bytes. 'val_bytes' is the number of bytes in all non-system version values, including meta values. 'sys_count' is the number of meta keys tracked under sys_bytes. 'sys_bytes' is the number of bytes stored in system-local kv-pairs. This tracks the same quantity as (key_bytes + val_bytes), but for system-local metadata keys (which aren't counted in either key_bytes or val_bytes). Each of the keys falling into this group is documented in keys/constants.go under the LocalPrefix constant and is prefixed by either LocalRangeIDPrefix or LocalRangePrefix. 'live_count' is the number of meta keys tracked under live_bytes. 'live_bytes' is the number of bytes stored in keys and values which can in principle be read by means of a Scan or Get in the far future, including intents but not deletion tombstones (or their intents). Note that the size of the meta kv pair (which could be explicit or implicit) is included in this. Only the meta kv pair counts for the actual length of the encoded key (regular pairs only count the timestamp suffix). 'intent_count' is the number of keys tracked under intent_bytes. It is equal to the number of meta keys in the system with a non-empty Transaction proto. 'intent_bytes' is the number of bytes in intent key-value pairs (without their meta keys).
Thanks for opening a backport. Please check the backport criteria before merging:
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
Add a brief release justification to the body of your PR to justify this backport. Some other things to consider:
|
@rafiss, sorry about that - I did a bad rebase and had to reopen the PR. |
This shouldn't be backported until we resolve #105274, since it breaks |
Beware that the problem in #105638 still exists. Is it clever to introduce this functionality in 23.1 if it's going to break for any customer with a sizeable cluster? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In an abundance of caution i'm going to retract my approval until we make progress on #105638.
We'll also need to solve #106097. |
Backport 1/1 commits from #103128.
/cc @cockroachdb/release
Release justification: Important observability feature
This commit adds object size estimates to
SHOW RANGES WITH DETAILS
.A new override of
crdb_internal.tenant_span_stats
is introducedto leverage batched span statistics requests to KV, so only 1 fan-out is
required.
Resolves: #97858
Epic: https://cockroachlabs.atlassian.net/browse/CRDB-24928
Release note (sql change): The SHOW RANGES command will now
emit span statistics when the DETAILS option is specified.
The statistics are included in a new column named 'span_stats',
as a JSON object.
The statistics are calculated for the identifier of each row:
SHOW RANGES WITH DETAILS
will compute span statistics for each range.SHOW RANGES WITH TABLES, DETAILS
will compute span statisticsfor each table, etc.
The 'span_stats' JSON object has the following keys:
'approximate_disk_bytes' is an approximation of the total on-disk size
of the given object.
'key_count' is the number of meta keys tracked under key_bytes.
'key_bytes' is the number of bytes stored in all non-system
point keys, including live, meta, old, and deleted keys.
Only meta keys really account for the "full" key; value
keys only for the timestamp suffix.
'val_count' is the number of meta values tracked under val_bytes.
'val_bytes' is the number of bytes in all non-system version
values, including meta values.
'sys_count' is the number of meta keys tracked under sys_bytes.
'sys_bytes' is the number of bytes stored in system-local kv-pairs.
This tracks the same quantity as (key_bytes + val_bytes), but
for system-local metadata keys (which aren't counted in either
key_bytes or val_bytes). Each of the keys falling into this group
is documented in keys/constants.go under the LocalPrefix constant
and is prefixed by either LocalRangeIDPrefix or LocalRangePrefix.
'live_count' is the number of meta keys tracked under live_bytes.
'live_bytes' is the number of bytes stored in keys and values which can
in principle be read by means of a Scan or Get in the far future,
including intents but not deletion tombstones (or their intents).
Note that the size of the meta kv pair (which could be explicit or implicit)
is included in this. Only the meta kv pair counts for the actual length
of the encoded key (regular pairs only count the timestamp suffix).
'intent_count' is the number of keys tracked under intent_bytes.
It is equal to the number of meta keys in the system with
a non-empty Transaction proto.
'intent_bytes' is the number of bytes in intent key-value
pairs (without their meta keys).