Sentry: error.go:20: unexpected error: forecasted histogram had first bucket with non-zero NumRange or DistinctRange: [{"name":"__forecast__","created_at":"2024-11-01 08:09:36.42883425 +0000 UTC","col... #134031
Assignees
Labels
branch-master
Failures and bugs on the master branch.
branch-release-24.3
Used to mark GA and release blockers, technical advisories, and bugs for 24.3
C-bug
Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
O-sentry
Originated from an in-the-wild panic report.
T-sql-queries
SQL Queries Team
X-blathers-triaged
blathers was able to find an owner
Comments
cockroach-sentry
added
C-bug
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Hi @cockroach-sentry, please add branch-* labels to identify which branch(es) this C-bug affects. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
yuzefovich
added
T-sql-queries
|
dup of #93892 |
|
In this one, it looks like we have some extra distinct count, so we call addOuterBuckets, and then while merging partial stats we call stripOuterBuckets, and then we forecast, and end up with a first bucket with the extra distinct count in the forecasted histogram. |
|
Here's a repro: CREATE TABLE a (
a INT PRIMARY KEY
) WITH (sql_stats_automatic_collection_enabled = false);
INSERT INTO a SELECT 2 * i * i FROM generate_series(0, 1000) s(i);
CREATE STATISTICS __auto__ FROM a;
INSERT INTO a VALUES (2000002);
SELECT pg_sleep(1);
CREATE STATISTICS __auto__ FROM a;
INSERT INTO a VALUES (2000004);
SELECT pg_sleep(1);
CREATE STATISTICS __auto__ FROM a;
INSERT INTO a VALUES (3000000);
SELECT pg_sleep(2);
CREATE STATISTICS __auto_partial__ FROM a USING EXTREMES;
SHOW STATISTICS FOR TABLE a WITH MERGE, FORECAST;
SELECT jsonb_array_elements(stat->'histo_buckets')
FROM (
SELECT jsonb_array_elements(statistics) AS stat
FROM [SHOW STATISTICS USING JSON FOR TABLE a WITH MERGE, FORECAST]
)
WHERE stat->'name' <@ '["__merged__", "__forecast__"]'; |
michae2
added a commit
to michae2/cockroach
that referenced
this issue
Nov 15, 2024
Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like cockroachdb#93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: cockroachdb#93892 Fixes: cockroachdb#134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None
michae2
added a commit
to michae2/cockroach
that referenced
this issue
Nov 16, 2024
Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like cockroachdb#93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: cockroachdb#93892 Fixes: cockroachdb#134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None
craig bot
pushed a commit
that referenced
this issue
Nov 16, 2024
135310: sql/stats: more fully undo addOuterBuckets in stripOuterBuckets r=yuzefovich a=michae2 Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like #93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: #93892 Fixes: #134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None Co-authored-by: Michael Erickson <[email protected]>
yuzefovich
added
branch-master
craig bot
pushed a commit
that referenced
this issue
Nov 20, 2024
135310: sql/stats: more fully undo addOuterBuckets in stripOuterBuckets r=yuzefovich,mgartner a=michae2 Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like #93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: #93892 Fixes: #134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None Co-authored-by: Michael Erickson <[email protected]>
blathers-crl bot
pushed a commit
that referenced
this issue
Nov 20, 2024
Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like #93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: #93892 Fixes: #134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None
sambhav-jain-16
pushed a commit
to sambhav-jain-16/cockroach
that referenced
this issue
Nov 20, 2024
Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like cockroachdb#93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: cockroachdb#93892 Fixes: cockroachdb#134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None
sambhav-jain-16
pushed a commit
to sambhav-jain-16/cockroach
that referenced
this issue
Nov 20, 2024
Near the end of `stats.(*histogram).adjustCounts`, if there is leftover DistinctCount after adjusting each histogram bucket, we call `addOuterBuckets`. This function creates two "outer" buckets that fully cover the lower and upper ends of the domain with the remaining DistinctCount. These "outer" buckets interfere with partial stats, so we remove them with `stripOuterBuckets` when merging partial stats. `stripOuterBuckets` was simply slicing off the two buckets, but this could potentially leave non-zero DistinctCount in the first bucket (which is added by `addOuterBuckets`). `stripOuterBuckets` needs to also zero the DistinctCount and NumRange of the first bucket to fully undo the actions of `addOuterBuckets`. We were only catching this when trying to forecast using the merged histogram, which would sometimes produce a forecasted histogram with non-zero DistinctRange in the first bucket, leading to the infamous "histogram had first bucket with non-zero NumRange or DistinctRange" Sentry failure. Unfortunately, I don't think this bug fully explains all of the Sentry failures that look like cockroachdb#93892 because many of those predate the introduction of partial stats. I believe this specific failure started with v24.3.0-alpha.0. Informs: cockroachdb#93892 Fixes: cockroachdb#134031 (No release note because automatic collection of partial stats is not yet enabled in any available release.) Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.
Sentry Link: https://cockroach-labs.sentry.io/issues/6035499143/?referrer=webhooks_plugin
Panic Message:
Stacktrace (expand for inline code snippets):
src/runtime/asm_amd64.s#L1694-L1696
pkg/util/stop/stopper.go#L497-L499
pkg/sql/stats/stats_cache.go#L529-L531
pkg/sql/stats/stats_cache.go#L503-L505
pkg/sql/stats/stats_cache.go#L501-L503
pkg/sql/stats/stats_cache.go#L854-L856
pkg/sql/stats/forecast.go#L147-L149
pkg/sql/stats/forecast.go#L380-L382
pkg/util/errorutil/error.go#L20-L22
src/runtime/asm_amd64.s#L1694-L1696
pkg/util/stop/stopper.go#L497-L499
pkg/sql/stats/stats_cache.go#L529-L531
pkg/sql/stats/stats_cache.go#L503-L505
pkg/sql/stats/stats_cache.go#L501-L503
pkg/sql/stats/stats_cache.go#L854-L856
pkg/sql/stats/forecast.go#L147-L149
pkg/sql/stats/forecast.go#L380-L382
pkg/util/errorutil/error.go#L19-L21
Tags
Jira issue: CRDB-43884
The text was updated successfully, but these errors were encountered: