Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

colexec: internal error caused by crdb_internal.range_stats #93806

Closed
knz opened this issue Dec 16, 2022 · 0 comments · Fixed by #93868
Closed

colexec: internal error caused by crdb_internal.range_stats #93806

knz opened this issue Dec 16, 2022 · 0 comments · Fixed by #93868
Labels
A-sql-execution Relating to SQL execution. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-queries SQL Queries Team

Comments

@knz
Copy link
Contributor

knz commented Dec 16, 2022

In #93644 I am modifying SHOW RANGES to execute crdb_internal.range_stats outside of the vtable generator for crdb_internal.ranges.

When I do this on top of master, I get non-deterministic crashes with the stack trace below.
The crash happens mostly when the operation runs soon after a test starts. This suggests a race with some initialization.
In most cases the execution is fine.

If I manually modify the FunctionProperties of crdb_internal.range_stats to set DistsqlBlocklist: true, the issue disappears. (edit: no, it doesn't. but it makes the problem appear less often.)

This can be reproduced as follows:

  1. download the branch at https://github.com/knz/cockroach/tree/20221216-distsql-bug
  2. run dev testlogic base --files='show_ranges' --config=local in a loop
split_at:21: SELECT start_key, end_key, replicas, lease_holder FROM [SHOW RANGES FROM TABLE t WITH DETAILS]
        expected success, but found
        (XX000) internal error: unexpected error from the vectorized engine: runtime error: index out of range [1] with length 1
        error.go:89: in func1()
        DETAIL: stack trace:
        github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:89: func1()
        GOROOT/src/runtime/panic.go:884: gopanic()
        GOROOT/src/runtime/panic.go:113: goPanicIndex()
        github.com/cockroachdb/cockroach/pkg/col/coldata/bytes.go:194: Get()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:67: appendKey()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:116: func1()
        github.com/cockroachdb/cockroach/pkg/sql/colmem/allocator.go:441: PerformOperation()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:91: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecprojconst/proj_const_right_ops.eg.go:22383: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecbase/cast.eg.go:10699: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/range_stats.go:79: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecprojconst/proj_const_right_ops.eg.go:22383: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecbase/cast.eg.go:10699: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecproj/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecproj/proj_non_const_ops.eg.go:4415: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:204: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/buffer.go:48: advance()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/case.go:195: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecproj/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecproj/proj_non_const_ops.eg.go:4415: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:204: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/buffer.go:48: advance()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/case.go:195: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:137: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecproj/bazel-out/k8-fastbuild/bin/pkg/sql/colexec/colexecproj/proj_non_const_ops.eg.go:4415: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124: Next()
        github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93: Next()

cc @yuzefovich for triage.

Jira issue: CRDB-22535
Epic: CRDB-22701

@knz knz added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-sql-execution Relating to SQL execution. labels Dec 16, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Dec 16, 2022
@craig craig bot closed this as completed in 94e03c0 Dec 18, 2022
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-sql-execution Relating to SQL execution. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-queries SQL Queries Team
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant