Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test: TestDataDriven failed #123951

Closed
cockroach-teamcity opened this issue May 10, 2024 · 3 comments · Fixed by #124941
Assignees
Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 10, 2024

pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven failed with artifacts on release-24.1 @ a919026d180c5f7b6299ffa413511ab6e3f5d477:

        	            	  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1089
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1052
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1027
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1837
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1751
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3902
        	            	  | github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster.(*Tenant).LookupTableDescriptorByID
        	            	  | 	github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster/tenant_state.go:197
        	            	  | pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven.func2.1
        	            	  | 	pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:126
        	            	  | github.com/cockroachdb/datadriven.RunTest.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:118
        	            	  | github.com/cockroachdb/datadriven.runDirective.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337
        	            	  | github.com/cockroachdb/datadriven.runDirective
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:342
        	            	  | github.com/cockroachdb/datadriven.runDirectiveOrSubTest
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:211
        	            	  | github.com/cockroachdb/datadriven.runTestInternal
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:184
        	            	  | github.com/cockroachdb/datadriven.RunTestAny
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144
        	            	  | github.com/cockroachdb/datadriven.RunTest
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117
        	            	  | pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven.func2
        	            	  | 	pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:90
        	            	  | github.com/cockroachdb/datadriven.Walk.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425
        	            	  | github.com/cockroachdb/datadriven.WalkAny
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436
        	            	  | github.com/cockroachdb/datadriven.WalkAny.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449
        	            	  | github.com/cockroachdb/datadriven.subTest.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:27
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1222
        	            	Wraps: (3) relation "[107]" does not exist
        	            	Error types: (1) *pgerror.withCandidateCode (2) *withstack.withStack (3) *errutil.leafError
        	Test:       	TestDataDriven/tables
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/88cfb1c77b9d258ed6991015ff838b5f/logTestDataDriven_tables4011239150
    --- FAIL: TestDataDriven/tables (1.99s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-38623

@cockroach-teamcity cockroach-teamcity added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels May 10, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 10, 2024
@rafiss rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label May 10, 2024
@rafiss
Copy link
Collaborator

rafiss commented May 14, 2024

    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9990/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test_/spanconfigsplitterccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/spanconfigccl/spanconfigsplitterccl/testdata/tables:34:
        exec-sql [0 args]
        DROP TABLE db.t2;
        ----
    tenant_state.go:197: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster/tenant_state.go:197
        	            				pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:126
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:118
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:342
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:211
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:184
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117
        	            				pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:90
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449
        	            				github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:27
        	Error:      	Received unexpected error:
        	            	relation "[107]" does not exist
        	            	(1) candidate pg code: 42P01
        	            	Wraps: (2) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/sqlerrors.NewUndefinedRelationError
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/sqlerrors/errors.go:175
        	            	  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descs.ByIDGetter.Table
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descs/getters.go:93
        	            	  | github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster.(*Tenant).LookupTableDescriptorByID.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster/tenant_state.go:201
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn.func1
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3903
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn.(*InternalDB).DescsTxn.func2
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1753
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1850
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/txn.go:1049
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1089
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1052
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1027
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1837
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1751
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3902
        	            	  | github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster.(*Tenant).LookupTableDescriptorByID
        	            	  | 	github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster/tenant_state.go:197
        	            	  | pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven.func2.1
        	            	  | 	pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:126
        	            	  | github.com/cockroachdb/datadriven.RunTest.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:118
        	            	  | github.com/cockroachdb/datadriven.runDirective.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337
        	            	  | github.com/cockroachdb/datadriven.runDirective
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:342
        	            	  | github.com/cockroachdb/datadriven.runDirectiveOrSubTest
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:211
        	            	  | github.com/cockroachdb/datadriven.runTestInternal
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:184
        	            	  | github.com/cockroachdb/datadriven.RunTestAny
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144
        	            	  | github.com/cockroachdb/datadriven.RunTest
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117
        	            	  | pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven.func2
        	            	  | 	pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:90
        	            	  | github.com/cockroachdb/datadriven.Walk.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425
        	            	  | github.com/cockroachdb/datadriven.WalkAny
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436
        	            	  | github.com/cockroachdb/datadriven.WalkAny.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449
        	            	  | github.com/cockroachdb/datadriven.subTest.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:27
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1222
        	            	Wraps: (3) relation "[107]" does not exist
        	            	Error types: (1) *pgerror.withCandidateCode (2) *withstack.withStack (3) *errutil.leafError
        	Test:       	TestDataDriven/tables

@rafiss
Copy link
Collaborator

rafiss commented May 14, 2024

CRDB logs

I240510 15:46:24.020602 26434 sql/gcjob/gc_job_utils.go:278 ⋮ [T10,Vcluster-10,nsql1,job=‹SCHEMA CHANGE GC id=967591801356550145›] 221  updated progress status: ‹deleting data›, payload: ‹tables:<id:107 > ranges_unsplit_done:true ›
I240510 15:46:24.022147 26434 sql/gcjob/table_garbage_collection.go:208 ⋮ [T10,Vcluster-10,nsql1,job=‹SCHEMA CHANGE GC id=967591801356550145›] 222  deleting data for table 107
I240510 15:46:24.053593 26434 sql/gcjob/gc_job_utils.go:278 ⋮ [T10,Vcluster-10,nsql1,job=‹SCHEMA CHANGE GC id=967591801356550145›] 223  updated progress status: ‹waiting for MVCC GC›, payload: ‹tables:<id:107 status:WAITING_FOR_MVCC_GC > ranges_unsplit_done:true ›
I240510 15:46:24.056160 26434 sql/gcjob/gc_job.go:446 ⋮ [T10,Vcluster-10,nsql1,job=‹SCHEMA CHANGE GC id=967591801356550145›] 224  not waiting for MVCC GC in /Tenant/10/Table/107 due to testing knob
I240510 15:46:24.056226 26434 sql/schema_changer.go:3148 ⋮ [T10,Vcluster-10,nsql1,job=‹SCHEMA CHANGE GC id=967591801356550145›] 225  removing table descriptor and zone config for table 107

It looks like we set this testing knob, which could be related:

// Instead of relying on the GC job to wait out TTLs and clear out descriptors,
// let's simply exclude dropped tables to simulate descriptors no longer existing.
// See comment on ExcludeDroppedDescriptorsFromLookup for more details.
ExcludeDroppedDescriptorsFromLookup: true,
// We run the reconciler manually in this test (through the span config
// test cluster).
ManagerDisableJobCreation: true,

I think we might need to change how the test controls schema change GC.

@rafiss rafiss self-assigned this May 14, 2024
@rafiss rafiss added the P-2 Issues/test failures with a fix SLA of 3 months label May 14, 2024
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven failed with artifacts on release-24.1 @ 556680db129ea635342767fdd916628409071ef1:

        	            	  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1089
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1052
        	            	  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/kv/db.go:1027
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1837
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/internal.go:1751
        	            	  | github.com/cockroachdb/cockroach/pkg/sql.DescsTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3902
        	            	  | github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster.(*Tenant).LookupTableDescriptorByID
        	            	  | 	github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster/tenant_state.go:197
        	            	  | pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven.func2.1
        	            	  | 	pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:126
        	            	  | github.com/cockroachdb/datadriven.RunTest.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:118
        	            	  | github.com/cockroachdb/datadriven.runDirective.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:337
        	            	  | github.com/cockroachdb/datadriven.runDirective
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:342
        	            	  | github.com/cockroachdb/datadriven.runDirectiveOrSubTest
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:211
        	            	  | github.com/cockroachdb/datadriven.runTestInternal
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:184
        	            	  | github.com/cockroachdb/datadriven.RunTestAny
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:144
        	            	  | github.com/cockroachdb/datadriven.RunTest
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:117
        	            	  | pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test.TestDataDriven.func2
        	            	  | 	pkg/ccl/spanconfigccl/spanconfigsplitterccl/spanconfigsplitterccl_test/pkg/ccl/spanconfigccl/spanconfigsplitterccl/datadriven_test.go:90
        	            	  | github.com/cockroachdb/datadriven.Walk.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:425
        	            	  | github.com/cockroachdb/datadriven.WalkAny
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:436
        	            	  | github.com/cockroachdb/datadriven.WalkAny.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:449
        	            	  | github.com/cockroachdb/datadriven.subTest.func1
        	            	  | 	github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/test_interface.go:27
        	            	  | testing.tRunner
        	            	  | 	GOROOT/src/testing/testing.go:1689
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1222
        	            	Wraps: (3) relation "[107]" does not exist
        	            	Error types: (1) *pgerror.withCandidateCode (2) *withstack.withStack (3) *errutil.leafError
        	Test:       	TestDataDriven/tables
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/88cfb1c77b9d258ed6991015ff838b5f/logTestDataDriven_tables2889382514
    --- FAIL: TestDataDriven/tables (2.04s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@exalate-issue-sync exalate-issue-sync bot removed the P-2 Issues/test failures with a fix SLA of 3 months label May 25, 2024
@exalate-issue-sync exalate-issue-sync bot assigned rafiss and unassigned rafiss May 25, 2024
@exalate-issue-sync exalate-issue-sync bot added the P-2 Issues/test failures with a fix SLA of 3 months label May 25, 2024
@craig craig bot closed this as completed in d77b6e7 Jun 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants