Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/ccl/testccl/sqlccl/sqlccl_test: TestTenantTempTableCleanup failed #111028

Closed
cockroach-teamcity opened this issue Sep 21, 2023 · 2 comments · Fixed by #111054
Closed

pkg/ccl/testccl/sqlccl/sqlccl_test: TestTenantTempTableCleanup failed #111028

cockroach-teamcity opened this issue Sep 21, 2023 · 2 comments · Fixed by #111054
Assignees
@knz
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). db-cy-23 O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-multitenant Issues owned by the multi-tenant virtual team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 21, 2023 •
edited by knz
Loading

Epic: CRDB-6671

pkg/ccl/testccl/sqlccl/sqlccl_test.TestTenantTempTableCleanup failed with artifacts on master @ 49eef749235f15d57153cb00f48dfd3b27be6c54:

      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:107 +0x9d6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x7c
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:408 +0x589
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x7c
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:408 +0x589
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:285 +0x144
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1464
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.testClusterFactoryImpl.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1968 +0xaf
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*testClusterFactoryImpl).NewTestCluster()
      <autogenerated>:1 +0x36
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.NewCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:300 +0x118
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:284 +0x56
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestTenantTempTableCleanup()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:90 +0x67c
  testing.tRunner()
      GOROOT/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1629 +0x47

Goroutine 57504 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1629 +0x805
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:2036 +0x8d
  testing.tRunner()
      GOROOT/src/testing/testing.go:1576 +0x216
  testing.runTests()
      GOROOT/src/testing/testing.go:2034 +0x87c
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1906 +0xb44
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestMain()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/main_test.go:32 +0x1c9
  main.main()
      main/bazel-out/k8-fastbuild/bin/pkg/ccl/testccl/sqlccl/sqlccl_test_/testmain.go:153 +0x6eb
  runtime.main()
      GOROOT/src/runtime/proc.go:250 +0x206
  github.com/cockroachdb/cockroach/pkg/util/parquet.box2DDecoder.decode()
      github.com/cockroachdb/cockroach/pkg/util/parquet/decoders.go:174 +0x44
  github.com/cockroachdb/cockroach/pkg/util/parquet.init.0()
      github.com/cockroachdb/cockroach/pkg/util/parquet/decoders.go:336 +0x10c
==================

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-31728
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Sep 21, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 21, 2023
@github-project-automation github-project-automation bot moved this to Triage in SQL Queries Sep 21, 2023
@yuzefovich
Copy link
Member 

==================
WARNING: DATA RACE
Read at 0x00c009751400 by goroutine 62401:
  github.com/cockroachdb/cockroach/pkg/settings.(*Values).SpecializedToVirtualCluster()
      github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:177 +0x3c
  github.com/cockroachdb/cockroach/pkg/sql.glob..func549()
      github.com/cockroachdb/cockroach/pkg/sql/vars.go:1622 +0x49
  github.com/cockroachdb/cockroach/pkg/sql.getSessionVarDefaultString()
      github.com/cockroachdb/cockroach/pkg/sql/set_var.go:195 +0xa8
  github.com/cockroachdb/cockroach/pkg/sql.NewInternalSessionData.func1()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:82 +0x204
  github.com/cockroachdb/cockroach/pkg/sql.(*sessionDataMutatorIterator).applyOnEachMutator()
      github.com/cockroachdb/cockroach/pkg/sql/exec_util.go:3036 +0x12f
  github.com/cockroachdb/cockroach/pkg/sql.NewInternalSessionData()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:73 +0x304
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).newInternalExecutorWithTxn()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:1614 +0xc5
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:1750 +0x2c7
  github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
      github.com/cockroachdb/cockroach/pkg/kv/txn.go:1006 +0x99
  github.com/cockroachdb/cockroach/pkg/kv.runTxn()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:1041 +0x6b
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:1004 +0xf0
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
      github.com/cockroachdb/cockroach/pkg/kv/db.go:979 +0x6a
  github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn-fm()
      <autogenerated>:1 +0x29
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:1746 +0x66a
  github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn()
      github.com/cockroachdb/cockroach/pkg/sql/internal.go:1668 +0xf0
  github.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).doTemporaryObjectCleanup()
      github.com/cockroachdb/cockroach/pkg/sql/temporary_schema.go:506 +0x59a
  github.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1()
      github.com/cockroachdb/cockroach/pkg/sql/temporary_schema.go:627 +0x438
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f6

Previous write at 0x00c009751400 by goroutine 57504:
  github.com/cockroachdb/cockroach/pkg/settings.(*Values).SpecializeForVirtualCluster()
      github.com/cockroachdb/cockroach/pkg/settings/pkg/settings/values.go:166 +0xa8
  github.com/cockroachdb/cockroach/pkg/server.makeTenantSQLServerArgs()
      github.com/cockroachdb/cockroach/pkg/server/tenant.go:1025 +0x1d7
  github.com/cockroachdb/cockroach/pkg/server.newTenantServer()
      github.com/cockroachdb/cockroach/pkg/server/tenant.go:298 +0x4d8
  github.com/cockroachdb/cockroach/pkg/server.NewSeparateProcessTenantServer()
      github.com/cockroachdb/cockroach/pkg/server/tenant.go:207 +0x16c
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).StartTenant()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:1619 +0x3137
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartTenant()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:308 +0xe6
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestTenantTempTableCleanup()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:124 +0xc16
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x7c
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:408 +0x589
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x7c
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:408 +0x589
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:285 +0x144
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1464
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.testClusterFactoryImpl.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1968 +0xaf
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*testClusterFactoryImpl).NewTestCluster()
      <autogenerated>:1 +0x36
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.NewCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:300 +0x118
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:284 +0x56
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestTenantTempTableCleanup()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:90 +0x67c
  testing.tRunner()
      GOROOT/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1629 +0x47

Goroutine 62401 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x619
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0x150
  github.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start()
      github.com/cockroachdb/cockroach/pkg/sql/temporary_schema.go:617 +0x53
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1554 +0xb64
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServerWrapper).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/tenant.go:800 +0x3724
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServerWrapper).Start()
      github.com/cockroachdb/cockroach/pkg/server/tenant.go:967 +0x44
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).StartTenant()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:1641 +0x3246
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartTenant()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:308 +0xe6
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestTenantTempTableCleanup()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:107 +0x9d6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x7c
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:408 +0x589
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x7c
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:408 +0x589
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:285 +0x144
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1464
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.testClusterFactoryImpl.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1968 +0xaf
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*testClusterFactoryImpl).NewTestCluster()
      <autogenerated>:1 +0x36
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.NewCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:300 +0x118
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:284 +0x56
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestTenantTempTableCleanup()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/temp_table_clean_test.go:90 +0x67c
  testing.tRunner()
      GOROOT/src/testing/testing.go:1576 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1629 +0x47

Goroutine 57504 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1629 +0x805
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:2036 +0x8d
  testing.tRunner()
      GOROOT/src/testing/testing.go:1576 +0x216
  testing.runTests()
      GOROOT/src/testing/testing.go:2034 +0x87c
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1906 +0xb44
  pkg/ccl/testccl/sqlccl/sqlccl_test_test.TestMain()
      pkg/ccl/testccl/sqlccl/sqlccl_test_test/pkg/ccl/testccl/sqlccl/main_test.go:32 +0x1c9
  main.main()
      main/bazel-out/k8-fastbuild/bin/pkg/ccl/testccl/sqlccl/sqlccl_test_/testmain.go:153 +0x6eb
  runtime.main()
      GOROOT/src/runtime/proc.go:250 +0x206
  github.com/cockroachdb/cockroach/pkg/util/parquet.box2DDecoder.decode()
      github.com/cockroachdb/cockroach/pkg/util/parquet/decoders.go:174 +0x44
  github.com/cockroachdb/cockroach/pkg/util/parquet.init.0()
      github.com/cockroachdb/cockroach/pkg/util/parquet/decoders.go:336 +0x10c
==================
I230921 09:26:13.650879 1 (gostd) testmain.go:108  [-] 1  Test //pkg/ccl/testccl/sqlccl:sqlccl_test exited with error code 66

cc @knz

@yuzefovich yuzefovich removed the T-sql-queries SQL Queries Team label Sep 21, 2023
@yuzefovich yuzefovich removed this from SQL Queries Sep 21, 2023
@yuzefovich yuzefovich added the T-multitenant Issues owned by the multi-tenant virtual team label Sep 21, 2023
@knz
Copy link
Contributor

knz commented Sep 21, 2023

Thanks. This almost by definition is a bug in the test code. I'll look after meeting.

@knz knz self-assigned this Sep 21, 2023
@knz knz mentioned this issue Sep 21, 2023
Merged
@craig craig bot closed this as completed in 6a2097e Sep 22, 2023
@knz knz added the db-cy-23 label Oct 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@knz knz

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). db-cy-23 O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-multitenant Issues owned by the multi-tenant virtual team
Projects
None yet
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

sqlccl: fix a data race in TestTenantTempTableCleanup knz/cockroach
3 participants
@knz @yuzefovich @cockroach-teamcity