ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100905

cockroach-teamcity · 2023-04-07T07:37:48Z

ccl/multiregionccl.TestEnsureLocalReadsOnGlobalTables failed with artifacts on release-23.1 @ 571a1744c72415894dc0e742bd89e74938ea71f4:

      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 358517 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider()
      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:98 +0x329
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1795 +0x3c9b
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:614 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).startServer()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:583 +0xd4
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:389 +0x850
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:233 +0xa8
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:232 +0x8b
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionClusterWithRegionList()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:128 +0x1f6
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionCluster()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:79 +0x1ef
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestEnsureLocalReadsOnGlobalTables()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/roundtrips_test.go:74 +0x35b
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 357876 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.runTests()
      GOROOT/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1726 +0xa84
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestMain()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/main_test.go:32 +0x1c9
  main.main()
      main/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/testmain.go:165 +0x748
==================

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100902 ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed [C-test-failure O-robot T-sql-schema branch-master]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!

Jira issue: CRDB-26684}

The text was updated successfully, but these errors were encountered:

rafiss · 2023-04-07T17:10:42Z

Race condition in the diskStatsMap. cc @cockroachdb/storage team to triage.

03:37:46     ==================
03:37:46     WARNING: DATA RACE
03:37:46     Read at 0x00c007daf438 by goroutine 358517:
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).empty()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:897 +0x6c
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).tryPopulateAdmissionDiskStats()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:870 +0xa7
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Node).GetPebbleMetrics()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:928 +0x104
03:37:46       github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
03:37:46           github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:107 +0x1e7
03:37:46     
03:37:46     Previous write at 0x00c007daf438 by goroutine 357876:
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).initDiskStatsMap()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:901 +0x90
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Node).registerEnginesForDiskStatsMap()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:921 +0x5ca4
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/server.go:1938 +0x5be5
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1426 +0x247
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/server.go:1893 +0x4e64
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Node).writeNodeStatus()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:1052 +0xf7
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/node.go:1012 +0xae
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/server.go:1831 +0x479c
03:37:46       github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
03:37:46           github.com/cockroachdb/cockroach/pkg/server/testserver.go:614 +0x8f
03:37:46       github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).startServer()
03:37:46           github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:583 +0xd4
03:37:46       github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
03:37:46           github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:389 +0x850
03:37:46       github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
03:37:46           github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:233 +0xa8
03:37:46       github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
03:37:46           github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
03:37:46       github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
03:37:46           github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
03:37:46       github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
03:37:46           github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:232 +0x8b
03:37:46       github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionClusterWithRegionList()
03:37:46           github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:128 +0x1f6
03:37:46       github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionCluster()
03:37:46           github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:79 +0x1ef
03:37:46       github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestEnsureLocalReadsOnGlobalTables()
03:37:46           github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/roundtrips_test.go:74 +0x35b
03:37:46       testing.tRunner()
03:37:46           GOROOT/src/testing/testing.go:1446 +0x216
03:37:46       testing.(*T).Run.func1()
03:37:46           GOROOT/src/testing/testing.go:1493 +0x47

nicktrav · 2023-04-07T17:27:47Z

This looks like #91414.

jbowens · 2023-04-07T20:10:57Z

@cockroachdb/admission-control

cockroach-teamcity · 2023-04-09T07:09:54Z

ccl/multiregionccl.TestEnsureLocalReadsOnGlobalTables failed with artifacts on release-23.1 @ de239a7438f44d382c9aefceb65d9c39911dabd2:

      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 363427 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider()
      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:98 +0x329
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1795 +0x3c9b
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:614 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).startServer()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:583 +0xd4
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:389 +0x850
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:233 +0xa8
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:232 +0x8b
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionClusterWithRegionList()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:128 +0x1f6
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionCluster()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:79 +0x1ef
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestEnsureLocalReadsOnGlobalTables()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/roundtrips_test.go:74 +0x35b
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 362787 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.runTests()
      GOROOT/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1726 +0xa84
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestMain()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/main_test.go:32 +0x1c9
  main.main()
      main/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/testmain.go:165 +0x748
==================

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100902 ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed [C-test-failure O-robot T-storage branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-04-10T07:09:46Z

ccl/multiregionccl.TestEnsureLocalReadsOnGlobalTables failed with artifacts on release-23.1 @ 2f96695f75b07c872ec5f146acc1fa198135768f:

      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 364898 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider()
      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:98 +0x329
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1795 +0x3c9b
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:614 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).startServer()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:583 +0xd4
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:389 +0x850
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:233 +0xa8
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:329 +0xe10
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:232 +0x8b
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionClusterWithRegionList()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:128 +0x1f6
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils.TestingCreateMultiRegionCluster()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/multiregionccltestutils/testutils.go:79 +0x1ef
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestEnsureLocalReadsOnGlobalTables()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/roundtrips_test.go:74 +0x35b
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 364257 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      GOROOT/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.runTests()
      GOROOT/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      GOROOT/src/testing/testing.go:1726 +0xa84
  github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestMain()
      github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/main_test.go:32 +0x1c9
  main.main()
      main/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/testmain.go:165 +0x748
==================

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100902 ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed [C-test-failure O-robot T-storage branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 7, 2023

cockroach-teamcity added this to the 23.1 milestone Apr 7, 2023

blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Apr 7, 2023

rafiss removed the T-sql-schema-deprecated Use T-sql-foundations instead label Apr 7, 2023

blathers-crl bot added the T-storage Storage Team label Apr 7, 2023

irfansharif closed this as completed Apr 10, 2023

jbowens added this to [Deprecated] Storage Jun 4, 2024

jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100905

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100905

cockroach-teamcity commented Apr 7, 2023 •

edited by cockroach-jira-scripts

Loading

rafiss commented Apr 7, 2023

nicktrav commented Apr 7, 2023

jbowens commented Apr 7, 2023

cockroach-teamcity commented Apr 9, 2023

cockroach-teamcity commented Apr 10, 2023

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100905

ccl/multiregionccl: TestEnsureLocalReadsOnGlobalTables failed #100905

Comments

cockroach-teamcity commented Apr 7, 2023 • edited by cockroach-jira-scripts Loading

rafiss commented Apr 7, 2023

nicktrav commented Apr 7, 2023

jbowens commented Apr 7, 2023

cockroach-teamcity commented Apr 9, 2023

cockroach-teamcity commented Apr 10, 2023

cockroach-teamcity commented Apr 7, 2023 •

edited by cockroach-jira-scripts

Loading