Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/kvccl/kvtenantccl: TestTenantTracesAreRedacted failed #101010

Closed
cockroach-teamcity opened this issue Apr 9, 2023 · 3 comments · Fixed by #101097 or #101148
Closed

ccl/kvccl/kvtenantccl: TestTenantTracesAreRedacted failed #101010

cockroach-teamcity opened this issue Apr 9, 2023 · 3 comments · Fixed by #101097 or #101148
Labels
A-admission-control A-observability-inf A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot.
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 9, 2023 •
edited by cockroach-jira-scripts
Loading

ccl/kvccl/kvtenantccl.TestTenantTracesAreRedacted failed with artifacts on master @ 85e41ca8d3d9edacf5ee3061a2591b159b9b0502:

      github.com/cockroachdb/cockroach/pkg/server/testserver.go:619 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:352 +0x1e1
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:347 +0xc4
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.testTenantTracesAreRedactedImpl()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:79 +0x435
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:38 +0x35
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse.func1()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23 +0x55
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 33903 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider()
      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:98 +0x329
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1799 +0x3cfb
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:619 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:352 +0x1e1
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:347 +0xc4
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.testTenantTracesAreRedactedImpl()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:79 +0x435
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:38 +0x35
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse.func1()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23 +0x55
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 33572 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:22 +0x84
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:37 +0x67
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/multi-tenant

This test on roachdash | Improve this report!

Jira issue: CRDB-26746
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Apr 9, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Apr 9, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/kvccl/kvtenantccl.TestTenantTracesAreRedacted failed with artifacts on master @ 85e41ca8d3d9edacf5ee3061a2591b159b9b0502:

      github.com/cockroachdb/cockroach/pkg/server/testserver.go:619 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:352 +0x1e1
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:347 +0xc4
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.testTenantTracesAreRedactedImpl()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:79 +0x435
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:38 +0x35
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse.func1()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23 +0x55
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 29941 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider()
      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:98 +0x329
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1799 +0x3cfb
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:619 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:352 +0x1e1
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:347 +0xc4
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.testTenantTracesAreRedactedImpl()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:79 +0x435
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:38 +0x35
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse.func1()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23 +0x55
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 29611 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:22 +0x84
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:37 +0x67
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================

Parameters: TAGS=bazel,gss,race

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@knz knz added A-storage Relating to our storage engine (Pebble) on-disk storage. A-observability-inf labels Apr 10, 2023
@knz
Copy link
Contributor

knz commented Apr 10, 2023

This is a legit pebble metric data race.

==================
WARNING: DATA RACE
Read at 0x00c006843438 by goroutine 29941:
  github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).empty()
      github.com/cockroachdb/cockroach/pkg/server/node.go:898 +0x6c
  github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).tryPopulateAdmissionDiskStats()
      github.com/cockroachdb/cockroach/pkg/server/node.go:871 +0xa7
  github.com/cockroachdb/cockroach/pkg/server.(*Node).GetPebbleMetrics()
      github.com/cockroachdb/cockroach/pkg/server/node.go:929 +0x104
  github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
      github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:107 +0x1e7

Previous write at 0x00c006843438 by goroutine 29611:
  github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).initDiskStatsMap()
      github.com/cockroachdb/cockroach/pkg/server/node.go:902 +0x90
  github.com/cockroachdb/cockroach/pkg/server.(*Node).registerEnginesForDiskStatsMap()
      github.com/cockroachdb/cockroach/pkg/server/node.go:922 +0x5d24
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1942 +0x5c65
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1434 +0x20e
  github.com/cockroachdb/cockroach/pkg/server.(*Server).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:1897 +0x4ec4
  github.com/cockroachdb/cockroach/pkg/server.(*TestServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:619 +0x8f
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:352 +0x1e1
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:347 +0xc4
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.testTenantTracesAreRedactedImpl()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:79 +0x435
  github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test.TestTenantTracesAreRedacted.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvtenantccl_test/pkg/ccl/kvccl/kvtenantccl/tenant_trace_test.go:38 +0x35
  github.com/cockroachdb/cockroach/pkg/testutils.RunTrueAndFalse.func1()
      github.com/cockroachdb/cockroach/pkg/testutils/subtest.go:23 +0x55
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

cc @jbowens @abarganier for triage.

@knz knz added GA-blocker branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 labels Apr 10, 2023
@jbowens
Copy link
Collaborator

jbowens commented Apr 10, 2023

This is a duplicate of #91414. @cockroachdb/admission-control

@jbowens jbowens removed the T-storage Storage Team label Apr 10, 2023
@knz knz mentioned this issue Apr 10, 2023
Merged
@craig craig bot closed this as completed in #101097 Apr 10, 2023
@craig craig bot closed this as completed in a73b33f Apr 10, 2023
@blathers-crl blathers-crl bot mentioned this issue Apr 10, 2023
Merged
@irfansharif irfansharif mentioned this issue Apr 10, 2023
Closed
irfansharif added a commit to irfansharif/cockroach that referenced this issue Apr 10, 2023
@irfansharif
server: fix a race condition during server initialization
bde558a 
Fixes cockroachdb#91414.
Fixes cockroachdb#101010.
Fixes cockroachdb#100902.

The call to registerEnginesForDiskStatsMap needs to wait until the store
IDs are known.

Release note: None
@irfansharif irfansharif mentioned this issue Apr 10, 2023
Merged
irfansharif added a commit to irfansharif/cockroach that referenced this issue Apr 10, 2023
@irfansharif
server: (re-)fix data race in server initialization
2ef89c9 
Fixes cockroachdb#91414.
Fixes  cockroachdb#101010.
Fixes cockroachdb#100902.

There was a race between registerEnginesForDiskStatsMap and
SetPebbleMetricsProvider, the latter making use of a map constructed by
the former but appeared in the opposite order in code.

Release note: None
craig bot pushed a commit that referenced this issue Apr 10, 2023
@irfansharif
Merge #101148
a5d61d2 
101148: server: (re-)fix data race in server initialization r=irfansharif a=irfansharif

Fixes #91414.
Fixes  #101010.
Fixes #100902.

There was a race between registerEnginesForDiskStatsMap and SetPebbleMetricsProvider, the latter making use of a map constructed by the former but appeared in the opposite order in code.

Release note: None

Co-authored-by: irfan sharif <[email protected]>
@blathers-crl blathers-crl bot mentioned this issue Apr 10, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Apr 10, 2023
@irfansharif
server: (re-)fix data race in server initialization
83ffd45 
Fixes #91414.
Fixes  #101010.
Fixes #100902.

There was a race between registerEnginesForDiskStatsMap and
SetPebbleMetricsProvider, the latter making use of a map constructed by
the former but appeared in the opposite order in code.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Apr 10, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Apr 10, 2023
@irfansharif
server: (re-)fix data race in server initialization
473bd86 
Fixes #91414.
Fixes  #101010.
Fixes #100902.

There was a race between registerEnginesForDiskStatsMap and
SetPebbleMetricsProvider, the latter making use of a map constructed by
the former but appeared in the opposite order in code.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-admission-control A-observability-inf A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-test-failure Broken test (automatically or manually discovered). GA-blocker O-robot Originated from a bot.
Projects
None yet
Milestone
23.1
3 participants
@knz @jbowens @cockroach-teamcity