Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvprober: TestProberDoesReadsAndWrites failed #109564

Closed
cockroach-teamcity opened this issue Aug 27, 2023 · 8 comments · Fixed by #110191
Closed

kv/kvprober: TestProberDoesReadsAndWrites failed #109564

cockroach-teamcity opened this issue Aug 27, 2023 · 8 comments · Fixed by #110191
Assignees
@joshimhoff
Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-kv KV Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 27, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvprober.TestProberDoesReadsAndWrites failed with artifacts on master @ 671500be0c22d14d3cb0089313cb49c736631bac:

      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:170 +0x4f2
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 11038 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x619
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:314 +0x338
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:345 +0x1c9
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2153 +0x6999
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:662 +0xe8
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:738 +0xe9
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:200 +0x132
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:193 +0xd6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnly()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:215 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:233 +0x7b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.initTestServer()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:379 +0xc7
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 9113 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:135 +0x378
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================
=== RUN   TestProberDoesReadsAndWrites/disabled
=== RUN   TestProberDoesReadsAndWrites/happy_path

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-30991
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team labels Aug 27, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Aug 27, 2023
@cockroach-teamcity
Copy link
Member Author

kv/kvprober.TestProberDoesReadsAndWrites failed with artifacts on master @ a900aa218054812b782c0b3d130b25296c0d14e3:

  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbe-fm()
      <autogenerated>:1 +0x6d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:339 +0x3a6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f6

Goroutine 11786 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:135 +0x378
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 13740 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x619
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:314 +0x338
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:345 +0x1c9
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2153 +0x6999
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:662 +0xe8
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:738 +0xe9
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:200 +0x132
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:193 +0xd6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnly()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:215 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:233 +0x7b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.initTestServer()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:379 +0xc7
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================
=== RUN   TestProberDoesReadsAndWrites/disabled
    test_server_shim.go:131: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestProberDoesReadsAndWrites/happy_path

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvprober.TestProberDoesReadsAndWrites failed with artifacts on master @ 85cbfffeaa60c3d40e51eb6be7e49eca4dcc8a18:

  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbe-fm()
      <autogenerated>:1 +0x6d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:339 +0x3a6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f6

Goroutine 11407 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:135 +0x378
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 13335 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x619
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:314 +0x338
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:345 +0x1c9
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2153 +0x6999
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:662 +0xe8
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:738 +0xe9
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:200 +0x132
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:193 +0xd6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnly()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:215 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:233 +0x7b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.initTestServer()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:379 +0xc7
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================
=== RUN   TestProberDoesReadsAndWrites/disabled
    test_server_shim.go:131: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestProberDoesReadsAndWrites/happy_path

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@nvanbenschoten
Copy link
Member

We see a data race between two concurrent readProbes, one performed by the test through a manual call to Prober.ReadProbe and another performed by the background kvprober loop. I don't see why this wouldn't be allowed by the test, and yet it went from never failing to suddenly failing regularly 2 days ago. @joshimhoff do you have any ideas about why this might have started failing regularly?

==================
WARNING: DATA RACE
Write at 0x00c004e4bd10 by goroutine 11407:
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*meta2Planner).next()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/planner.go:168 +0x444
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbeImpl()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:371 +0x182
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbe()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:361 +0x5fd
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).ReadProbe()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/helpers_test.go:34 +0x50b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:170 +0x4f2
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Previous read at 0x00c004e4bd10 by goroutine 13335:
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*meta2Planner).next()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/planner.go:164 +0x36b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbeImpl()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:371 +0x182
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbe()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:361 +0xe4
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbe-fm()
      <autogenerated>:1 +0x6d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:339 +0x3a6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f6

@joshimhoff
Copy link
Collaborator

joshimhoff commented Aug 29, 2023 •
edited
Loading

I don't immediately know why this started firing two days ago, but I think this is a true data race, since the planner is not thread-safe by design.

// planner abstracts deciding on ranges to probe. It is public to integration
// test from kvprober_test.
//
// planner is NOT thread-safe.
type planner interface {
	// Next returns a Step for the prober to execute on. A Step is a decision on
	// on what range to probe next. Executing on a series of Steps returned by
	// repeated calls to Next should lead to an even distribution of probes over
	// ranges, at least in the limit.
	//
	// Errors may be temporary or persistent; retrying is an acceptable response;
	// callers should take care to backoff when retrying as always.
	next(ctx context.Context) (Step, error)
}

Should be a test-only bug, as in prod the manual call to kvprober.ReadProbe won't happen.

I am primary SRE oncall this week, so I can't fix this immediately. I can fix it best-effort this week and def next week.

@nvanbenschoten
Copy link
Member

I don't understand why, but a bisect is pointing at 41a0097 as the source of the new flakiness.

I am primary SRE oncall this week, so I can't fix this immediately. I can fix it best-effort this week and def next week.

Thanks! I'll skip this test in the meantime.

@cockroach-teamcity
Copy link
Member Author

kv/kvprober.TestProberDoesReadsAndWrites failed with artifacts on master @ 36bd8296bad475a8183e474bf4aeb08bb83f8a2d:

  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).readProbe-fm()
      <autogenerated>:1 +0x6d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:339 +0x3a6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x1f6

Goroutine 10975 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:182 +0x40c
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 12886 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x619
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:314 +0x338
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:345 +0x1c9
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2153 +0x6999
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:662 +0xe8
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:738 +0xe9
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0x132
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:194 +0xd6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnly()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:216 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:234 +0x7b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.initTestServer()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:379 +0xc7
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func5()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:186 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================
=== RUN   TestProberDoesReadsAndWrites/disabled
    test_server_shim.go:132: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestProberDoesReadsAndWrites/happy_path

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvprober.TestProberDoesReadsAndWrites failed with artifacts on master @ 6f589302865e24a3843a31cd7686514b5430f626:

      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 14941 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x619
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:314 +0x338
  github.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober/kvprober.go:345 +0x1c9
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2153 +0x6999
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:662 +0xe8
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:738 +0xe9
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:201 +0x132
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnlyE()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:194 +0xd6
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServerOnly()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:216 +0x92
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartServer()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_server_shim.go:234 +0x7b
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.initTestServer()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:379 +0xc7
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites.func4()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:139 +0x224
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47

Goroutine 13019 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1493 +0x75d
  github.com/cockroachdb/cockroach/pkg/kv/kvprober_test.TestProberDoesReadsAndWrites()
      github.com/cockroachdb/cockroach/pkg/kv/kvprober_test/pkg/kv/kvprober/kvprober_integration_test.go:135 +0x378
  testing.tRunner()
      GOROOT/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1493 +0x47
==================
=== RUN   TestProberDoesReadsAndWrites/disabled
    test_server_shim.go:132: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestProberDoesReadsAndWrites/happy_path
    test_server_shim.go:132: automatically injected virtual cluster under test; see comment at top of test_server_shim.go for details.

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Aug 31, 2023
@nvanbenschoten
kv: skip TestProberDoesReadsAndWrites
75b99bf 
Informs cockroachdb#109564.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Aug 31, 2023
Merged
craig bot pushed a commit that referenced this issue Aug 31, 2023
@nvanbenschoten
Merge #109825
c86e1a1 
109825: kv: skip TestProberDoesReadsAndWrites r=nvanbenschoten a=nvanbenschoten

Informs #109564.

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
@kvoli kvoli added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Sep 1, 2023
@joshimhoff joshimhoff mentioned this issue Sep 7, 2023
Merged
@joshimhoff
Copy link
Collaborator

#110191

craig bot pushed a commit that referenced this issue Sep 8, 2023
@joshimhoff @yuzefovich
Merge #110191 #110275
dada1a4 
110191: kvprober: make the planner thread-safe to fix test-only race r=nvanbenschoten a=joshimhoff

#109564

**kvprober: make the planner thread-safe to fix test-only race**

This commit makes the planner thread-safe, in order to fix a test-only data race. This commit also un-skips the test that was skipped, when this bug was discovered.

Fixes: #109564
Release note: None.

110275: tenantcostclient: relax expectation in RU estimate test r=yuzefovich a=yuzefovich

In #106769 we tightened the RU estimate test to use 0.05 allowance, but we just saw a case where the difference was on the order of 0.2, so this commit bumps the allowed delta to 0.25 (which is still much better than 0.75 before #106769).

Fixes: #109732.

Release note: None

Co-authored-by: Josh Imhoff <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in 8bf46dc Sep 8, 2023
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joshimhoff joshimhoff

Labels
A-testing Testing tools and infrastructure branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-kv KV Team
Projects
No open projects
KV
Archived in project
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

kvprober: make the planner thread-safe to fix test-only race joshimhoff/cockroach
4 participants
@joshimhoff @nvanbenschoten @cockroach-teamcity @kvoli