Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/roachtest: TestVMPreemptionPolling failed #135267

Closed
cockroach-teamcity opened this issue Nov 15, 2024 · 4 comments · Fixed by #135312
Closed

cmd/roachtest: TestVMPreemptionPolling failed #135267

cockroach-teamcity opened this issue Nov 15, 2024 · 4 comments · Fixed by #135312
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-testeng TestEng Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 15, 2024 •
edited by cockroach-jira-scripts
Loading

cmd/roachtest.TestVMPreemptionPolling failed on master @ 7d943e9c3875319fe2dc35981c8385c16eb7ee27:

=== RUN   TestVMPreemptionPolling
=== RUN   TestVMPreemptionPolling/polling_cancels_test
2024/11/15 12:57:01 test_runner.go:373: test_run_id: test_user-1731675421
2024/11/15 12:57:01 test_runner.go:534: Creating new cluster for test preemption: n0cpu4 (arch="amd64")
2024/11/15 12:57:01 test_runner.go:434: PASS
=== RUN   TestVMPreemptionPolling/polling_doesn't_catch_preemption
==================
WARNING: DATA RACE
Write at 0x0000146c1b28 by goroutine 745:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling.func5()
      pkg/cmd/roachtest/test_test.go:754 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Previous read at 0x0000146c1b28 by goroutine 741:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs.func1()
      pkg/cmd/roachtest/test_runner.go:2148 +0x9b

Goroutine 745 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling()
      pkg/cmd/roachtest/test_test.go:752 +0xb73
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 741 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs()
      pkg/cmd/roachtest/test_runner.go:2143 +0x25a
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.(*testRunner).runTest.func2()
      pkg/cmd/roachtest/test_runner.go:1305 +0x1a4
==================

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-44427
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-testeng TestEng Team labels Nov 15, 2024
@DarrylWong DarrylWong removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 15, 2024
DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 15, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
3b0e46f 
This changes monitorForPreemptedVMs to read pollPreemptionInterval
just once, so when unit tests change the value we don't run into
concurrent access.

Fixes: cockroachdb#135267
Epic: none
Release note: none
@DarrylWong DarrylWong mentioned this issue Nov 15, 2024
Merged
@cockroach-teamcity
Copy link
Member Author

cmd/roachtest.TestVMPreemptionPolling failed on master @ e83bc46aa42f2476b4b11b9703b8038c660dc980:

=== RUN   TestVMPreemptionPolling
=== RUN   TestVMPreemptionPolling/polling_cancels_test
2024/11/16 11:56:00 test_runner.go:373: test_run_id: test_user-1731758160
2024/11/16 11:56:00 test_runner.go:534: Creating new cluster for test preemption: n0cpu4 (arch="amd64")
2024/11/16 11:56:00 test_runner.go:434: PASS
=== RUN   TestVMPreemptionPolling/polling_doesn't_catch_preemption
==================
WARNING: DATA RACE
Write at 0x0000146e6b88 by goroutine 766:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling.func5()
      pkg/cmd/roachtest/test_test.go:754 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Previous read at 0x0000146e6b88 by goroutine 762:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs.func1()
      pkg/cmd/roachtest/test_runner.go:2148 +0x9b

Goroutine 766 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling()
      pkg/cmd/roachtest/test_test.go:752 +0xb73
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 762 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs()
      pkg/cmd/roachtest/test_runner.go:2143 +0x25a
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.(*testRunner).runTest.func2()
      pkg/cmd/roachtest/test_runner.go:1305 +0x1a4
==================

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

cmd/roachtest.TestVMPreemptionPolling failed on master @ 562232945d7bd199d1d07f067708072b4420f90a:

=== RUN   TestVMPreemptionPolling
=== RUN   TestVMPreemptionPolling/polling_doesn't_catch_preemption
==================
WARNING: DATA RACE
Write at 0x0000146e6b88 by goroutine 759:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling.func5()
      pkg/cmd/roachtest/test_test.go:754 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Previous read at 0x0000146e6b88 by goroutine 755:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs.func1()
      pkg/cmd/roachtest/test_runner.go:2148 +0x9b

Goroutine 759 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling()
      pkg/cmd/roachtest/test_test.go:752 +0xb73
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 755 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs()
      pkg/cmd/roachtest/test_runner.go:2143 +0x25a
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.(*testRunner).runTest.func2()
      pkg/cmd/roachtest/test_runner.go:1305 +0x1a4
==================
=== RUN   TestVMPreemptionPolling/polling_cancels_test
2024/11/17 11:35:44 test_runner.go:373: test_run_id: test_user-1731843344
2024/11/17 11:35:44 test_runner.go:534: Creating new cluster for test preemption: n0cpu4 (arch="amd64")
2024/11/17 11:35:44 test_runner.go:434: PASS

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

cmd/roachtest.TestVMPreemptionPolling failed on master @ 863987fd15e0776e15e006b41b9ac8b1e7000d16:

=== RUN   TestVMPreemptionPolling
=== RUN   TestVMPreemptionPolling/polling_cancels_test
2024/11/18 11:42:47 test_runner.go:373: test_run_id: test_user-1731930167
2024/11/18 11:42:47 test_runner.go:534: Creating new cluster for test preemption: n0cpu4 (arch="amd64")
2024/11/18 11:42:47 test_runner.go:434: PASS
=== RUN   TestVMPreemptionPolling/polling_doesn't_catch_preemption
==================
WARNING: DATA RACE
Write at 0x0000146e6b68 by goroutine 745:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling.func5()
      pkg/cmd/roachtest/test_test.go:754 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Previous read at 0x0000146e6b68 by goroutine 741:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs.func1()
      pkg/cmd/roachtest/test_runner.go:2148 +0x9b

Goroutine 745 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling()
      pkg/cmd/roachtest/test_test.go:752 +0xb73
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 741 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs()
      pkg/cmd/roachtest/test_runner.go:2143 +0x25a
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.(*testRunner).runTest.func2()
      pkg/cmd/roachtest/test_runner.go:1305 +0x1a4
==================

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 18, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
f3dcca2 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 18, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
befb39d 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
@cockroach-teamcity
Copy link
Member Author

cmd/roachtest.TestVMPreemptionPolling failed on master @ d2fa9e6b658854e2217b8fe9133f3a976b62ea1a:

=== RUN   TestVMPreemptionPolling
=== RUN   TestVMPreemptionPolling/polling_cancels_test
2024/11/19 12:50:05 test_runner.go:373: test_run_id: test_user-1732020605
2024/11/19 12:50:05 test_runner.go:534: Creating new cluster for test preemption: n0cpu4 (arch="amd64")
2024/11/19 12:50:05 test_runner.go:434: PASS
=== RUN   TestVMPreemptionPolling/polling_doesn't_catch_preemption
==================
WARNING: DATA RACE
Write at 0x00001474ee08 by goroutine 752:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling.func5()
      pkg/cmd/roachtest/test_test.go:753 +0x84
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Previous read at 0x00001474ee08 by goroutine 748:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs.func1()
      pkg/cmd/roachtest/test_runner.go:2148 +0x9b

Goroutine 752 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.TestVMPreemptionPolling()
      pkg/cmd/roachtest/test_test.go:751 +0xb73
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 748 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.monitorForPreemptedVMs()
      pkg/cmd/roachtest/test_runner.go:2143 +0x25a
  github.com/cockroachdb/cockroach/pkg/cmd/roachtest.(*testRunner).runTest.func2()
      pkg/cmd/roachtest/test_runner.go:1305 +0x1a4
==================

Parameters:

  • attempt=1
  • race=true
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Nov 19, 2024
@jbowens @DarrylWong @fqazi
Merge #134671 #135312 #135715
a7a0350 
134671: storage: ingest columnar-block sstables when enabled r=RaduBerinde a=jbowens

When the active cluster verison is sufficiently high and the columnar blocks cluster setting is enabled, build sstables for ingestion in table format TableFormatPebblev5 with columnar blocks.

Epic: none
Release note: none

135312: roachtest: fix TestVMPreemptionPolling data race r=srosenberg,herkolategan a=DarrylWong

This change switches to pollPreemptionInterval to be a mutex protected struct instead, as multiple unit tests modify it and can lead to a data race without.

Fixes: #135267
Epic: none
Release note: none

135715: ccl/schemachangerccl: deflake TestBackupSuccess r=fqazi a=fqazi

Previously, the backup and restore tests for the declarative schema changer could flake because the latest completed schema job could not be sufficiently determined by the finish_time. This was because this time only has resolution of seconds at most, so multiple jobs could have the same finish time. To address this, this patch also sorts the jobs by job_id as well, which should be increasing for later jobs.

Fixes: #131583

Release note: None

Co-authored-by: Jackson Owens <[email protected]>
Co-authored-by: DarrylWong <[email protected]>
Co-authored-by: Faizan Qazi <[email protected]>
@craig craig bot closed this as completed in 8e3466d Nov 19, 2024
DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 19, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
1f4456a 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 19, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
422785c 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 19, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
807e1a0 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
DarrylWong added a commit to DarrylWong/fork that referenced this issue Nov 19, 2024
@DarrylWong
roachtest: fix TestVMPreemptionPolling data race
e529c2e 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
sambhav-jain-16 pushed a commit to sambhav-jain-16/cockroach that referenced this issue Nov 20, 2024
@DarrylWong @sambhav-jain-16
roachtest: fix TestVMPreemptionPolling data race
050e7b8 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
sambhav-jain-16 pushed a commit to sambhav-jain-16/cockroach that referenced this issue Nov 20, 2024
@DarrylWong @sambhav-jain-16
roachtest: fix TestVMPreemptionPolling data race
6cbda2e 
This change switches to pollPreemptionInterval to be a
mutex protected struct instead, as multiple unit tests
modify it and can lead to a data race without.

Fixes: cockroachdb#135267
Epic: none
Release note: none
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-testeng TestEng Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachtest: fix TestVMPreemptionPolling data race DarrylWong/fork
2 participants
@cockroach-teamcity @DarrylWong