Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: failover/non-system/pause failed #123179

Closed
cockroach-teamcity opened this issue Apr 27, 2024 · 4 comments
Closed

roachtest: failover/non-system/pause failed #123179

cockroach-teamcity opened this issue Apr 27, 2024 · 4 comments
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 27, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.failover/non-system/pause failed with artifacts on master @ 5d02bd9ff6b2bccecf6d43fc6cd647167b91f782:

(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/pause/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-38226
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Apr 27, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Apr 27, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.failover/non-system/pause failed with artifacts on master @ 5d02bd9ff6b2bccecf6d43fc6cd647167b91f782:

(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/pause/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@miraradeva
Copy link
Contributor

Similar to #123186; node 5 crashed with:

panic: runtime error: index out of range [1968] with length 300

goroutine 274872 gp=0xc006e42c40 m=9 mp=0xc0013d8808 [running]:
panic({0x60c44e0?, 0xc0238cdf98?})
	GOROOT/src/runtime/panic.go:779 +0x158 fp=0xc006b5eb00 sp=0xc006b5ea50 pc=0x49bd98
runtime.goPanicIndex(0x7b0, 0x12c)
	GOROOT/src/runtime/panic.go:114 +0x7c fp=0xc006b5eb40 sp=0xc006b5eb00 pc=0x49a6dc
github.com/cockroachdb/cockroach/pkg/storage/disk.(*monitorTracer).String(0xc001c44dc0)
	github.com/cockroachdb/cockroach/pkg/storage/disk/monitor_tracer.go:140 +0x4d7 fp=0xc006b5ef08 sp=0xc006b5eb40 pc=0x2014e37
github.com/cockroachdb/cockroach/pkg/storage/disk.(*Monitor).LogTrace(...)
	github.com/cockroachdb/cockroach/pkg/storage/disk/monitor.go:275
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func4.1()
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1460 +0x4f fp=0xc006b5efb0 sp=0xc006b5ef08 pc=0x208006f
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).async.func1()
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1378 +0x59 fp=0xc006b5efe0 sp=0xc006b5efb0 pc=0x207f4b9
runtime.goexit({})
	src/runtime/asm_amd64.s:1695 +0x1 fp=0xc006b5efe8 sp=0xc006b5efe0 pc=0x4d7e01
created by github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).async in goroutine 274871
	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1376 +0x7c

Pinging @cockroachdb/storage.

@miraradeva miraradeva added the T-storage Storage Team label Apr 29, 2024
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Apr 29, 2024
@andrewbaptist andrewbaptist added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Apr 29, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.failover/non-system/pause failed with artifacts on master @ 2b5baff23f9b53da26f0fc1a763ca43384b464b0:

(failover.go:1766).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/pause/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_encrypted=false
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

This test on roachdash | Improve this report!

@jbowens
Copy link
Collaborator

jbowens commented Apr 29, 2024

Fixed by #123218

@jbowens jbowens closed this as completed Apr 29, 2024
@exalate-issue-sync exalate-issue-sync bot removed the T-kv KV Team label Apr 29, 2024
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
@github-project-automation github-project-automation bot moved this to roachtest/unit test backlog in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Projects
No open projects
KV
Status: roachtest/unit test backlog
[Deprecated] Storage
Archived in project
Milestone
24.1
Development

No branches or pull requests
4 participants
@jbowens @cockroach-teamcity @andrewbaptist @miraradeva