roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #89100

cockroach-teamcity · 2022-09-30T16:28:46Z

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on release-22.2 @ c69571de022f48d95d76fb39e378cc9ab9a30afe:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
	monitor.go:127,tpcc.go:1113,tpcc.go:950,test_runner.go:919: monitor failure: monitor task failed: Non-zero exit code: 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1113
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:950
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) Non-zero exit code: 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng _{This test on roachdash | Improve this report!

Jira issue: CRDB-20118
Epic CRDB-20293}

The text was updated successfully, but these errors were encountered:

renatolabs · 2022-09-30T17:37:36Z

Same as #86987.

cockroach-teamcity · 2022-10-01T16:10:53Z

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on release-22.2 @ 600df9f5c387a07fd9b4ba5e54f8b8240645176f:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
	monitor.go:127,tpcc.go:1113,tpcc.go:950,test_runner.go:930: monitor failure: monitor task failed: Non-zero exit code: 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1113
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:950
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) Non-zero exit code: 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-10-03T16:13:29Z

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on release-22.2 @ 36e4cb1677e055e67fa3cfdce81dea18cf10df33:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
	monitor.go:127,tpcc.go:1113,tpcc.go:950,test_runner.go:930: monitor failure: monitor task failed: Non-zero exit code: 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1113
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:950
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	main/pkg/cmd/roachtest/monitor.go:171
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor task failed
		Wraps: (5) Non-zero exit code: 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *install.NonZeroExitCode

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-11-01T15:19:04Z

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on release-22.2 @ fe99dbf5702d1dedeb97d0bac7cb646dc2ec379c:

test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
	monitor.go:127,tpcc.go:1196,tpcc.go:1033,test_runner.go:930: monitor failure: monitor command failure: unexpected node event: 6: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCCBench
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1196
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCCBenchSpec.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:1033
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 6: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

renatolabs · 2022-11-01T15:53:48Z

Disk stall detected on n6:

I221101 15:07:57.600520 11639 1@storage/pebble.go:1005 â‹® [n5] 382  the server is terminating due to a fatal error (see the DEV channel for details)
F221101 15:07:57.600557 11639 storage/pebble.go:1005 â‹® [n5] 383  disk stall detected: pebble unable to write to â€¹/mnt/data1/cockroach/auxiliary/sideloading/r0XXXX/r562/i36.t7â€º in 20.01 seconds

This error is preceded by likely related timed out hearbeat warnings:

W221101 15:07:55.156600 223 kv/kvserver/liveness/liveness.go:885 â‹® [n5,liveness-hb] 376  slow heartbeat took 4.518066869s; err=context deadline exceeded
W221101 15:07:55.156650 223 kv/kvserver/liveness/liveness.go:787 â‹® [n5,liveness-hb] 377  failed node liveness heartbeat: â€¹operation "node liveness heartbeat" timed out after 4.518s (given timeout 4.5s)â€º: context deadline exceeded

This seems like a test flake (hardware problem?). I'll let @cockroachdb/storage take a quick look to confirm if that makes sense.

itsbilal · 2022-11-01T18:34:03Z

@renatolabs yes, that just looks like the hardware / infrastructure got overwhelmed and we were waiting 20s for a write. Likely nothing to do here as this usually happens due to things outside our control (AWS/GCP issues, noisy neighbour, something like that), and it seems like a one-off case too.

We could consider bumping the disk stall threshold back up (it used to be 60s), but maybe all it'd have done here is delayed the same outcome by roughly a minute.

nicktrav · 2022-11-01T18:37:57Z

We could consider bumping the disk stall threshold back up (it used to be 60s)

This was fairly heavily litigated over in #81075. I don't think it's worth going back to 60s, unless this is becoming very noisy / toilsome.

itsbilal · 2022-11-01T18:45:52Z

Ah yeah. Didn't mean to suggest it should go all the way up to 60s, there could be a sweet spot in between - but since this isn't a noisy failure at all I think sticking with the status quo makes sense.

nicktrav · 2022-11-01T18:55:43Z

Looked in the Pebble logs for n6, and I see a bunch of stalled writes, to separate files.

Agree with Bilal's assessment - infra flake.

If we keep seeing these due to the 20s thresholds, let's reconsider.

cockroach-teamcity added this to the 22.2 milestone Sep 30, 2022

blathers-crl bot added the T-testeng TestEng Team label Sep 30, 2022

renatolabs removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 30, 2022

renatolabs mentioned this issue Sep 30, 2022

workload: Bump prepare timeout to 90 minute #88641

Merged

cockroach-teamcity mentioned this issue Oct 26, 2022

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #90700

Closed

nicktrav added the T-storage Storage Team label Nov 1, 2022

blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Nov 1, 2022

nicktrav self-assigned this Nov 1, 2022

exalate-issue-sync bot removed the T-testeng TestEng Team label Nov 1, 2022

nicktrav closed this as completed Nov 1, 2022

nicktrav added the X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue label Dec 12, 2022

irfansharif mentioned this issue Jun 7, 2023

roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #104244

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #89100

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #89100

cockroach-teamcity commented Sep 30, 2022 •

edited by exalate-issue-sync bot

Loading

renatolabs commented Sep 30, 2022

cockroach-teamcity commented Oct 1, 2022

cockroach-teamcity commented Oct 3, 2022

cockroach-teamcity commented Nov 1, 2022

renatolabs commented Nov 1, 2022

itsbilal commented Nov 1, 2022

nicktrav commented Nov 1, 2022

itsbilal commented Nov 1, 2022

nicktrav commented Nov 1, 2022

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #89100

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #89100

Comments

cockroach-teamcity commented Sep 30, 2022 • edited by exalate-issue-sync bot Loading

renatolabs commented Sep 30, 2022

cockroach-teamcity commented Oct 1, 2022

cockroach-teamcity commented Oct 3, 2022

cockroach-teamcity commented Nov 1, 2022

renatolabs commented Nov 1, 2022

itsbilal commented Nov 1, 2022

nicktrav commented Nov 1, 2022

itsbilal commented Nov 1, 2022

nicktrav commented Nov 1, 2022

cockroach-teamcity commented Sep 30, 2022 •

edited by exalate-issue-sync bot

Loading