Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc/headroom/n4cpu16 failed #80063

Closed
cockroach-teamcity opened this issue Apr 16, 2022 · 2 comments
Closed

roachtest: tpcc/headroom/n4cpu16 failed #80063

cockroach-teamcity opened this issue Apr 16, 2022 · 2 comments
Assignees
@kvoli
Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv KV Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Apr 16, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 771432d1099e516dbc11827c5458886c176e73e3:

	cluster.go:1928,tpcc.go:160,tpcc.go:176,tpcc.go:223,tpcc.go:333,test_runner.go:875: output in run_105718.238099380_n1_cockroach_workload_fixtures_import_tpcc: ././cockroach workload fixtures import tpcc --warehouses=875  returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1962
		  | main.(*clusterImpl).Run
		  | 	main/pkg/cmd/roachtest/cluster.go:1926
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:160
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:176
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCC
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:223
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCC.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:333
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_105718.238099380_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) ././cockroach workload fixtures import tpcc --warehouses=875  returned
		  | stderr:
		  | I220416 10:57:19.142963 1 ccl/workloadccl/fixture.go:318  [-] 1  starting import of 9 tables
		  | I220416 10:57:19.459233 119 ccl/workloadccl/fixture.go:483  [-] 2  imported 46 KiB in warehouse table (875 rows, 0 index entries, took 224.677462ms, 0.20 MiB/s)
		  | I220416 10:57:22.723703 120 ccl/workloadccl/fixture.go:483  [-] 3  imported 880 KiB in district table (8750 rows, 0 index entries, took 3.489024795s, 0.25 MiB/s)
		  | I220416 10:57:23.273837 125 ccl/workloadccl/fixture.go:483  [-] 4  imported 7.9 MiB in item table (100000 rows, 0 index entries, took 4.03915341s, 1.95 MiB/s)
		  | I220416 10:57:32.485273 124 ccl/workloadccl/fixture.go:483  [-] 5  imported 117 MiB in new_order table (7875000 rows, 0 index entries, took 13.250624705s, 8.83 MiB/s)
		  | I220416 10:58:38.856624 123 ccl/workloadccl/fixture.go:483  [-] 6  imported 1.4 GiB in order table (26250000 rows, 26250000 index entries, took 1m19.621876248s, 18.13 MiB/s)
		  | I220416 10:58:48.820644 122 ccl/workloadccl/fixture.go:483  [-] 7  imported 1.9 GiB in history table (26250000 rows, 0 index entries, took 1m29.585916714s, 21.55 MiB/s)
		  | I220416 11:00:21.360088 121 ccl/workloadccl/fixture.go:483  [-] 8  imported 15 GiB in customer table (26250000 rows, 26250000 index entries, took 3m2.125480338s, 84.82 MiB/s)
		  | I220416 11:01:38.581759 126 ccl/workloadccl/fixture.go:483  [-] 9  imported 26 GiB in stock table (87500000 rows, 0 index entries, took 4m19.346994198s, 104.03 MiB/s)
		  | I220416 11:01:59.016580 127 ccl/workloadccl/fixture.go:483  [-] 10  imported 15 GiB in order_line table (262499205 rows, 0 index entries, took 4m39.781933446s, 54.69 MiB/s)
		  | I220416 11:01:59.049722 1 ccl/workloadccl/fixture.go:326  [-] 11  imported 60 GiB bytes in 9 tables (took 4m39.906518932s, 218.75 MiB/s)
		  | I220416 11:02:00.479648 1 ccl/workloadccl/cliccl/fixtures.go:343  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I220416 11:02:00.522039 1 workload/tpcc/tpcc.go:516  [-] 13  check 3.3.2.1 took 42.290321ms
		  | I220416 11:02:11.943221 1 workload/tpcc/tpcc.go:516  [-] 14  check 3.3.2.2 took 11.421065535s
		  | I220416 11:02:14.076969 1 workload/tpcc/tpcc.go:516  [-] 15  check 3.3.2.3 took 2.133685948s
		  | I220416 11:05:06.369643 1 workload/tpcc/tpcc.go:516  [-] 16  check 3.3.2.4 took 2m52.292621357s
		  | I220416 11:05:45.560035 1 workload/tpcc/tpcc.go:516  [-] 17  check 3.3.2.5 took 39.190310133s
		  | I220416 11:08:48.539156 1 workload/tpcc/tpcc.go:516  [-] 18  check 3.3.2.7 took 3m2.979027273s
		  | Error: check failed: 3.3.2.7: driver: bad connection
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ././cockroach workload fixtures import tpcc --warehouses=875
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-15638
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 16, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Apr 16, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 771432d1099e516dbc11827c5458886c176e73e3:

		  |  2366.0s        0            0.0           30.7      0.0      0.0      0.0      0.0 stockLevel
		  |  2367.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 delivery
		  |  2367.0s        0            0.0          306.9      0.0      0.0      0.0      0.0 newOrder
		  |  2367.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 orderStatus
		  |  2367.0s        0            0.0          306.8      0.0      0.0      0.0      0.0 payment
		  |  2367.0s        0            0.0           30.7      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  2368.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 delivery
		  |  2368.0s        0            0.0          306.8      0.0      0.0      0.0      0.0 newOrder
		  |  2368.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 orderStatus
		  |  2368.0s        0            0.0          306.6      0.0      0.0      0.0      0.0 payment
		  |  2368.0s        0            0.0           30.7      0.0      0.0      0.0      0.0 stockLevel
		  |  2369.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 delivery
		  |  2369.0s        0            0.0          306.7      0.0      0.0      0.0      0.0 newOrder
		  |  2369.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 orderStatus
		  |  2369.0s        0            0.0          306.5      0.0      0.0      0.0      0.0 payment
		  |  2369.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 stockLevel
		  |  2370.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 delivery
		  |  2370.0s        0            0.0          306.6      0.0      0.0      0.0      0.0 newOrder
		  |  2370.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 orderStatus
		  |  2370.0s        0            0.0          306.4      0.0      0.0      0.0      0.0 payment
		  |  2370.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 stockLevel
		  |  2371.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 delivery
		  |  2371.0s        0            0.0          306.4      0.0      0.0      0.0      0.0 newOrder
		  |  2371.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 orderStatus
		  |  2371.0s        0            0.0          306.2      0.0      0.0      0.0      0.0 payment
		  |  2371.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  2372.0s        0            0.0           30.5      0.0      0.0      0.0      0.0 delivery
		  |  2372.0s        0            0.0          306.3      0.0      0.0      0.0      0.0 newOrder
		  |  2372.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 orderStatus
		  |  2372.0s        0            0.0          306.1      0.0      0.0      0.0      0.0 payment
		  |  2372.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 stockLevel
		  |  2373.0s        0            0.0           30.5      0.0      0.0      0.0      0.0 delivery
		  |  2373.0s        0            0.0          306.2      0.0      0.0      0.0      0.0 newOrder
		  |  2373.0s        0            0.0           30.5      0.0      0.0      0.0      0.0 orderStatus
		  |  2373.0s        0            0.0          306.0      0.0      0.0      0.0      0.0 payment
		  |  2373.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 stockLevel
		  |  2374.0s        0            0.0           30.5      0.0      0.0      0.0      0.0 delivery
		  |  2374.0s        0            0.0          306.0      0.0      0.0      0.0      0.0 newOrder
		  |  2374.0s        0            0.0           30.5      0.0      0.0      0.0      0.0 orderStatus
		  |  2374.0s        0            0.0          305.9      0.0      0.0      0.0      0.0 payment
		  |  2374.0s        0            0.0           30.6      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=1470 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@kvoli kvoli self-assigned this Apr 16, 2022
@kvoli kvoli added A-kv Anything in KV that doesn't belong in a more specific category. and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Apr 16, 2022
@kvoli
Copy link
Collaborator

kvoli commented Apr 16, 2022

tracking issue #80072, caused by #76609. Resolved in #80071.

@kvoli kvoli closed this as completed Apr 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kvoli kvoli

Labels
A-kv Anything in KV that doesn't belong in a more specific category. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jlinder @cockroach-teamcity @kvoli