Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpcc/headroom/n4cpu16 failed [retry err bubbles up] #73713

Closed
cockroach-teamcity opened this issue Dec 11, 2021 · 12 comments
Closed

roachtest: tpcc/headroom/n4cpu16 failed [retry err bubbles up] #73713

cockroach-teamcity opened this issue Dec 11, 2021 · 12 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. S-1 High impact: many users impacted, serious risk of high unavailability or data loss

Comments

@cockroach-teamcity
Copy link
Member

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 406e0fe1e44cabc8a3384703c642469567c6915e:

The test failed on branch=master, cloud=aws:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/headroom/n4cpu16/run_1
	cluster.go:1638,tpcc.go:134,tpcc.go:140,tpcc.go:165,tpcc.go:212,tpcc.go:321,test_runner.go:779: cluster.PutE: put /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/workload failed
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1652
		  | [...repeated from below...]
		Wraps: (2) cluster.PutE
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1436
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:718
		  | main.(*clusterImpl).PutE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1652
		  | main.(*clusterImpl).Put
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1637
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:134
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:140
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:165
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:212
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpcc.go:321
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:779
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (4) put /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/workload failed
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Dec 11, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 78419450178335b31f542bd1b14fefdf4ecee0e8:

		  |  1560.0s        0            4.0            8.9  32212.3 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1561.0s        0            2.0            8.8  47244.6 103079.2 103079.2 103079.2 delivery
		  |  1561.0s        0           47.0           89.4  51539.6 103079.2 103079.2 103079.2 newOrder
		  |  1561.0s        0            4.0            8.9    369.1 103079.2 103079.2 103079.2 orderStatus
		  |  1561.0s        0           32.0           88.4  34359.7 103079.2 103079.2 103079.2 payment
		  |  1561.0s        0            3.0            8.9  66572.0 103079.2 103079.2 103079.2 stockLevel
		  |  1562.0s        0            8.0            8.8  32212.3 103079.2 103079.2 103079.2 delivery
		  |  1562.0s        0           51.0           89.3  21474.8 103079.2 103079.2 103079.2 newOrder
		  |  1562.0s        0            5.0            8.9  81604.4 103079.2 103079.2 103079.2 orderStatus
		  |  1562.0s        0           43.0           88.4  36507.2 103079.2 103079.2 103079.2 payment
		  |  1562.0s        0            6.0            8.9  10737.4 103079.2 103079.2 103079.2 stockLevel
		  |  1563.0s        0            1.0            8.8  24696.1  24696.1  24696.1  24696.1 delivery
		  |  1563.0s        0           50.9           89.3  66572.0 103079.2 103079.2 103079.2 newOrder
		  |  1563.0s        0            4.0            8.9   7247.8 103079.2 103079.2 103079.2 orderStatus
		  |  1563.0s        0           49.9           88.3  10200.5 103079.2 103079.2 103079.2 payment
		  |  1563.0s        0            3.0            8.9    704.6  49392.1  49392.1  49392.1 stockLevel
		  |  1564.0s        0            3.0            8.8  10737.4  36507.2  36507.2  36507.2 delivery
		  |  1564.0s        0           45.0           89.3  49392.1 103079.2 103079.2 103079.2 newOrder
		  |  1564.0s        0            6.0            8.9  64424.5 103079.2 103079.2 103079.2 orderStatus
		  |  1564.0s        0           45.0           88.3  36507.2 103079.2 103079.2 103079.2 payment
		  |  1564.0s        0           11.0            8.9  47244.6  85899.3 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1565.0s        0            4.0            8.8  40802.2 103079.2 103079.2 103079.2 delivery
		  |  1565.0s        0           26.0           89.3  27917.3 103079.2 103079.2 103079.2 newOrder
		  |  1565.0s        0            3.0            8.9  49392.1  85899.3  85899.3  85899.3 orderStatus
		  |  1565.0s        0           36.0           88.3  34359.7 103079.2 103079.2 103079.2 payment
		  |  1565.0s        0            5.0            8.9  45097.2 103079.2 103079.2 103079.2 stockLevel
		  |  1566.0s        0            4.0            8.8  36507.2 103079.2 103079.2 103079.2 delivery
		  |  1566.0s        0           34.0           89.2  36507.2 103079.2 103079.2 103079.2 newOrder
		  |  1566.0s        0            7.0            8.9  47244.6 103079.2 103079.2 103079.2 orderStatus
		  |  1566.0s        0           43.0           88.3  36507.2 103079.2 103079.2 103079.2 payment
		  |  1566.0s        0            7.0            8.9  34359.7 103079.2 103079.2 103079.2 stockLevel
		  |  1567.0s        0            5.0            8.8  36507.2 103079.2 103079.2 103079.2 delivery
		  |  1567.0s        0           48.0           89.2  53687.1 103079.2 103079.2 103079.2 newOrder
		  |  1567.0s        0            9.0            8.9  26843.5  81604.4  81604.4  81604.4 orderStatus
		  |  1567.0s        0           48.0           88.2  51539.6 103079.2 103079.2 103079.2 payment
		  |  1567.0s        0           10.0            8.9  64424.5  98784.2  98784.2  98784.2 stockLevel
		  |  1568.0s        0           12.0            8.8 103079.2 103079.2 103079.2 103079.2 delivery
		  |  1568.0s        0           34.0           89.2  27917.3 103079.2 103079.2 103079.2 newOrder
		  |  1568.0s        0            5.0            8.9  64424.5 103079.2 103079.2 103079.2 orderStatus
		  |  1568.0s        0           46.0           88.2  36507.2 103079.2 103079.2 103079.2 payment
		  |  1568.0s        0            7.0            8.9  66572.0 103079.2 103079.2 103079.2 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=875 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 78419450178335b31f542bd1b14fefdf4ecee0e8:

		  |  2465.0s        0            1.0           18.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2466.0s        0            4.0           18.0  85899.3 103079.2 103079.2 103079.2 delivery
		  |  2466.0s        0           38.0          182.1 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2466.0s        0            0.0           18.1      0.0      0.0      0.0      0.0 orderStatus
		  |  2466.0s        0           24.0          180.7 103079.2 103079.2 103079.2 103079.2 payment
		  |  2466.0s        0            1.0           18.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2467.0s        0            1.0           18.0 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2467.0s        0           31.0          182.1 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2467.0s        0            3.0           18.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2467.0s        0           36.0          180.6 103079.2 103079.2 103079.2 103079.2 payment
		  |  2467.0s        0            1.0           18.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2468.0s        0            4.0           18.0  81604.4 103079.2 103079.2 103079.2 delivery
		  |  2468.0s        0           38.0          182.0 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2468.0s        0            6.0           18.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2468.0s        0           22.0          180.5 103079.2 103079.2 103079.2 103079.2 payment
		  |  2468.0s        0            3.0           18.0 103079.2 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  2469.0s        0            5.0           18.0 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2469.0s        0           37.0          182.0 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2469.0s        0            2.0           18.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2469.0s        0           31.0          180.5 103079.2 103079.2 103079.2 103079.2 payment
		  |  2469.0s        0            4.0           18.0 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2470.0s        0            5.0           18.0 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2470.0s        0           24.0          181.9 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2470.0s        0            2.0           18.1  77309.4 103079.2 103079.2 103079.2 orderStatus
		  |  2470.0s        0           26.0          180.4 103079.2 103079.2 103079.2 103079.2 payment
		  |  2470.0s        0            0.0           18.0      0.0      0.0      0.0      0.0 stockLevel
		  |  2471.0s        0            2.0           18.0 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2471.0s        0           26.0          181.8 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2471.0s        0            4.0           18.0 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2471.0s        0           27.0          180.3 103079.2 103079.2 103079.2 103079.2 payment
		  |  2471.0s        0            1.0           18.0 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2472.0s        0            5.0           18.0 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2472.0s        0           31.0          181.8 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2472.0s        0            3.0           18.0 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2472.0s        0           31.0          180.3 103079.2 103079.2 103079.2 103079.2 payment
		  |  2472.0s        0            3.0           18.0 103079.2 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  2473.0s        0            1.0           18.0 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2473.0s        0           24.0          181.7 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2473.0s        0            3.0           18.0 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2473.0s        0           33.0          180.2 103079.2 103079.2 103079.2 103079.2 payment
		  |  2473.0s        0            3.0           18.0 103079.2 103079.2 103079.2 103079.2 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=1470 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 4b41789120e019ab015e6dbb924df763897ebadb:

		  |   885.0s        0            4.0            9.5  38654.7 103079.2 103079.2 103079.2 stockLevel
		  |   886.0s        0            3.0            9.6  14495.5  77309.4  77309.4  77309.4 delivery
		  |   886.0s        0           55.9           97.9  45097.2 103079.2 103079.2 103079.2 newOrder
		  |   886.0s        0            6.0            9.7    100.7  60129.5  60129.5  60129.5 orderStatus
		  |   886.0s        0           58.9           96.0  45097.2 103079.2 103079.2 103079.2 payment
		  |   886.0s        0            6.0            9.5  32212.3  49392.1  49392.1  49392.1 stockLevel
		  |   887.0s        0            8.0            9.6   9663.7  77309.4  77309.4  77309.4 delivery
		  |   887.0s        0           62.0           97.8  47244.6 103079.2 103079.2 103079.2 newOrder
		  |   887.0s        0            7.0            9.7     39.8  90194.3  90194.3  90194.3 orderStatus
		  |   887.0s        0           52.0           96.0  45097.2 103079.2 103079.2 103079.2 payment
		  |   887.0s        0           12.0            9.5  42949.7 103079.2 103079.2 103079.2 stockLevel
		  |   888.0s        0            6.0            9.6  15032.4 103079.2 103079.2 103079.2 delivery
		  |   888.0s        0           56.1           97.8  45097.2 103079.2 103079.2 103079.2 newOrder
		  |   888.0s        0            4.0            9.7    134.2 103079.2 103079.2 103079.2 orderStatus
		  |   888.0s        0           55.1           95.9  38654.7 103079.2 103079.2 103079.2 payment
		  |   888.0s        0            6.0            9.5  49392.1  85899.3  85899.3  85899.3 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   889.0s        0            3.0            9.6  98784.2 103079.2 103079.2 103079.2 delivery
		  |   889.0s        0           65.0           97.7  26843.5 103079.2 103079.2 103079.2 newOrder
		  |   889.0s        0            5.0            9.7  42949.7  68719.5  68719.5  68719.5 orderStatus
		  |   889.0s        0           64.0           95.9  28991.0 103079.2 103079.2 103079.2 payment
		  |   889.0s        0            8.0            9.5   5905.6  98784.2  98784.2  98784.2 stockLevel
		  |   890.0s        0           11.0            9.6  73014.4 103079.2 103079.2 103079.2 delivery
		  |   890.0s        0           67.0           97.7  53687.1 103079.2 103079.2 103079.2 newOrder
		  |   890.0s        0            8.0            9.7  36507.2  90194.3  90194.3  90194.3 orderStatus
		  |   890.0s        0           62.0           95.8  38654.7 103079.2 103079.2 103079.2 payment
		  |   890.0s        0            8.0            9.5  53687.1 103079.2 103079.2 103079.2 stockLevel
		  |   891.0s        0           10.0            9.6  13958.6  90194.3  90194.3  90194.3 delivery
		  |   891.0s        0           41.0           97.6  51539.6 103079.2 103079.2 103079.2 newOrder
		  |   891.0s        0            6.0            9.7    503.3  49392.1  49392.1  49392.1 orderStatus
		  |   891.0s        0           52.0           95.8  42949.7 103079.2 103079.2 103079.2 payment
		  |   891.0s        0            6.0            9.5  49392.1 103079.2 103079.2 103079.2 stockLevel
		  |   892.0s        0            3.0            9.6  51539.6  94489.3  94489.3  94489.3 delivery
		  |   892.0s        0           54.9           97.6  45097.2 103079.2 103079.2 103079.2 newOrder
		  |   892.0s        0            4.0            9.7  15032.4 103079.2 103079.2 103079.2 orderStatus
		  |   892.0s        0           51.9           95.7  36507.2 103079.2 103079.2 103079.2 payment
		  |   892.0s        0            2.0            9.5  94489.3 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   893.0s        0            2.0            9.6  10200.5  85899.3  85899.3  85899.3 delivery
		  |   893.0s        0           44.1           97.5  21474.8 103079.2 103079.2 103079.2 newOrder
		  |   893.0s        0            9.0            9.7  11274.3 103079.2 103079.2 103079.2 orderStatus
		  |   893.0s        0           68.1           95.7  38654.7 103079.2 103079.2 103079.2 payment
		  |   893.0s        0            6.0            9.5   8321.5 103079.2 103079.2 103079.2 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=875 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@tbg
Copy link
Member

tbg commented Jan 17, 2022

Don't know what the actual problem here is (workload exits 1, but I don't see any obvious problem at the end of its log?) but I found this in n1's log, and those are AssertionFailedf so they shouldn't happen:

I220117 14:44:42.837230 3971458 kv/txn.go:903 ⋮ [n1] 2375  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/56/1/465/0›], [txn: 6ca59860]
I220117 14:44:42.864988 3971505 kv/txn.go:903 ⋮ [n1] 2376  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/57/1/189/5/0›], [txn: c99c41a4]
I220117 14:44:42.918416 3971542 kv/txn.go:903 ⋮ [n1] 2377  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/60/1/75/1/-2115/0›], [txn: 9b3819aa]
I220117 14:44:43.036074 3971572 kv/txn.go:903 ⋮ [n1] 2378  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/57/1/306/8/0›], [txn: 5f8c2300]
I220117 14:44:43.173231 3971913 kv/txn.go:903 ⋮ [n1] 2379  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/56/1/351/0›], [txn: a9cb1461]
I220117 14:44:43.332384 3971976 kv/txn.go:903 ⋮ [n1] 2380  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/57/1/642/8/0›], [txn: 8c2814c6]
I220117 14:44:43.370393 3972328 kv/txn.go:903 ⋮ [n1] 2381  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/57/1/702/9/0›], [txn: 324a29ff]
I220117 14:44:43.752326 3972903 kv/txn.go:903 ⋮ [n1] 2382  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/57/1/183/4/0›], [txn: c2a892d9]
I220117 14:44:43.934247 3973151 kv/txn.go:903 ⋮ [n1] 2383  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/57/1/438/7/0›], [txn: 794c6075]
I220117 14:44:44.100559 3973179 kv/txn.go:903 ⋮ [n1] 2384  async rollback failed: concurrent txn use detected. ba: EndTxn(abort) [‹/Table/60/1/477/1/-2113/0›], [txn: 870269da]

This was added in 6e1d241 cc @andreimatei

@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 912964e02ddd951c77d4f71981ae18b3894e9084:

		  |  2528.0s        0            2.0           11.1  77309.4  90194.3  90194.3  90194.3 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  2529.0s        0            7.0           11.1  90194.3 103079.2 103079.2 103079.2 delivery
		  |  2529.0s        0           31.0          112.2 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2529.0s        0            4.0           11.1    151.0 103079.2 103079.2 103079.2 orderStatus
		  |  2529.0s        0           28.0          111.2 103079.2 103079.2 103079.2 103079.2 payment
		  |  2529.0s        0            0.0           11.1      0.0      0.0      0.0      0.0 stockLevel
		  |  2530.0s        0            4.0           11.1  98784.2 103079.2 103079.2 103079.2 delivery
		  |  2530.0s        0           37.0          112.2 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2530.0s        0            4.0           11.1  45097.2 103079.2 103079.2 103079.2 orderStatus
		  |  2530.0s        0           33.0          111.1 103079.2 103079.2 103079.2 103079.2 payment
		  |  2530.0s        0            6.0           11.1  98784.2 103079.2 103079.2 103079.2 stockLevel
		  |  2531.0s        0            3.0           11.1 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2531.0s        0           29.0          112.2 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2531.0s        0            5.0           11.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2531.0s        0           25.0          111.1 103079.2 103079.2 103079.2 103079.2 payment
		  |  2531.0s        0            4.0           11.1  81604.4 103079.2 103079.2 103079.2 stockLevel
		  |  2532.0s        0            3.0           11.1 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2532.0s        0           26.0          112.1 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2532.0s        0            2.0           11.1  12884.9 103079.2 103079.2 103079.2 orderStatus
		  |  2532.0s        0           27.0          111.1  73014.4 103079.2 103079.2 103079.2 payment
		  |  2532.0s        0            1.0           11.1  73014.4  73014.4  73014.4  73014.4 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  2533.0s        0            5.0           11.1 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2533.0s        0           42.0          112.1 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2533.0s        0            5.0           11.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2533.0s        0           36.0          111.0 103079.2 103079.2 103079.2 103079.2 payment
		  |  2533.0s        0            5.0           11.1  81604.4 103079.2 103079.2 103079.2 stockLevel
		  |  2534.0s        0            7.0           11.1 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2534.0s        0           34.0          112.1 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2534.0s        0            3.0           11.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2534.0s        0           26.0          111.0 103079.2 103079.2 103079.2 103079.2 payment
		  |  2534.0s        0            3.0           11.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2535.0s        0            6.0           11.1 103079.2 103079.2 103079.2 103079.2 delivery
		  |  2535.0s        0           38.0          112.1 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2535.0s        0            3.0           11.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  2535.0s        0           30.0          111.0  68719.5 103079.2 103079.2 103079.2 payment
		  |  2535.0s        0            5.0           11.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  2536.0s        0            2.0           11.1  42949.7 103079.2 103079.2 103079.2 delivery
		  |  2536.0s        0           31.0          112.0 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  2536.0s        0            2.0           11.1  31138.5 103079.2 103079.2 103079.2 orderStatus
		  |  2536.0s        0           26.0          110.9 103079.2 103079.2 103079.2 103079.2 payment
		  |  2536.0s        0            1.0           11.1  12348.0  12348.0  12348.0  12348.0 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=1470 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@tbg tbg mentioned this issue Jan 20, 2022
Open
@tbg
Copy link
Member

tbg commented Jan 20, 2022

concurrent txn use detected

Filed #75209 for this so that it gets some attention.

@tbg
Copy link
Member

tbg commented Jan 20, 2022

The actual problem is here (maybe related to the assertion, how knows):

#75209

This is the same as in #74892 (comment)

@tbg tbg changed the title roachtest: tpcc/headroom/n4cpu16 failed roachtest: tpcc/headroom/n4cpu16 failed [retry err bubbles up] Jan 20, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 58ceac139a7e83052171121b28026a7366f16f7e:

		  |  1185.0s        0            3.0            8.6  64424.5 103079.2 103079.2 103079.2 stockLevel
		  |  1186.0s        0            5.0            8.6  64424.5 103079.2 103079.2 103079.2 delivery
		  |  1186.0s        0           40.9           87.5  51539.6 103079.2 103079.2 103079.2 newOrder
		  |  1186.0s        0            5.0            8.7   1409.3  55834.6  55834.6  55834.6 orderStatus
		  |  1186.0s        0           36.9           86.0  47244.6 103079.2 103079.2 103079.2 payment
		  |  1186.0s        0            4.0            8.6  77309.4 103079.2 103079.2 103079.2 stockLevel
		  |  1187.0s        0            2.0            8.6  33286.0  34359.7  34359.7  34359.7 delivery
		  |  1187.0s        0           40.0           87.5  60129.5 103079.2 103079.2 103079.2 newOrder
		  |  1187.0s        0            5.0            8.7  19327.4  77309.4  77309.4  77309.4 orderStatus
		  |  1187.0s        0           38.0           86.0  38654.7 103079.2 103079.2 103079.2 payment
		  |  1187.0s        0            0.0            8.6      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1188.0s        0            3.0            8.6  73014.4 103079.2 103079.2 103079.2 delivery
		  |  1188.0s        0           36.0           87.4  38654.7 103079.2 103079.2 103079.2 newOrder
		  |  1188.0s        0            6.0            8.7   3355.4 103079.2 103079.2 103079.2 orderStatus
		  |  1188.0s        0           33.0           85.9  36507.2 103079.2 103079.2 103079.2 payment
		  |  1188.0s        0            3.0            8.6  40802.2 103079.2 103079.2 103079.2 stockLevel
		  |  1189.0s        0            6.0            8.6  33286.0 103079.2 103079.2 103079.2 delivery
		  |  1189.0s        0           46.0           87.4  36507.2 103079.2 103079.2 103079.2 newOrder
		  |  1189.0s        0            4.0            8.7  15569.3 103079.2 103079.2 103079.2 orderStatus
		  |  1189.0s        0           35.0           85.9  49392.1 103079.2 103079.2 103079.2 payment
		  |  1189.0s        0            1.0            8.6   8589.9   8589.9   8589.9   8589.9 stockLevel
		  |  1190.0s        0            1.0            8.6  24696.1  24696.1  24696.1  24696.1 delivery
		  |  1190.0s        0           31.0           87.3  53687.1 103079.2 103079.2 103079.2 newOrder
		  |  1190.0s        0            2.0            8.7  31138.5 103079.2 103079.2 103079.2 orderStatus
		  |  1190.0s        0           30.0           85.8  42949.7 103079.2 103079.2 103079.2 payment
		  |  1190.0s        0            4.0            8.6  12348.0  85899.3  85899.3  85899.3 stockLevel
		  |  1191.0s        0            3.0            8.6  90194.3 103079.2 103079.2 103079.2 delivery
		  |  1191.0s        0           35.0           87.3  45097.2 103079.2 103079.2 103079.2 newOrder
		  |  1191.0s        0            5.0            8.7  49392.1 103079.2 103079.2 103079.2 orderStatus
		  |  1191.0s        0           27.0           85.8  20401.1 103079.2 103079.2 103079.2 payment
		  |  1191.0s        0            3.0            8.6 103079.2 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1192.0s        0            4.0            8.6  20401.1  85899.3  85899.3  85899.3 delivery
		  |  1192.0s        0           32.0           87.2  49392.1 103079.2 103079.2 103079.2 newOrder
		  |  1192.0s        0            4.0            8.7  77309.4 103079.2 103079.2 103079.2 orderStatus
		  |  1192.0s        0           29.0           85.7  47244.6 103079.2 103079.2 103079.2 payment
		  |  1192.0s        0            7.0            8.6  85899.3 103079.2 103079.2 103079.2 stockLevel
		  |  1193.0s        0            5.0            8.6 103079.2 103079.2 103079.2 103079.2 delivery
		  |  1193.0s        0           24.0           87.2  40802.2 103079.2 103079.2 103079.2 newOrder
		  |  1193.0s        0            2.0            8.7  13958.6  66572.0  66572.0  66572.0 orderStatus
		  |  1193.0s        0           30.0           85.7  40802.2 103079.2 103079.2 103079.2 payment
		  |  1193.0s        0            3.0            8.6  15569.3 103079.2 103079.2 103079.2 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=875 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ dc07599dc9db1acd5afa3a6537297815f25c1fca:

		  |  1882.0s        0            2.0           11.8  64424.5  77309.4  77309.4  77309.4 stockLevel
		  |  1883.0s        0            3.0           11.9  85899.3 103079.2 103079.2 103079.2 delivery
		  |  1883.0s        0           29.0          119.2  62277.0 103079.2 103079.2 103079.2 newOrder
		  |  1883.0s        0            3.0           11.8  51539.6 103079.2 103079.2 103079.2 orderStatus
		  |  1883.0s        0           44.0          118.1  60129.5 103079.2 103079.2 103079.2 payment
		  |  1883.0s        0            2.0           11.8  32212.3  42949.7  42949.7  42949.7 stockLevel
		  |  1884.0s        0            1.0           11.9  49392.1  49392.1  49392.1  49392.1 delivery
		  |  1884.0s        0           45.0          119.2  66572.0 103079.2 103079.2 103079.2 newOrder
		  |  1884.0s        0            1.0           11.8  51539.6  51539.6  51539.6  51539.6 orderStatus
		  |  1884.0s        0           31.0          118.0  64424.5 103079.2 103079.2 103079.2 payment
		  |  1884.0s        0            2.0           11.8    738.2  55834.6  55834.6  55834.6 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1885.0s        0           11.0           11.9  62277.0 103079.2 103079.2 103079.2 delivery
		  |  1885.0s        0           27.0          119.1  73014.4 103079.2 103079.2 103079.2 newOrder
		  |  1885.0s        0            0.0           11.8      0.0      0.0      0.0      0.0 orderStatus
		  |  1885.0s        0           34.0          118.0  47244.6 103079.2 103079.2 103079.2 payment
		  |  1885.0s        0            1.0           11.8  85899.3  85899.3  85899.3  85899.3 stockLevel
		  |  1886.0s        0            7.0           11.9  28991.0  85899.3  85899.3  85899.3 delivery
		  |  1886.0s        0           50.9          119.1  64424.5 103079.2 103079.2 103079.2 newOrder
		  |  1886.0s        0            6.0           11.8  60129.5 103079.2 103079.2 103079.2 orderStatus
		  |  1886.0s        0           50.9          118.0  62277.0 103079.2 103079.2 103079.2 payment
		  |  1886.0s        0            2.0           11.8  42949.7  60129.5  60129.5  60129.5 stockLevel
		  |  1887.0s        0            4.0           11.9  30064.8 103079.2 103079.2 103079.2 delivery
		  |  1887.0s        0           36.0          119.1  51539.6 103079.2 103079.2 103079.2 newOrder
		  |  1887.0s        0            2.0           11.8  22548.6  27917.3  27917.3  27917.3 orderStatus
		  |  1887.0s        0           36.0          117.9  55834.6 103079.2 103079.2 103079.2 payment
		  |  1887.0s        0            4.0           11.8  81604.4 103079.2 103079.2 103079.2 stockLevel
		  |  1888.0s        0            9.0           11.9  42949.7 103079.2 103079.2 103079.2 delivery
		  |  1888.0s        0           36.0          119.0  81604.4 103079.2 103079.2 103079.2 newOrder
		  |  1888.0s        0            4.0           11.8     50.3  81604.4  81604.4  81604.4 orderStatus
		  |  1888.0s        0           45.0          117.9  62277.0 103079.2 103079.2 103079.2 payment
		  |  1888.0s        0            6.0           11.8  45097.2 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1889.0s        0            9.0           11.9  42949.7  77309.4  77309.4  77309.4 delivery
		  |  1889.0s        0           42.0          119.0  66572.0 103079.2 103079.2 103079.2 newOrder
		  |  1889.0s        0            8.0           11.8  73014.4 103079.2 103079.2 103079.2 orderStatus
		  |  1889.0s        0           31.0          117.8  33286.0 103079.2 103079.2 103079.2 payment
		  |  1889.0s        0            2.0           11.8  38654.7  49392.1  49392.1  49392.1 stockLevel
		  |  1890.0s        0            2.0           11.9   5100.3  81604.4  81604.4  81604.4 delivery
		  |  1890.0s        0           27.0          118.9  77309.4 103079.2 103079.2 103079.2 newOrder
		  |  1890.0s        0            9.0           11.8  40802.2 103079.2 103079.2 103079.2 orderStatus
		  |  1890.0s        0           36.0          117.8  42949.7 103079.2 103079.2 103079.2 payment
		  |  1890.0s        0            2.0           11.8 103079.2 103079.2 103079.2 103079.2 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=875 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ dc07599dc9db1acd5afa3a6537297815f25c1fca:

		  |  5299.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		  |  5300.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5300.0s        0            3.0          131.7 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5300.0s        0            1.0           13.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  5300.0s        0            4.0          130.8 103079.2 103079.2 103079.2 103079.2 payment
		  |  5300.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  5301.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5301.0s        0            2.0          131.7 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5301.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 orderStatus
		  |  5301.0s        0            6.0          130.8 103079.2 103079.2 103079.2 103079.2 payment
		  |  5301.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		  |  5302.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5302.0s        0            6.0          131.7 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5302.0s        0            1.0           13.1 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  5302.0s        0            1.0          130.8 103079.2 103079.2 103079.2 103079.2 payment
		  |  5302.0s        0            1.0           13.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  5303.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5303.0s        0            1.0          131.6 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5303.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 orderStatus
		  |  5303.0s        0            6.0          130.8 103079.2 103079.2 103079.2 103079.2 payment
		  |  5303.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		  |  5304.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5304.0s        0            3.0          131.6 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5304.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 orderStatus
		  |  5304.0s        0            1.0          130.7 103079.2 103079.2 103079.2 103079.2 payment
		  |  5304.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  5305.0s        0            1.0           13.1 103079.2 103079.2 103079.2 103079.2 delivery
		  |  5305.0s        0            0.0          131.6      0.0      0.0      0.0      0.0 newOrder
		  |  5305.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 orderStatus
		  |  5305.0s        0            2.0          130.7 103079.2 103079.2 103079.2 103079.2 payment
		  |  5305.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		  |  5306.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5306.0s        0            5.0          131.6 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5306.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 orderStatus
		  |  5306.0s        0            0.0          130.7      0.0      0.0      0.0      0.0 payment
		  |  5306.0s        0            1.0           13.1 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  5307.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 delivery
		  |  5307.0s        0            1.0          131.5 103079.2 103079.2 103079.2 103079.2 newOrder
		  |  5307.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 orderStatus
		  |  5307.0s        0            2.0          130.7   6174.0 103079.2 103079.2 103079.2 payment
		  |  5307.0s        0            0.0           13.1      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=1470 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ e1068d77afbd39b162978281c9da7cbea49c1c3a:

		  |  1034.0s        0            3.0           10.9  42949.7 103079.2 103079.2 103079.2 stockLevel
		  |  1035.0s        0            5.0           10.9  98784.2 103079.2 103079.2 103079.2 delivery
		  |  1035.0s        0           36.0          111.0  51539.6 103079.2 103079.2 103079.2 newOrder
		  |  1035.0s        0            5.0           10.9  47244.6  77309.4  77309.4  77309.4 orderStatus
		  |  1035.0s        0           33.0          109.3 103079.2 103079.2 103079.2 103079.2 payment
		  |  1035.0s        0            3.0           10.9   6442.5  47244.6  47244.6  47244.6 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1036.0s        0            6.0           10.9  27917.3  98784.2  98784.2  98784.2 delivery
		  |  1036.0s        0           23.0          110.9  73014.4 103079.2 103079.2 103079.2 newOrder
		  |  1036.0s        0            3.0           10.9 103079.2 103079.2 103079.2 103079.2 orderStatus
		  |  1036.0s        0           42.1          109.2  77309.4 103079.2 103079.2 103079.2 payment
		  |  1036.0s        0            2.0           10.9  94489.3 103079.2 103079.2 103079.2 stockLevel
		  |  1037.0s        0            6.0           10.9  73014.4 103079.2 103079.2 103079.2 delivery
		  |  1037.0s        0           39.0          110.9  51539.6 103079.2 103079.2 103079.2 newOrder
		  |  1037.0s        0            4.0           10.9  12348.0 103079.2 103079.2 103079.2 orderStatus
		  |  1037.0s        0           36.0          109.2  28991.0 103079.2 103079.2 103079.2 payment
		  |  1037.0s        0            3.0           10.9  11811.2 103079.2 103079.2 103079.2 stockLevel
		  |  1038.0s        0            4.0           10.9  38654.7  85899.3  85899.3  85899.3 delivery
		  |  1038.0s        0           31.0          110.8  77309.4 103079.2 103079.2 103079.2 newOrder
		  |  1038.0s        0            2.0           10.9    234.9  12884.9  12884.9  12884.9 orderStatus
		  |  1038.0s        0           47.0          109.1  47244.6 103079.2 103079.2 103079.2 payment
		  |  1038.0s        0            5.0           10.9  15569.3  45097.2  45097.2  45097.2 stockLevel
		  |  1039.0s        0            3.0           10.9 103079.2 103079.2 103079.2 103079.2 delivery
		  |  1039.0s        0           39.1          110.7  73014.4 103079.2 103079.2 103079.2 newOrder
		  |  1039.0s        0            4.0           10.9     22.0 103079.2 103079.2 103079.2 orderStatus
		  |  1039.0s        0           47.1          109.1  68719.5 103079.2 103079.2 103079.2 payment
		  |  1039.0s        0            2.0           10.9 103079.2 103079.2 103079.2 103079.2 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1040.0s        0            4.0           10.9  11274.3  73014.4  73014.4  73014.4 delivery
		  |  1040.0s        0           46.9          110.7  73014.4 103079.2 103079.2 103079.2 newOrder
		  |  1040.0s        0            2.0           10.9    738.2  68719.5  68719.5  68719.5 orderStatus
		  |  1040.0s        0           34.9          109.0  42949.7 103079.2 103079.2 103079.2 payment
		  |  1040.0s        0            4.0           10.9  73014.4 103079.2 103079.2 103079.2 stockLevel
		  |  1041.0s        0            1.0           10.9   9126.8   9126.8   9126.8   9126.8 delivery
		  |  1041.0s        0           33.0          110.6  77309.4 103079.2 103079.2 103079.2 newOrder
		  |  1041.0s        0            8.0           10.9  23622.3 103079.2 103079.2 103079.2 orderStatus
		  |  1041.0s        0           42.0          108.9  77309.4 103079.2 103079.2 103079.2 payment
		  |  1041.0s        0            4.0           10.9 103079.2 103079.2 103079.2 103079.2 stockLevel
		  |  1042.0s        0            3.0           10.9  77309.4 103079.2 103079.2 103079.2 delivery
		  |  1042.0s        0           39.0          110.5  60129.5 103079.2 103079.2 103079.2 newOrder
		  |  1042.0s        0            2.0           10.9     56.6  40802.2  40802.2  40802.2 orderStatus
		  |  1042.0s        0           30.0          108.8  62277.0 103079.2 103079.2 103079.2 payment
		  |  1042.0s        0            9.0           10.9 103079.2 103079.2 103079.2 103079.2 stockLevel
		Wraps: (8) COMMAND_PROBLEM
		Wraps: (9) Node 4. Command with error:
		  | ``````
		  | ./cockroach workload run tpcc --warehouses=875 --histograms=perf/stats.json  --ramp=5m0s --duration=2h0m0s --prometheus-port=2112 --pprofport=33333  {pgurl:1-3}
		  | ``````
		Wraps: (10) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *cluster.WithCommandDetails (8) errors.Cmd (9) *hintdetail.withDetail (10) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@tbg tbg added the S-1 High impact: many users impacted, serious risk of high unavailability or data loss label Feb 1, 2022
@tbg
Copy link
Member

tbg commented Feb 2, 2022

Used to fail daily, no longer does.

@tbg tbg closed this as completed Feb 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. S-1 High impact: many users impacted, serious risk of high unavailability or data loss
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tbg @cockroach-teamcity