roachtest: tpcc/multiregion/survive=region/chaos=true failed #78619

cockroach-teamcity · 2022-03-28T14:49:09Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:268,tpcc.go:578,test_runner.go:866: error at from 2022-03-28T13:33:46Z, to 2022-03-28T13:38:26Z on metric workload_tpcc_newOrder_success_total{instance="35.237.31.87:2122"}: expected successes to not increase, found from 6763.000000, to 6765.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-03-28T13:33:46Z, to 2022-03-28T13:38:26Z on metric workload_tpcc_newOrder_success_total{instance="35.237.31.87:2122"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkDowntime.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:97
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkDowntime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:267
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected successes to not increase, found from 6763.000000, to 6765.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master]

/cc @cockroachdb/multiregion _{This test on roachdash | Improve this report!

Jira issue: CRDB-14198}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-04-13T14:37:08Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ f3116d2166b3a43b5c22eabe3b76ddc958ccdb00:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:875: error at from 2022-04-13T13:51:59Z, to 2022-04-13T13:56:39Z on metric workload_tpcc_delivery_success_total{instance="34.148.27.248:2122"}: expected successes to not increase, found from 1794.000000, to 1795.000000
		(1) secondary error attachment
		  | error at from 2022-04-13T13:51:59Z, to 2022-04-13T13:56:39Z on metric workload_tpcc_payment_success_total{instance="34.148.27.248:2122"}: expected successes to not increase, found from 17827.000000, to 17828.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-04-13T13:51:59Z, to 2022-04-13T13:56:39Z on metric workload_tpcc_payment_success_total{instance="34.148.27.248:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkDowntime.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:97
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkDowntime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:86
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:267
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected successes to not increase, found from 17827.000000, to 17828.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-04-13T13:51:59Z, to 2022-04-13T13:56:39Z on metric workload_tpcc_delivery_success_total{instance="34.148.27.248:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkDowntime.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:97
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkDowntime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:86
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:267
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected successes to not increase, found from 1794.000000, to 1795.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-22T15:04:30Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ fd435e6d411a79425eba7ecdf82f0efb66025550:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:875: error at from 2022-04-22T13:28:26Z, to 2022-04-22T13:33:06Z on metric workload_tpcc_newOrder_error_total{instance="35.231.35.54:2111"}: expected <=200 errors, found from 31.000000, to 496.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-04-22T13:28:26Z, to 2022-04-22T13:33:06Z on metric workload_tpcc_newOrder_error_total{instance="35.231.35.54:2111"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 31.000000, to 496.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-24T14:34:07Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 6397420966d4ee3fa80f31ef2d3fccd74060b759:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 652240.000000, to 653867.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) secondary error attachment
		  | error at from 2022-04-24T12:58:42Z, to 2022-04-24T13:03:22Z on metric workload_tpcc_payment_error_total{instance="34.148.198.127:2122"}: expected <=200 errors, found from 0.000000, to 211.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-04-24T12:58:42Z, to 2022-04-24T13:03:22Z on metric workload_tpcc_payment_error_total{instance="34.148.198.127:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 211.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (7) error at from 2022-04-24T12:58:42Z, to 2022-04-24T13:03:22Z on metric workload_tpcc_newOrder_error_total{instance="34.148.198.127:2122"}
		Wraps: (8) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (9) expected <=200 errors, found from 0.000000, to 475.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *withstack.withStack (7) *errutil.withPrefix (8) *withstack.withStack (9) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-27T14:56:13Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 9875e24d312b4abc54597913d7a424bace7bd023:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:875: error at from 2022-04-27T13:18:30Z, to 2022-04-27T13:23:10Z on metric workload_tpcc_newOrder_error_total{instance="35.185.18.90:2122"}: expected <=200 errors, found from 15.000000, to 1023.000000
		(1) secondary error attachment
		  | error at from 2022-04-27T13:18:30Z, to 2022-04-27T13:23:10Z on metric workload_tpcc_payment_error_total{instance="35.185.18.90:2122"}: expected <=200 errors, found from 0.000000, to 876.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-04-27T13:18:30Z, to 2022-04-27T13:23:10Z on metric workload_tpcc_payment_error_total{instance="35.185.18.90:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 876.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-04-27T13:18:30Z, to 2022-04-27T13:23:10Z on metric workload_tpcc_newOrder_error_total{instance="35.185.18.90:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 15.000000, to 1023.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-17T14:45:32Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ f6578028f0b029cbb623bd278c57c6bef85c4835:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 601684.000000, to 602047.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) secondary error attachment
		  | error at from 2022-05-17T14:09:01Z, to 2022-05-17T14:13:41Z on metric workload_tpcc_payment_error_total{instance="34.139.122.241:2112"}: expected <=200 errors, found from 627420.000000, to 627820.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-05-17T14:09:01Z, to 2022-05-17T14:13:41Z on metric workload_tpcc_payment_error_total{instance="34.139.122.241:2112"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 627420.000000, to 627820.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (4) error at from 2022-05-17T14:09:01Z, to 2022-05-17T14:13:41Z on metric workload_tpcc_payment_error_total{instance="34.139.122.241:2121"}
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) expected <=200 errors, found from 833894.000000, to 834303.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-20T14:24:25Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ fed727622d3541bf9f6ea1d168184f354a822f0e:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:875: error at from 2022-05-20T12:48:35Z, to 2022-05-20T12:53:15Z on metric workload_tpcc_newOrder_error_total{instance="35.231.58.147:2122"}: expected <=200 errors, found from 0.000000, to 433.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-05-20T12:48:35Z, to 2022-05-20T12:53:15Z on metric workload_tpcc_newOrder_error_total{instance="35.231.58.147:2122"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 0.000000, to 433.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-22T14:07:01Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 7f8c647f1a51e0fe50271337ef4c6bba898c6a8d:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 59695.000000, to 59898.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) secondary error attachment
		  | error at from 2022-05-22T13:30:30Z, to 2022-05-22T13:35:10Z on metric workload_tpcc_payment_error_total{instance="34.139.160.109:2122"}: expected <=200 errors, found from 597022.000000, to 598851.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-05-22T13:30:30Z, to 2022-05-22T13:35:10Z on metric workload_tpcc_payment_error_total{instance="34.139.160.109:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 597022.000000, to 598851.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (4) error at from 2022-05-22T13:30:30Z, to 2022-05-22T13:35:10Z on metric workload_tpcc_newOrder_error_total{instance="34.139.160.109:2122"}
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) expected <=200 errors, found from 597042.000000, to 599087.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-23T14:57:09Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ f143fd81c898f540a0a0db467b5f18beb32644cf:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 392.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) secondary error attachment
		  | error at from 2022-05-23T13:21:27Z, to 2022-05-23T13:26:07Z on metric workload_tpcc_newOrder_error_total{instance="34.73.180.78:2121"}: expected <=200 errors, found from 0.000000, to 341.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-05-23T13:21:27Z, to 2022-05-23T13:26:07Z on metric workload_tpcc_newOrder_error_total{instance="34.73.180.78:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 341.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (6) error at from 2022-05-23T13:21:27Z, to 2022-05-23T13:26:07Z on metric workload_tpcc_newOrder_error_total{instance="34.73.180.78:2111"}
		Wraps: (7) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (8) expected <=200 errors, found from 0.000000, to 595.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-24T14:39:59Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ e961382aae33bbe6b4d557a1d2a2eec111334a4e:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 842.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-05-24T13:02:55Z, to 2022-05-24T13:07:35Z on metric workload_tpcc_newOrder_error_total{instance="34.148.43.212:2122"}: expected <=200 errors, found from 37.000000, to 1643.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-05-24T13:02:55Z, to 2022-05-24T13:07:35Z on metric workload_tpcc_newOrder_error_total{instance="34.148.43.212:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 37.000000, to 1643.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-05-24T13:02:55Z, to 2022-05-24T13:07:35Z on metric workload_tpcc_newOrder_error_total{instance="34.148.43.212:2111"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 49.000000, to 996.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-27T14:39:02Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ d64792534335d20233269318e2b1b4ce694d394e:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-05-27T13:02:23Z, to 2022-05-27T13:07:03Z on metric workload_tpcc_newOrder_error_total{instance="34.138.111.86:2122"}: expected <=200 errors, found from 32.000000, to 274.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-05-27T13:02:23Z, to 2022-05-27T13:07:03Z on metric workload_tpcc_newOrder_error_total{instance="34.138.111.86:2122"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 32.000000, to 274.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-05-31T14:18:25Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 68ed08b9fbb9862411138deed1c39eac7702ae48:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 60377.000000, to 60786.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) secondary error attachment
		  | error at from 2022-05-31T13:41:46Z, to 2022-05-31T13:46:26Z on metric workload_tpcc_payment_error_total{instance="34.74.74.182:2122"}: expected <=200 errors, found from 603900.000000, to 607884.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-05-31T13:41:46Z, to 2022-05-31T13:46:26Z on metric workload_tpcc_payment_error_total{instance="34.74.74.182:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 603900.000000, to 607884.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (4) error at from 2022-05-31T13:41:46Z, to 2022-05-31T13:46:26Z on metric workload_tpcc_newOrder_error_total{instance="34.74.74.182:2122"}
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) expected <=200 errors, found from 603886.000000, to 607897.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-01T14:56:15Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 242d5100e778983cf0b56d11ffaf766ea54530f7:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 851.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) secondary error attachment
		  | error at from 2022-06-01T13:19:49Z, to 2022-06-01T13:24:29Z on metric workload_tpcc_newOrder_error_total{instance="104.196.67.95:2122"}: expected <=200 errors, found from 0.000000, to 1560.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-01T13:19:49Z, to 2022-06-01T13:24:29Z on metric workload_tpcc_newOrder_error_total{instance="104.196.67.95:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1560.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (4) error at from 2022-06-01T13:19:49Z, to 2022-06-01T13:24:29Z on metric workload_tpcc_newOrder_error_total{instance="104.196.67.95:2111"}
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) expected <=200 errors, found from 0.000000, to 1245.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-02T14:24:59Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 73ddb31f9b156f429f93a62cb9ce22da1fb2fe83:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-02T12:49:02Z, to 2022-06-02T12:53:42Z on metric workload_tpcc_newOrder_error_total{instance="34.148.133.137:2122"}: expected <=200 errors, found from 26.000000, to 283.000000
		(1) secondary error attachment
		  | error at from 2022-06-02T12:49:02Z, to 2022-06-02T12:53:42Z on metric workload_tpcc_payment_error_total{instance="34.148.133.137:2122"}: expected <=200 errors, found from 0.000000, to 230.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-02T12:49:02Z, to 2022-06-02T12:53:42Z on metric workload_tpcc_payment_error_total{instance="34.148.133.137:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 230.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-06-02T12:49:02Z, to 2022-06-02T12:53:42Z on metric workload_tpcc_newOrder_error_total{instance="34.148.133.137:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 26.000000, to 283.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-04T14:50:12Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 380f15c04826b432273cbd7ffe96b335ffc90396:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-04T13:14:08Z, to 2022-06-04T13:18:48Z on metric workload_tpcc_newOrder_error_total{instance="34.75.224.64:2111"}: expected <=200 errors, found from 0.000000, to 464.000000
		(1) secondary error attachment
		  | error at from 2022-06-04T13:14:08Z, to 2022-06-04T13:18:48Z on metric workload_tpcc_newOrder_error_total{instance="34.75.224.64:2122"}: expected <=200 errors, found from 0.000000, to 891.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-04T13:14:08Z, to 2022-06-04T13:18:48Z on metric workload_tpcc_newOrder_error_total{instance="34.75.224.64:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 891.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-06-04T13:14:08Z, to 2022-06-04T13:18:48Z on metric workload_tpcc_newOrder_error_total{instance="34.75.224.64:2111"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 0.000000, to 464.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker sync-me sync-me-5]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-06T14:36:36Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 380f15c04826b432273cbd7ffe96b335ffc90396:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-06T13:00:34Z, to 2022-06-06T13:05:14Z on metric workload_tpcc_newOrder_error_total{instance="35.196.158.89:2111"}: expected <=200 errors, found from 0.000000, to 863.000000
		(1) secondary error attachment
		  | error at from 2022-06-06T13:00:34Z, to 2022-06-06T13:05:14Z on metric workload_tpcc_payment_error_total{instance="35.196.158.89:2111"}: expected <=200 errors, found from 0.000000, to 755.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-06T13:00:34Z, to 2022-06-06T13:05:14Z on metric workload_tpcc_payment_error_total{instance="35.196.158.89:2111"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 755.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-06-06T13:00:34Z, to 2022-06-06T13:05:14Z on metric workload_tpcc_newOrder_error_total{instance="35.196.158.89:2111"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 0.000000, to 863.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker sync-me sync-me-5]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-07T14:47:09Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ af4263069438f09c68f3eaa661ba80323fc22e74:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-07T13:10:56Z, to 2022-06-07T13:15:36Z on metric workload_tpcc_newOrder_error_total{instance="34.138.26.248:2111"}: expected <=200 errors, found from 0.000000, to 1220.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-06-07T13:10:56Z, to 2022-06-07T13:15:36Z on metric workload_tpcc_newOrder_error_total{instance="34.138.26.248:2111"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 0.000000, to 1220.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker sync-me sync-me-5]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-12T14:50:02Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 3181b7faa2f3b41d6a15ab4b74d2c60bcfe5132d:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-12T13:14:11Z, to 2022-06-12T13:18:51Z on metric workload_tpcc_newOrder_error_total{instance="34.75.245.30:2122"}: expected <=200 errors, found from 19.000000, to 314.000000
		(1) secondary error attachment
		  | error at from 2022-06-12T13:14:11Z, to 2022-06-12T13:18:51Z on metric workload_tpcc_payment_error_total{instance="34.75.245.30:2122"}: expected <=200 errors, found from 0.000000, to 283.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-12T13:14:11Z, to 2022-06-12T13:18:51Z on metric workload_tpcc_payment_error_total{instance="34.75.245.30:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 283.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-06-12T13:14:11Z, to 2022-06-12T13:18:51Z on metric workload_tpcc_newOrder_error_total{instance="34.75.245.30:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 19.000000, to 314.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker sync-me sync-me-5]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-13T14:42:17Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 3181b7faa2f3b41d6a15ab4b74d2c60bcfe5132d:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-13T13:07:36Z, to 2022-06-13T13:12:16Z on metric workload_tpcc_newOrder_error_total{instance="34.138.223.38:2122"}: expected <=200 errors, found from 40.000000, to 802.000000
		(1) secondary error attachment
		  | error at from 2022-06-13T13:07:36Z, to 2022-06-13T13:12:16Z on metric workload_tpcc_payment_error_total{instance="34.138.223.38:2122"}: expected <=200 errors, found from 0.000000, to 707.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-13T13:07:36Z, to 2022-06-13T13:12:16Z on metric workload_tpcc_payment_error_total{instance="34.138.223.38:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 707.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-06-13T13:07:36Z, to 2022-06-13T13:12:16Z on metric workload_tpcc_newOrder_error_total{instance="34.138.223.38:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 40.000000, to 802.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker sync-me sync-me-5]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-06-15T14:12:59Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ fc456a26830067b6dfbb8cd87e093a28d9b833d1:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-06-15T12:36:27Z, to 2022-06-15T12:41:07Z on metric workload_tpcc_newOrder_error_total{instance="34.73.218.144:2122"}: expected <=200 errors, found from 16.000000, to 984.000000
		(1) secondary error attachment
		  | error at from 2022-06-15T12:36:27Z, to 2022-06-15T12:41:07Z on metric workload_tpcc_payment_error_total{instance="34.73.218.144:2122"}: expected <=200 errors, found from 0.000000, to 834.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-06-15T12:36:27Z, to 2022-06-15T12:41:07Z on metric workload_tpcc_payment_error_total{instance="34.73.218.144:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 834.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-06-15T12:36:27Z, to 2022-06-15T12:41:07Z on metric workload_tpcc_newOrder_error_total{instance="34.73.218.144:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 16.000000, to 984.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-21.2 release-blocker sync-me sync-me-5]
roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-release-22.1.0]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-12T14:31:48Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 3de0153d6b39a9637bd4720bd1bc8f648c8475b4:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1254.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) secondary error attachment
		  | error at from 2022-07-12T12:55:29Z, to 2022-07-12T13:00:09Z on metric workload_tpcc_newOrder_error_total{instance="35.227.126.107:2121"}: expected <=200 errors, found from 0.000000, to 1447.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-07-12T12:55:29Z, to 2022-07-12T13:00:09Z on metric workload_tpcc_newOrder_error_total{instance="35.227.126.107:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1447.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (7) error at from 2022-07-12T12:55:29Z, to 2022-07-12T13:00:09Z on metric workload_tpcc_newOrder_error_total{instance="35.227.126.107:2111"}
		Wraps: (8) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (9) expected <=200 errors, found from 0.000000, to 1178.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *withstack.withStack (7) *errutil.withPrefix (8) *withstack.withStack (9) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-13T14:45:51Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 248de09e56f0ea639b73e64a33d6bcb9bb0b9cc8:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 420136.000000, to 420848.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-07-13T13:40:01Z, to 2022-07-13T13:44:41Z on metric workload_tpcc_newOrder_error_total{instance="35.196.76.26:2112"}: expected <=200 errors, found from 314606.000000, to 315424.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-07-13T13:40:01Z, to 2022-07-13T13:44:41Z on metric workload_tpcc_newOrder_error_total{instance="35.196.76.26:2112"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 314606.000000, to 315424.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-07-13T13:40:01Z, to 2022-07-13T13:44:41Z on metric workload_tpcc_newOrder_error_total{instance="35.196.76.26:2121"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 420192.000000, to 420962.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-15T14:43:21Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 7c33e5e63c0972c91805c059f394492e11d67c2a:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 852.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) secondary error attachment
		  | error at from 2022-07-15T13:07:26Z, to 2022-07-15T13:12:06Z on metric workload_tpcc_newOrder_error_total{instance="35.227.21.210:2121"}: expected <=200 errors, found from 0.000000, to 751.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-07-15T13:07:26Z, to 2022-07-15T13:12:06Z on metric workload_tpcc_newOrder_error_total{instance="35.227.21.210:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 751.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (7) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (8) error at from 2022-07-15T13:07:26Z, to 2022-07-15T13:12:06Z on metric workload_tpcc_newOrder_error_total{instance="35.227.21.210:2111"}
		Wraps: (9) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (10) expected <=200 errors, found from 1.000000, to 1328.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *secondary.withSecondaryError (7) *withstack.withStack (8) *errutil.withPrefix (9) *withstack.withStack (10) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-17T14:42:42Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 99a26550e5957764762c4d72fd28f4723e05fd2e:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-07-17T13:06:27Z, to 2022-07-17T13:11:07Z on metric workload_tpcc_newOrder_error_total{instance="34.139.109.27:2122"}: expected <=200 errors, found from 26.000000, to 718.000000
		(1) secondary error attachment
		  | error at from 2022-07-17T13:06:27Z, to 2022-07-17T13:11:07Z on metric workload_tpcc_payment_error_total{instance="34.139.109.27:2122"}: expected <=200 errors, found from 0.000000, to 599.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-07-17T13:06:27Z, to 2022-07-17T13:11:07Z on metric workload_tpcc_payment_error_total{instance="34.139.109.27:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 599.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-07-17T13:06:27Z, to 2022-07-17T13:11:07Z on metric workload_tpcc_newOrder_error_total{instance="34.139.109.27:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 26.000000, to 718.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-25T14:24:50Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ a30a663cbd9323d34d50f343dd038af64671e25f:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-07-25T12:49:51Z, to 2022-07-25T12:54:31Z on metric workload_tpcc_newOrder_error_total{instance="34.138.81.55:2122"}: expected <=200 errors, found from 19.000000, to 231.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-07-25T12:49:51Z, to 2022-07-25T12:54:31Z on metric workload_tpcc_newOrder_error_total{instance="34.138.81.55:2122"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 19.000000, to 231.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-07-31T21:11:47Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 60293f9fdb49de3c0e0a7ad4bf49f92cec05b1a5:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1853.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) secondary error attachment
		  | error at from 2022-07-31T19:35:30Z, to 2022-07-31T19:40:10Z on metric workload_tpcc_newOrder_error_total{instance="35.231.0.154:2121"}: expected <=200 errors, found from 0.000000, to 1572.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-07-31T19:35:30Z, to 2022-07-31T19:40:10Z on metric workload_tpcc_newOrder_error_total{instance="35.231.0.154:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1572.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (7) error at from 2022-07-31T19:35:30Z, to 2022-07-31T19:40:10Z on metric workload_tpcc_newOrder_error_total{instance="35.231.0.154:2111"}
		Wraps: (8) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (9) expected <=200 errors, found from 0.000000, to 1975.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *withstack.withStack (7) *errutil.withPrefix (8) *withstack.withStack (9) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-02T14:04:22Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ c7b53317f766d7bbdc24ea78f73c49cc21dbfa63:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-08-02T12:27:18Z, to 2022-08-02T12:31:58Z on metric workload_tpcc_newOrder_error_total{instance="34.138.233.8:2122"}: expected <=200 errors, found from 8.000000, to 470.000000
		(1) secondary error attachment
		  | error at from 2022-08-02T12:27:18Z, to 2022-08-02T12:31:58Z on metric workload_tpcc_payment_error_total{instance="34.138.233.8:2122"}: expected <=200 errors, found from 0.000000, to 411.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-02T12:27:18Z, to 2022-08-02T12:31:58Z on metric workload_tpcc_payment_error_total{instance="34.138.233.8:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 411.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-08-02T12:27:18Z, to 2022-08-02T12:31:58Z on metric workload_tpcc_newOrder_error_total{instance="34.138.233.8:2122"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 8.000000, to 470.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-04T13:59:42Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ b924e9dd007645c7f0a258dca7771739d97c2fe7:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-08-04T12:53:34Z, to 2022-08-04T12:58:14Z on metric workload_tpcc_newOrder_error_total{instance="34.74.199.84:2121"}: expected <=200 errors, found from 404986.000000, to 406392.000000
		(1) secondary error attachment
		  | error at from 2022-08-04T12:53:34Z, to 2022-08-04T12:58:14Z on metric workload_tpcc_newOrder_error_total{instance="34.74.199.84:2112"}: expected <=200 errors, found from 315339.000000, to 316844.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-04T12:53:34Z, to 2022-08-04T12:58:14Z on metric workload_tpcc_newOrder_error_total{instance="34.74.199.84:2112"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 315339.000000, to 316844.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (3) error at from 2022-08-04T12:53:34Z, to 2022-08-04T12:58:14Z on metric workload_tpcc_newOrder_error_total{instance="34.74.199.84:2121"}
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (5) expected <=200 errors, found from 404986.000000, to 406392.000000
		Error types: (1) *secondary.withSecondaryError (2) *withstack.withStack (3) *errutil.withPrefix (4) *withstack.withStack (5) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-09T14:04:44Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 275457fc4d237dd6ed731a644f02ab31ae275045:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 743.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-08-09T12:28:58Z, to 2022-08-09T12:33:38Z on metric workload_tpcc_newOrder_error_total{instance="35.243.176.223:2122"}: expected <=200 errors, found from 0.000000, to 1252.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-09T12:28:58Z, to 2022-08-09T12:33:38Z on metric workload_tpcc_newOrder_error_total{instance="35.243.176.223:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1252.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-08-09T12:28:58Z, to 2022-08-09T12:33:38Z on metric workload_tpcc_newOrder_error_total{instance="35.243.176.223:2111"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 0.000000, to 747.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-11T14:23:14Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ be21fa95793fd7b37fd8bc05115c6fb2978271c5:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1684.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-08-11T12:46:19Z, to 2022-08-11T12:50:59Z on metric workload_tpcc_newOrder_error_total{instance="35.185.30.251:2112"}: expected <=200 errors, found from 0.000000, to 655.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-11T12:46:19Z, to 2022-08-11T12:50:59Z on metric workload_tpcc_newOrder_error_total{instance="35.185.30.251:2112"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 655.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-08-11T12:46:19Z, to 2022-08-11T12:50:59Z on metric workload_tpcc_newOrder_error_total{instance="35.185.30.251:2121"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 0.000000, to 548.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-12T15:19:36Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 4da42a4fad47ca6646a77f81bd4daddadbeb2b27:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 566.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) secondary error attachment
		  | error at from 2022-08-12T13:43:13Z, to 2022-08-12T13:47:53Z on metric workload_tpcc_newOrder_error_total{instance="34.148.18.230:2121"}: expected <=200 errors, found from 0.000000, to 518.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-12T13:43:13Z, to 2022-08-12T13:47:53Z on metric workload_tpcc_newOrder_error_total{instance="34.148.18.230:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 518.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (7) error at from 2022-08-12T13:43:13Z, to 2022-08-12T13:47:53Z on metric workload_tpcc_newOrder_error_total{instance="34.148.18.230:2111"}
		Wraps: (8) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (9) expected <=200 errors, found from 0.000000, to 1531.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *withstack.withStack (7) *errutil.withPrefix (8) *withstack.withStack (9) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-13T14:20:55Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 00a2f32a01b0997517e57cabb02993862ac6ec51:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 505.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) secondary error attachment
		  | error at from 2022-08-13T12:44:48Z, to 2022-08-13T12:49:28Z on metric workload_tpcc_newOrder_error_total{instance="35.196.158.89:2121"}: expected <=200 errors, found from 0.000000, to 298.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-13T12:44:48Z, to 2022-08-13T12:49:28Z on metric workload_tpcc_newOrder_error_total{instance="35.196.158.89:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 298.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (4) error at from 2022-08-13T12:44:48Z, to 2022-08-13T12:49:28Z on metric workload_tpcc_newOrder_error_total{instance="35.196.158.89:2111"}
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) expected <=200 errors, found from 0.000000, to 1030.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-15T14:31:42Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 4ec3e89f4065e78285afbd5261e36ffc6b551274:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-08-15T12:55:45Z, to 2022-08-15T13:00:25Z on metric workload_tpcc_newOrder_error_total{instance="34.75.72.235:2122"}: expected <=200 errors, found from 0.000000, to 245.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-08-15T12:55:45Z, to 2022-08-15T13:00:25Z on metric workload_tpcc_newOrder_error_total{instance="34.75.72.235:2122"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 0.000000, to 245.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-19T14:44:17Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 08bc326ba8e877a0e62fe4b3cbd4320882aecda3:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 818.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (9) secondary error attachment
		  | error at from 2022-08-19T13:08:03Z, to 2022-08-19T13:12:43Z on metric workload_tpcc_newOrder_error_total{instance="34.139.4.8:2121"}: expected <=200 errors, found from 0.000000, to 794.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-19T13:08:03Z, to 2022-08-19T13:12:43Z on metric workload_tpcc_newOrder_error_total{instance="34.139.4.8:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 794.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (10) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (11) error at from 2022-08-19T13:08:03Z, to 2022-08-19T13:12:43Z on metric workload_tpcc_newOrder_error_total{instance="34.139.4.8:2111"}
		Wraps: (12) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (13) expected <=200 errors, found from 0.000000, to 2995.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *secondary.withSecondaryError (7) *secondary.withSecondaryError (8) *secondary.withSecondaryError (9) *secondary.withSecondaryError (10) *withstack.withStack (11) *errutil.withPrefix (12) *withstack.withStack (13) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-20T14:18:11Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 1985775e222793428fbadf4dd3e2e4aba89ad6e7:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 52.000000, to 303.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (2) secondary error attachment
		  | error at from 2022-08-20T12:42:17Z, to 2022-08-20T12:46:57Z on metric workload_tpcc_newOrder_error_total{instance="35.196.46.116:2121"}: expected <=200 errors, found from 42.000000, to 297.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-20T12:42:17Z, to 2022-08-20T12:46:57Z on metric workload_tpcc_newOrder_error_total{instance="35.196.46.116:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 42.000000, to 297.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (4) error at from 2022-08-20T12:42:17Z, to 2022-08-20T12:46:57Z on metric workload_tpcc_newOrder_error_total{instance="35.196.46.116:2111"}
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (6) expected <=200 errors, found from 35.000000, to 246.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-21T14:36:40Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 1985775e222793428fbadf4dd3e2e4aba89ad6e7:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-08-21T13:02:34Z, to 2022-08-21T13:07:14Z on metric workload_tpcc_newOrder_error_total{instance="35.185.11.97:2111"}: expected <=200 errors, found from 49.000000, to 304.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-08-21T13:02:34Z, to 2022-08-21T13:07:14Z on metric workload_tpcc_newOrder_error_total{instance="35.185.11.97:2111"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 49.000000, to 304.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-23T14:13:25Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 714fa0ad80c499cbd96ba97c560a9b414c61104f:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 301.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) secondary error attachment
		  | error at from 2022-08-23T12:38:53Z, to 2022-08-23T12:43:33Z on metric workload_tpcc_newOrder_error_total{instance="35.227.86.161:2121"}: expected <=200 errors, found from 0.000000, to 413.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-23T12:38:53Z, to 2022-08-23T12:43:33Z on metric workload_tpcc_newOrder_error_total{instance="35.227.86.161:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 413.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (7) error at from 2022-08-23T12:38:53Z, to 2022-08-23T12:43:33Z on metric workload_tpcc_newOrder_error_total{instance="35.227.86.161:2111"}
		Wraps: (8) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (9) expected <=200 errors, found from 0.000000, to 470.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *withstack.withStack (7) *errutil.withPrefix (8) *withstack.withStack (9) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-24T14:47:14Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ 5b20043919860dbf896247ed1bcf9480972c7998:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1087.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-08-24T13:10:33Z, to 2022-08-24T13:15:13Z on metric workload_tpcc_newOrder_error_total{instance="34.148.146.165:2122"}: expected <=200 errors, found from 20.000000, to 1220.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-24T13:10:33Z, to 2022-08-24T13:15:13Z on metric workload_tpcc_newOrder_error_total{instance="34.148.146.165:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 20.000000, to 1220.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-08-24T13:10:33Z, to 2022-08-24T13:15:13Z on metric workload_tpcc_newOrder_error_total{instance="34.148.146.165:2111"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 8.000000, to 1118.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-28T14:18:22Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ fd04c2f3d7675d5a4a5a6c7d0c53cc502b019529:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 320190.000000, to 321145.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (6) secondary error attachment
		  | error at from 2022-08-28T13:12:15Z, to 2022-08-28T13:16:55Z on metric workload_tpcc_newOrder_error_total{instance="34.74.142.43:2121"}: expected <=200 errors, found from 411055.000000, to 411970.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-08-28T13:12:15Z, to 2022-08-28T13:16:55Z on metric workload_tpcc_newOrder_error_total{instance="34.74.142.43:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 411055.000000, to 411970.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (7) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (8) error at from 2022-08-28T13:12:15Z, to 2022-08-28T13:16:55Z on metric workload_tpcc_newOrder_error_total{instance="34.74.142.43:2111"}
		Wraps: (9) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (10) expected <=200 errors, found from 403463.000000, to 405453.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *secondary.withSecondaryError (6) *secondary.withSecondaryError (7) *withstack.withStack (8) *errutil.withPrefix (9) *withstack.withStack (10) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-29T14:22:52Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ d6511dce69b90c2160ad9e62090e22ed47cee420:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-08-29T13:46:36Z, to 2022-08-29T13:51:16Z on metric workload_tpcc_newOrder_error_total{instance="35.231.78.145:2111"}: expected <=200 errors, found from 813577.000000, to 815238.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-08-29T13:46:36Z, to 2022-08-29T13:51:16Z on metric workload_tpcc_newOrder_error_total{instance="35.231.78.145:2111"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 813577.000000, to 815238.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-08-30T14:36:08Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ ff9b1bf2604b343c886667e3961a08814de316b5:

The test failed on branch=release-22.1, cloud=gce:
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1
	tpcc.go:267,tpcc.go:582,test_runner.go:883: error at from 2022-08-30T12:59:19Z, to 2022-08-30T13:03:59Z on metric workload_tpcc_newOrder_error_total{instance="34.75.1.14:2122"}: expected <=200 errors, found from 31.000000, to 698.000000
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (2) error at from 2022-08-30T12:59:19Z, to 2022-08-30T13:03:59Z on metric workload_tpcc_newOrder_error_total{instance="34.75.1.14:2122"}
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (4) expected <=200 errors, found from 31.000000, to 698.000000
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-02T14:09:25Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ a6c3138c2ab50b466ae1cc40409aa6a3b403129d:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 512.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-09-02T12:33:26Z, to 2022-09-02T12:38:06Z on metric workload_tpcc_newOrder_error_total{instance="35.237.138.165:2121"}: expected <=200 errors, found from 0.000000, to 492.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-09-02T12:33:26Z, to 2022-09-02T12:38:06Z on metric workload_tpcc_newOrder_error_total{instance="35.237.138.165:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 492.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-09-02T12:33:26Z, to 2022-09-02T12:38:06Z on metric workload_tpcc_newOrder_error_total{instance="35.237.138.165:2111"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 0.000000, to 996.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-06T14:32:35Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ a70cc0eed145522afbf1e9f0307fb5c554562050:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 353.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) secondary error attachment
		  | error at from 2022-09-06T12:56:12Z, to 2022-09-06T13:00:52Z on metric workload_tpcc_newOrder_error_total{instance="34.74.70.131:2121"}: expected <=200 errors, found from 0.000000, to 279.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-09-06T12:56:12Z, to 2022-09-06T13:00:52Z on metric workload_tpcc_newOrder_error_total{instance="34.74.70.131:2121"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 279.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (6) error at from 2022-09-06T12:56:12Z, to 2022-09-06T13:00:52Z on metric workload_tpcc_newOrder_error_total{instance="34.74.70.131:2111"}
		Wraps: (7) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (8) expected <=200 errors, found from 0.000000, to 2157.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *withstack.withStack (6) *errutil.withPrefix (7) *withstack.withStack (8) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot T-kv branch-master release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-09-07T14:17:55Z

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on release-22.1 @ a70cc0eed145522afbf1e9f0307fb5c554562050:

		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 202.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (3) secondary error attachment
		  | error at from 2022-09-07T12:41:26Z, to 2022-09-07T12:46:06Z on metric workload_tpcc_newOrder_error_total{instance="35.196.82.240:2122"}: expected <=200 errors, found from 0.000000, to 1648.000000
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  |   | [...repeated from below...]
		  | Wraps: (2) error at from 2022-09-07T12:41:26Z, to 2022-09-07T12:46:06Z on metric workload_tpcc_newOrder_error_total{instance="35.196.82.240:2122"}
		  | Wraps: (3) attached stack trace
		  |   -- stack trace:
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  |   | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  |   | runtime.goexit
		  |   | 	GOROOT/src/runtime/asm_amd64.s:1581
		  | Wraps: (4) expected <=200 errors, found from 0.000000, to 1648.000000
		  | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError
		Wraps: (4) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:202
		  | [...repeated from below...]
		Wraps: (5) error at from 2022-09-07T12:41:26Z, to 2022-09-07T12:46:06Z on metric workload_tpcc_newOrder_error_total{instance="35.196.82.240:2112"}
		Wraps: (6) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkMetrics
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:201
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).checkUptime
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:51
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*tpccChaosEventProcessor).listen.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/drt.go:277
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (7) expected <=200 errors, found from 0.000000, to 205.000000
		Error types: (1) *secondary.withSecondaryError (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *withstack.withStack (7) *errutil.leafError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711 roachtest: tpcc/multiregion/survive=region/chaos=true failed [C-test-failure O-roachtest O-robot T-kv blocks-22.2.0-beta.1 branch-master release-blocker]

_{This test on roachdash | Improve this report!}

irfansharif · 2022-09-07T22:38:41Z

Think we just want to backport #80526.

irfansharif · 2022-09-08T00:18:24Z

#87543.

cockroach-teamcity mentioned this issue Mar 29, 2022

roachtest: tpcc/multiregion/survive=region/chaos=true failed #74742

Closed

otan removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 31, 2022

cockroach-teamcity mentioned this issue Apr 22, 2022

roachtest: tpcc/multiregion/survive=region/chaos=true failed #80381

Closed

jlinder added the sync-me-3 label May 24, 2022

jlinder removed the sync-me-3 label May 27, 2022

cockroach-teamcity mentioned this issue May 31, 2022

roachtest: tpcc/multiregion/survive=region/chaos=true failed #82126

Closed

cockroach-teamcity mentioned this issue Aug 7, 2022

roachtest: tpcc/multiregion/survive=region/chaos=true failed #85711

Closed

irfansharif self-assigned this Sep 7, 2022

irfansharif closed this as completed Sep 8, 2022

irfansharif mentioned this issue Sep 9, 2022

roachtest: tpcc/multiregion/survive=region/chaos=true failed #87595

Closed

roachtest: tpcc/multiregion/survive=region/chaos=true failed #78619

roachtest: tpcc/multiregion/survive=region/chaos=true failed #78619

Comments

cockroach-teamcity commented Mar 28, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Apr 13, 2022

cockroach-teamcity commented Apr 22, 2022

cockroach-teamcity commented Apr 24, 2022

cockroach-teamcity commented Apr 27, 2022

cockroach-teamcity commented May 17, 2022

cockroach-teamcity commented May 20, 2022

cockroach-teamcity commented May 22, 2022

cockroach-teamcity commented May 23, 2022

cockroach-teamcity commented May 24, 2022

cockroach-teamcity commented May 27, 2022

cockroach-teamcity commented May 31, 2022

cockroach-teamcity commented Jun 1, 2022

cockroach-teamcity commented Jun 2, 2022

cockroach-teamcity commented Jun 4, 2022

cockroach-teamcity commented Jun 6, 2022

cockroach-teamcity commented Jun 7, 2022

cockroach-teamcity commented Jun 12, 2022

cockroach-teamcity commented Jun 13, 2022

cockroach-teamcity commented Jun 15, 2022

cockroach-teamcity commented Jul 12, 2022

cockroach-teamcity commented Jul 13, 2022

cockroach-teamcity commented Jul 15, 2022

cockroach-teamcity commented Jul 17, 2022

cockroach-teamcity commented Jul 25, 2022

cockroach-teamcity commented Jul 31, 2022

cockroach-teamcity commented Aug 2, 2022

cockroach-teamcity commented Aug 4, 2022

cockroach-teamcity commented Aug 9, 2022

cockroach-teamcity commented Aug 11, 2022

cockroach-teamcity commented Aug 12, 2022

cockroach-teamcity commented Aug 13, 2022

cockroach-teamcity commented Aug 15, 2022

cockroach-teamcity commented Aug 19, 2022

cockroach-teamcity commented Aug 20, 2022

cockroach-teamcity commented Aug 21, 2022

cockroach-teamcity commented Aug 23, 2022

cockroach-teamcity commented Aug 24, 2022

cockroach-teamcity commented Aug 28, 2022

cockroach-teamcity commented Aug 29, 2022

cockroach-teamcity commented Aug 30, 2022

cockroach-teamcity commented Sep 2, 2022

cockroach-teamcity commented Sep 6, 2022

cockroach-teamcity commented Sep 7, 2022

irfansharif commented Sep 7, 2022

irfansharif commented Sep 8, 2022

cockroach-teamcity commented Mar 28, 2022 •

edited by cockroach-jira-scripts

Loading