roachtest: tpcc/mixed-headroom/n5cpu16 failed #52693

cockroach-teamcity · 2020-08-12T15:53:38Z

cockroach-teamcity · 2020-08-19T11:55:55Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@ebd5c732f83009acc9c6f5859ca95e74e5453a1c:


	cluster.go:1612,context.go:135,cluster.go:1601,test_runner.go:823: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2195516-1597818221-76-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: dead
		4: 4721
		1: dead
		2: 4419
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 1: dead
		  | (1) attached stack trace
		  |   | main.glob..func13
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (2) 3 safe details enclosed
		  | Wraps: (3) 1: dead
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errors.errorString
		Wraps: (3) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (4) 3 safe details enclosed
		Wraps: (5) 3: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *safedetails.withSafeDetails (5) *errors.errorString

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #52186 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-22T13:45:45Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@69ffd78d5bbab0d8f77cf1f2254e8a5fcbdf902f:

		  |  2029.0s        0           19.0           19.0     30.4     54.5     75.5     75.5 stockLevel
		  |  2030.0s        0           24.0           19.0     50.3     58.7     67.1     67.1 delivery
		  |  2030.0s        0          185.1          189.0     29.4     46.1     54.5     75.5 newOrder
		  |  2030.0s        0           16.0           19.0      6.3      8.9      9.4      9.4 orderStatus
		  |  2030.0s        0          205.1          190.4     16.8     27.3     39.8     44.0 payment
		  |  2030.0s        0           16.0           19.0     25.2     50.3     58.7     58.7 stockLevel
		  |  2031.0s        0           15.0           19.0     52.4     71.3     75.5     75.5 delivery
		  |  2031.0s        0          197.7          189.0     28.3     54.5     65.0     71.3 newOrder
		  |  2031.0s        0           17.0           19.0      7.3     10.5     11.5     11.5 orderStatus
		  |  2031.0s        0          206.7          190.4     15.2     23.1     39.8     41.9 payment
		  |  2031.0s        0           20.0           19.0     25.2     39.8     46.1     46.1 stockLevel
		  |  2032.0s        0           16.0           19.0     48.2     58.7     58.7     58.7 delivery
		  |  2032.0s        0          173.2          189.0     29.4     41.9     65.0     83.9 newOrder
		  |  2032.0s        0           22.0           19.0      6.6     10.0     11.0     11.0 orderStatus
		  |  2032.0s        0          197.3          190.4     16.3     25.2     44.0     46.1 payment
		  |  2032.0s        0           23.0           19.0     25.2     58.7     75.5     75.5 stockLevel
		Wraps: (5) exit status 20
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:2597,tpcc.go:187,tpcc.go:286,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2585
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2593
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:286
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) 1 safe detail enclosed
		Wraps: (3) monitor failure
		Wraps: (4) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2641
		Wraps: (5) 1 safe detail enclosed
		Wraps: (6) monitor task failed
		Wraps: (7) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2555
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (8) 1 safe detail enclosed
		Wraps: (9) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *withstack.withStack (5) *safedetails.withSafeDetails (6) *errutil.withMessage (7) *withstack.withStack (8) *safedetails.withSafeDetails (9) *errors.errorString

More

Artifacts: /tpcc/mixed-headroom/n5cpu16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-08-24T21:03:41Z

Original failure should've been fixed by #53011 or #52750.

Error: error in newOrder: missing stock row

This looks concerning? Doesn't look like recent changes to workload tpcc explain this. Assigning to self to bring up in weekly review.

cockroach-teamcity · 2020-08-26T10:10:20Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@ef55609797c92e46eb9efd08facc9db49558291d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:2167,tpcc.go:147,tpcc.go:157,tpcc.go:168,tpcc.go:286,test_runner.go:754: output in run_101000.245_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2223072-1598421936-85-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2165
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:286
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_101000.245_n5_workload_fixtures_load_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2223072-1598421936-85-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | Error: finding fixture: fixture not found: workload/tpcc/version=2.1.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=1000
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=1000  {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-05T21:07:42Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@ee9935e7e7c00186b3571c83ec2067f1344c17d9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	test_runner.go:801: test timed out (10h0m0s)

	cluster.go:2206,tpcc.go:147,tpcc.go:160,tpcc.go:171,tpcc.go:289,test_runner.go:754: output in run_110846.084_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2249638-1599286091-75-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2284
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2204
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:160
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:171
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:289
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_110846.084_n5_workload_fixtures_load_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2249638-1599286091-75-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200905 11:08:47.988321 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200905 11:08:48.706867 155 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 711.611836ms (100000 rows, 0 index entries, 11 MiB)
		  | I200905 11:09:02.267277 150 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 14.272101794s (10000 rows, 0 index entries, 70 KiB)
		  | I200905 11:09:49.979028 154 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 1m1.983328506s (9000000 rows, 0 index entries, 2.0 MiB)
		  |
		  | stdout:
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #53535 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202008261913_v20.2.0-beta.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-07T12:01:22Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@8b9e8dc32e73cfdfc1999da35d61e5cc9a2b35ec:

		  |   352.0s        0           39.0           18.4  10200.5  14495.5  15569.3  15569.3 delivery
		  |   352.0s        0          198.0          173.4  11274.3  13958.6  16106.1  16643.0 newOrder
		  |   352.0s        0           25.0           18.2   9126.8  13958.6  15032.4  15032.4 orderStatus
		  |   352.0s        0          342.0          182.7  13421.8  16106.1  16106.1  16643.0 payment
		  |   352.0s        0           15.0           18.4   9126.8  10200.5  10737.4  10737.4 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   353.0s        0           14.0           18.4   9126.8  10737.4  15569.3  15569.3 delivery
		  |   353.0s        0          229.0          173.6  12884.9  17179.9  17179.9  17179.9 newOrder
		  |   353.0s        0           16.0           18.2   8589.9   9663.7   9663.7   9663.7 orderStatus
		  |   353.0s        0          280.0          182.9  10737.4  15569.3  16643.0  16643.0 payment
		  |   353.0s        0           14.0           18.4   9126.8  11811.2  13421.8  13421.8 stockLevel
		  |   354.0s        0           19.0           18.4   8053.1  14495.5  15032.4  15032.4 delivery
		  |   354.0s        0          473.5          174.4  11811.2  18253.6  18253.6  18253.6 newOrder
		  |   354.0s        0           22.0           18.2   7247.8   9663.7  10737.4  10737.4 orderStatus
		  |   354.0s        0          248.7          183.1  10200.5  16106.1  17179.9  17179.9 payment
		  |   354.0s        0           26.0           18.4   7516.2  12884.9  15569.3  15569.3 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2648,tpcc.go:190,tpcc.go:289,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:190
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:289
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #53535 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202008261913_v20.2.0-beta.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-11T13:49:41Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@83484491a490f659098eacee8058533ce5300e84:

		  |  6760.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 delivery
		  |  6760.0s        0            0.0          189.8      0.0      0.0      0.0      0.0 newOrder
		  |  6760.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 orderStatus
		  |  6760.0s        0            0.0          190.2      0.0      0.0      0.0      0.0 payment
		  |  6760.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  6761.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 delivery
		  |  6761.0s        0            0.0          189.7      0.0      0.0      0.0      0.0 newOrder
		  |  6761.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 orderStatus
		  |  6761.0s        0            0.0          190.1      0.0      0.0      0.0      0.0 payment
		  |  6761.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 stockLevel
		  |  6762.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 delivery
		  |  6762.0s        0            0.0          189.7      0.0      0.0      0.0      0.0 newOrder
		  |  6762.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 orderStatus
		  |  6762.0s        0            0.0          190.1      0.0      0.0      0.0      0.0 payment
		  |  6762.0s        0            0.0           19.0      0.0      0.0      0.0      0.0 stockLevel
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	cluster.go:2648,tpcc.go:190,tpcc.go:289,test_runner.go:754: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:190
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:289
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2606
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #53535 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-provisional_202008261913_v20.2.0-beta.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

andreimatei · 2020-09-11T14:20:04Z

last failure is an ambiguous error (intent missing and record aborted) which shouldn't happen in 20.2 any more

nvanbenschoten · 2020-09-22T15:01:55Z

last failure is an ambiguous error (intent missing and record aborted) which shouldn't happen in 20.2 any more

We've scared ourselves a few times on this. This is expected because this is a mixed version roachtest so we still expect to see this from time to time.

cockroach-teamcity · 2020-09-29T11:14:03Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@0c12d8b101a853f441926ad758c4c2882ecd33ee:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:2209,tpcc.go:126,tpcc.go:138,tpcc.go:149,tpcc.go:267,test_runner.go:755: output in run_111344.166_n5_workload_fixtures_import_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2328528-1601360230-83-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2207
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:126
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:267
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_111344.166_n5_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2328528-1601360230-83-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned
		  | stderr:
		  | I200929 11:13:45.625512 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | Error: importing fixture: importing table order: pq: expected tpcc version "2.2.0" but got "2.1.0"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #54926 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-29T11:49:57Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on release-20.2@bdb8cd0e7b2f25a08569a56464838486b6d16421:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:2209,tpcc.go:126,tpcc.go:138,tpcc.go:149,tpcc.go:267,test_runner.go:755: output in run_114932.706_n5_workload_fixtures_import_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2328570-1601361596-80-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2207
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:126
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:267
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_114932.706_n5_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2328570-1601361596-80-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned
		  | stderr:
		  | I200929 11:49:34.133883 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | Error: importing fixture: importing table district: pq: expected tpcc version "2.2.0" but got "2.1.0"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #54926 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-09-29T15:48:52Z

Recent failures are the same as #54137 (comment).

cockroach-teamcity · 2020-09-30T11:12:09Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on master@5a7f32fd41cb84d35562db09761912ff9c903700:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:2209,tpcc.go:126,tpcc.go:138,tpcc.go:149,tpcc.go:267,test_runner.go:755: output in run_111143.290_n5_workload_fixtures_import_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2332303-1601446302-82-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2207
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:126
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:267
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_111143.290_n5_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2332303-1601446302-82-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned
		  | stderr:
		  | I200930 11:11:44.718230 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | Error: importing fixture: importing table new_order: pq: expected tpcc version "2.2.0" but got "2.1.0"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #55041 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: tpcc/mixed-headroom/n5cpu16 failed #55023 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-30T11:23:45Z

(roachtest).tpcc/mixed-headroom/n5cpu16 failed on release-20.2@f180b8d178f7c81dbb73e7f4b139bd7b7bf829dc:

The test failed on branch=release-20.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpcc/mixed-headroom/n5cpu16/run_1
	cluster.go:2209,tpcc.go:126,tpcc.go:138,tpcc.go:149,tpcc.go:267,test_runner.go:755: output in run_112325.976_n5_workload_fixtures_import_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2332369-1601446347-82-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2207
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:126
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.registerTPCC.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:267
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_112325.976_n5_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2332369-1601446347-82-n5cpu16:5 -- ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1} returned
		  | stderr:
		  | I200930 11:23:27.400514 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | Error: importing fixture: importing table customer: pq: expected tpcc version "2.2.0" but got "2.1.0"
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures import tpcc --warehouses=909  --deprecated-fk-indexes {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /tpcc/mixed-headroom/n5cpu16
Related:

roachtest: tpcc/mixed-headroom/n5cpu16 failed #55041 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: tpcc/mixed-headroom/n5cpu16 failed #55023 roachtest: tpcc/mixed-headroom/n5cpu16 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

54743: kvcoord: properly redact 'have been waiting' message in DistSender r=knz a=tbg Release note: None 55050: roachtest: use crdb's workload for fixtures import tpcc r=nvanbenschoten a=nvanbenschoten Fixes #55042. Fixes #55041. Fixes #55039. Fixes #55038. Fixes #55037. Fixes #55036. Fixes #55035. Fixes #55033. Fixes #55029. Fixes #55024. Fixes #55022. Fixes #55020. Fixes #55019. Fixes #55018. Fixes #55017. Fixes #55016. Fixes #55013. Fixes #55010. Fixes #55009. Fixes #55008. Fixes #55003. Fixes #55002. Fixes #54998. Fixes #54995. Fixes #54822. Fixes #52693. Fixes #54802. We were already doing this in some places, but needed it in others. Co-authored-by: Tobias Grieger <[email protected]> Co-authored-by: Nathan VanBenschoten <[email protected]>

Fixes #55042. Fixes #55041. Fixes #55039. Fixes #55038. Fixes #55037. Fixes #55036. Fixes #55035. Fixes #55033. Fixes #55029. Fixes #55024. Fixes #55022. Fixes #55020. Fixes #55019. Fixes #55018. Fixes #55017. Fixes #55016. Fixes #55013. Fixes #55010. Fixes #55009. Fixes #55008. Fixes #55003. Fixes #55002. Fixes #54998. Fixes #54995. Fixes #54822. Fixes #52693. Fixes #54802. We were already doing this in some places, but needed it in others.

cockroach-teamcity added this to the 20.2 milestone Aug 12, 2020

This comment has been minimized.

Sign in to view

cockroach-teamcity mentioned this issue Aug 27, 2020

roachtest: tpcc/mixed-headroom/n5cpu16 failed #53535

Closed

tbg assigned andreimatei Aug 27, 2020

dt added the branch-release-20.2 label Sep 14, 2020

nvanbenschoten removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Sep 14, 2020

cockroach-teamcity mentioned this issue Sep 29, 2020

roachtest: tpcc/mixed-headroom/n5cpu16 failed #54926

Closed

This was referenced Sep 30, 2020

roachtest: tpcc/mixed-headroom/n5cpu16 failed #55023

Closed

roachtest: tpcc/mixed-headroom/n5cpu16 failed #55041

Closed

nvanbenschoten mentioned this issue Sep 30, 2020

roachtest: use crdb's workload for fixtures import tpcc #55050

Merged

craig bot closed this as completed in #55050 Sep 30, 2020

craig bot closed this as completed in 20fc710 Sep 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpcc/mixed-headroom/n5cpu16 failed #52693

roachtest: tpcc/mixed-headroom/n5cpu16 failed #52693

cockroach-teamcity commented Aug 12, 2020 •

edited by tbg

Loading

This comment has been minimized.

cockroach-teamcity commented Aug 19, 2020

cockroach-teamcity commented Aug 22, 2020

irfansharif commented Aug 24, 2020

cockroach-teamcity commented Aug 26, 2020

cockroach-teamcity commented Sep 5, 2020

cockroach-teamcity commented Sep 7, 2020

cockroach-teamcity commented Sep 11, 2020

andreimatei commented Sep 11, 2020

nvanbenschoten commented Sep 22, 2020

cockroach-teamcity commented Sep 29, 2020

cockroach-teamcity commented Sep 29, 2020

irfansharif commented Sep 29, 2020

cockroach-teamcity commented Sep 30, 2020

cockroach-teamcity commented Sep 30, 2020

roachtest: tpcc/mixed-headroom/n5cpu16 failed #52693

roachtest: tpcc/mixed-headroom/n5cpu16 failed #52693

Comments

cockroach-teamcity commented Aug 12, 2020 • edited by tbg Loading

This comment has been minimized.

cockroach-teamcity commented Aug 19, 2020

cockroach-teamcity commented Aug 22, 2020

irfansharif commented Aug 24, 2020

cockroach-teamcity commented Aug 26, 2020

cockroach-teamcity commented Sep 5, 2020

cockroach-teamcity commented Sep 7, 2020

cockroach-teamcity commented Sep 11, 2020

andreimatei commented Sep 11, 2020

nvanbenschoten commented Sep 22, 2020

cockroach-teamcity commented Sep 29, 2020

cockroach-teamcity commented Sep 29, 2020

irfansharif commented Sep 29, 2020

cockroach-teamcity commented Sep 30, 2020

cockroach-teamcity commented Sep 30, 2020

cockroach-teamcity commented Aug 12, 2020 •

edited by tbg

Loading