roachtest: scrub/index-only/tpcc/w=100 failed #44069

cockroach-teamcity · 2020-01-16T14:39:32Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1693614-1579160767-67-n5cpu4:5 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		    s        0           14.0           15.1     52.4    104.9    151.0    151.0 orderStatus
		       69.0s        0          103.0          149.7    570.4   3087.0   4563.4   6174.0 payment
		       69.0s        0           16.0           15.1    167.8   1140.9   1275.1   1275.1 stockLevel
		       70.0s        0           15.0           15.0    805.3   2415.9   2415.9   2415.9 delivery
		       70.0s        0          153.0          150.7    671.1   1409.3   1610.6   2147.5 newOrder
		       70.0s        0           11.0           15.1    104.9    151.0    243.3    243.3 orderStatus
		       70.0s        0          149.0          149.7    637.5   2281.7   4026.5   4160.7 payment
		       70.0s        0           15.0           15.1    142.6    335.5   1140.9   1140.9 stockLevel
		       71.0s        0           15.0           15.0    805.3   1275.1   1610.6   1610.6 delivery
		       71.0s        0          150.0          150.7    570.4   1208.0   1342.2   1476.4 newOrder
		       71.0s        0           15.0           15.1     56.6    176.2    192.9    192.9 orderStatus
		       71.0s        0          156.0          149.8    486.5   2013.3   3221.2   3355.4 payment
		       71.0s        0           13.0           15.1     75.5    167.8    176.2    176.2 stockLevel
		       72.0s        0           12.0           14.9    704.6   1677.7   2080.4   2080.4 delivery
		       72.0s        0          132.9          150.5    503.3   1073.7   1409.3   1610.6 newOrder
		       72.0s        0           20.0           15.1     48.2    192.9    243.3    243.3 orderStatus
		       72.0s        0          152.9          149.8    486.5   2281.7   3221.2   4563.4 payment
		       72.0s        0           14.0           15.1     54.5    176.2    285.2    285.2 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		       73.0s        0           14.0           14.9    671.1   2080.4   2147.5   2147.5 delivery
		       73.0s        0          144.0          150.4    604.0   1275.1   1409.3   1543.5 newOrder
		       73.0s        0           18.0           15.2     75.5    226.5    226.5    226.5 orderStatus
		       73.0s        0          155.0          149.9    520.1   2415.9   3221.2   4026.5 payment
		       73.0s        0           15.0           15.1    184.5    369.1    453.0    453.0 stockLevel
		    E200116 14:30:27.946952 1 workload/cli/run.go:444  error in payment: EOF
		       74.0s    14667            5.0           14.8    469.8   1140.9   1140.9   1140.9 delivery
		       74.0s    14667           17.0          148.6    302.0    973.1   1006.6   1006.6 newOrder
		       74.0s    14667            0.0           15.0      0.0      0.0      0.0      0.0 orderStatus
		       74.0s    14667           31.0          148.3    142.6   1208.0   5368.7   5368.7 payment
		       74.0s    14667            4.0           14.9     22.0     52.4     52.4     52.4 stockLevel
		    E200116 14:30:28.946869 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.109:26257: connect: connection refused
		       75.0s    19242            0.0           14.6      0.0      0.0      0.0      0.0 delivery
		       75.0s    19242            0.0          146.6      0.0      0.0      0.0      0.0 newOrder
		       75.0s    19242            0.0           14.8      0.0      0.0      0.0      0.0 orderStatus
		       75.0s    19242            0.0          146.3      0.0      0.0      0.0      0.0 payment
		       75.0s    19242            0.0           14.7      0.0      0.0      0.0      0.0 stockLevel
		    E200116 14:30:29.946946 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.109:26257: connect: connection refused
		       76.0s    28010            0.0           14.4      0.0      0.0      0.0      0.0 delivery
		       76.0s    28010            0.0          144.7      0.0      0.0      0.0      0.0 newOrder
		       76.0s    28010            0.0           14.6      0.0      0.0      0.0      0.0 orderStatus
		       76.0s    28010            0.0          144.4      0.0      0.0      0.0      0.0 payment
		       76.0s    28010            0.0           14.5      0.0      0.0      0.0      0.0 stockLevel
		    E200116 14:30:30.947023 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.109:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /scrub/index-only/tpcc/w=100

make stressrace TESTS=scrub/index-only/tpcc/w=100 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-02-13T13:05:27Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@2739821b911d777fa2a927295d699b559360a802:

		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_130137.635_n5_workload_fixtures_load_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.setupTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-98-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned:
		    stderr:
		    I200213 13:01:39.768317 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7
		    Error:  exit status 1
		    
		    stdout::
		  - exit status 1

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-04T00:53:21Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@5f9a71adb995837bcff27b9456188018434be4b8:

		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_145744.748_n5_workload_fixtures_load_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.setupTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1780216-1583223197-78-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned:
		    stderr:
		    214 46 ccl/workloadccl/fixture.go:586  loaded 99 KiB table district in 901.876673ms (1000 rows, 0 index entries, 110 KiB)
		    I200303 14:58:18.534667 98 ccl/workloadccl/fixture.go:586  loaded 11 MiB table new_order in 31.615563655s (900000 rows, 0 index entries, 361 KiB)
		    I200303 14:58:36.284229 49 ccl/workloadccl/fixture.go:586  loaded 126 MiB table order in 49.364901366s (3000000 rows, 3000000 index entries, 2.6 MiB)
		    I200303 14:58:57.344151 99 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 1m10.425017109s (100000 rows, 0 index entries, 113 KiB)
		    I200303 14:59:21.844690 48 ccl/workloadccl/fixture.go:586  loaded 421 MiB table history in 1m34.925612997s (3000000 rows, 6000000 index entries, 4.4 MiB)
		    I200303 14:59:57.588832 47 ccl/workloadccl/fixture.go:586  loaded 1.7 GiB table customer in 2m10.669384976s (3000000 rows, 3000000 index entries, 13 MiB)
		    I200303 15:00:44.169909 100 ccl/workloadccl/fixture.go:586  loaded 3.1 GiB table stock in 2m57.250718974s (10000000 rows, 10000000 index entries, 18 MiB)
		    
		    stdout::
		  - context canceled

	cluster.go:2050,cluster.go:2069,cluster.go:2173,cluster.go:1470,context.go:135,cluster.go:1467,test_runner.go:774: context canceled

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-05T00:15:26Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@d6a52a8f6f07b3b6b32af2243075e5365fb21c45:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200304-1783198/scrub/index-only/tpcc/w=100/run_1
	test_runner.go:756: test timed out (10h0m0s)

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-06T00:18:20Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@c9b189b3a2171c8da78c75fe80dd02933e18a5e5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200305-1786843/scrub/index-only/tpcc/w=100/run_1
	test_runner.go:756: test timed out (10h0m0s)

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-07T16:06:30Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@752dea867f3aeb142e98c22f8d320ce19041aa8d:

		      197.0s        0            6.0           11.9     33.6    285.2    285.2    285.2 orderStatus
		      197.0s        0           93.0          118.0    805.3   5368.7   8589.9  13958.6 payment
		      197.0s        0            7.0           11.8    285.2    872.4    872.4    872.4 stockLevel
		      198.0s        0            7.0           11.9    503.3   1040.2   1040.2   1040.2 delivery
		      198.0s        0           75.0          117.9    838.9   2550.1   3221.2   3355.4 newOrder
		      198.0s        0           13.0           11.9     52.4    151.0    243.3    243.3 orderStatus
		      198.0s        0           69.0          117.8    805.3   4563.4   9126.8  10200.5 payment
		      198.0s        0            7.0           11.8    419.4   1140.9   1140.9   1140.9 stockLevel
		      199.0s    14852            2.0           11.8    318.8    419.4    419.4    419.4 delivery
		      199.0s    14852            0.0          117.3      0.0      0.0      0.0      0.0 newOrder
		      199.0s    14852            0.0           11.9      0.0      0.0      0.0      0.0 orderStatus
		      199.0s    14852            1.0          117.2    704.6    704.6    704.6    704.6 payment
		      199.0s    14852            1.0           11.8   1409.3   1409.3   1409.3   1409.3 stockLevel
		      200.0s    28020            0.0           11.8      0.0      0.0      0.0      0.0 delivery
		      200.0s    28020            0.0          116.7      0.0      0.0      0.0      0.0 newOrder
		      200.0s    28020            0.0           11.8      0.0      0.0      0.0      0.0 orderStatus
		      200.0s    28020            0.0          116.6      0.0      0.0      0.0      0.0 payment
		      200.0s    28020            0.0           11.7      0.0      0.0      0.0      0.0 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      201.0s    40783            0.0           11.7      0.0      0.0      0.0      0.0 delivery
		      201.0s    40783            0.0          116.2      0.0      0.0      0.0      0.0 newOrder
		      201.0s    40783            0.0           11.7      0.0      0.0      0.0      0.0 orderStatus
		      201.0s    40783            0.0          116.0      0.0      0.0      0.0      0.0 payment
		      201.0s    40783            0.0           11.6      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

	cluster.go:2050,cluster.go:2069,cluster.go:2173,scrub.go:68,cluster.go:2344,errgroup.go:57: context canceled

	cluster.go:2368,tpcc.go:168,scrub.go:54,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 3: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1793930-1583568484-82-n5cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: dead
		1: 3994
		2: 3740
		4: 3806
		Error:  3: dead

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-08T15:22:27Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@dfa5bd527ae7d7373dd03c62118df87a87a77130:

		      190.0s        0           10.0           12.1     11.5    134.2    134.2    134.2 orderStatus
		      190.0s        0           86.0          120.4    318.8   2415.9   4831.8   6710.9 payment
		      190.0s        0            9.0           12.2    234.9    436.2    436.2    436.2 stockLevel
		      191.0s        0            7.0           12.3    159.4    302.0    302.0    302.0 delivery
		      191.0s        0           73.0          120.6    268.4   1040.2   1275.1   1275.1 newOrder
		      191.0s        0            7.0           12.1     15.7     60.8     60.8     60.8 orderStatus
		      191.0s        0           62.0          120.1    302.0   1744.8   2818.6   4026.5 payment
		      191.0s        0            8.0           12.2     88.1    604.0    604.0    604.0 stockLevel
		      192.0s    20534            0.0           12.2      0.0      0.0      0.0      0.0 delivery
		      192.0s    20534            0.0          119.9      0.0      0.0      0.0      0.0 newOrder
		      192.0s    20534            0.0           12.1      0.0      0.0      0.0      0.0 orderStatus
		      192.0s    20534            0.0          119.4      0.0      0.0      0.0      0.0 payment
		      192.0s    20534            0.0           12.1      0.0      0.0      0.0      0.0 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      193.0s    32174            0.0           12.2      0.0      0.0      0.0      0.0 delivery
		      193.0s    32174            0.0          119.3      0.0      0.0      0.0      0.0 newOrder
		      193.0s    32174            0.0           12.0      0.0      0.0      0.0      0.0 orderStatus
		      193.0s    32174            0.0          118.8      0.0      0.0      0.0      0.0 payment
		      193.0s    32174            0.0           12.1      0.0      0.0      0.0      0.0 stockLevel
		      194.0s    42968            0.0           12.1      0.0      0.0      0.0      0.0 delivery
		      194.0s    42968            0.0          118.7      0.0      0.0      0.0      0.0 newOrder
		      194.0s    42968            0.0           11.9      0.0      0.0      0.0      0.0 orderStatus
		      194.0s    42968            0.0          118.2      0.0      0.0      0.0      0.0 payment
		      194.0s    42968            0.0           12.0      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

	cluster.go:2050,cluster.go:2069,cluster.go:2173,scrub.go:68,cluster.go:2344,errgroup.go:57: context canceled

	cluster.go:2368,tpcc.go:168,scrub.go:54,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 3: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1795062-1583655253-72-n5cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: dead
		2: 11683
		1: 12419
		4: 8201
		Error:  3: dead

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-10T00:01:03Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200309-1796240/scrub/index-only/tpcc/w=100/run_1
	scrub.go:79,cluster.go:2344,errgroup.go:57: pq: server is not accepting clients

	cluster.go:2368,tpcc.go:168,scrub.go:54,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2416
		    main.(*monitor).wait.func4
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2445
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor command failure:
		  - signal: interrupt

	cluster.go:2050,cluster.go:2069,cluster.go:2173,cluster.go:1470,context.go:135,cluster.go:1467,test_runner.go:774: context canceled

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-10T23:54:49Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@72c4a1bd411f2f82bf9aaa22883821a946614148:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200310-1799071/scrub/index-only/tpcc/w=100/run_1
	scrub.go:79,cluster.go:2344,errgroup.go:57: pq: server is not accepting clients

	cluster.go:2368,tpcc.go:168,scrub.go:54,test_runner.go:747: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2416
		    main.(*monitor).wait.func4
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2445
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor command failure:
		  - signal: interrupt

	cluster.go:2050,cluster.go:2069,cluster.go:2173,cluster.go:1470,context.go:135,cluster.go:1467,test_runner.go:797: context canceled

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-17T00:30:53Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@d8bdc938177ea4b5226b95ee6cc51af0951054fc:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200316-1809758/scrub/index-only/tpcc/w=100/run_1
	scrub.go:79,cluster.go:2344,errgroup.go:57: pq: server is not accepting clients

	cluster.go:2368,tpcc.go:168,scrub.go:54,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2416
		    main.(*monitor).wait.func4
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2445
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor command failure:
		  - signal: interrupt

	cluster.go:2050,cluster.go:2069,cluster.go:2173,cluster.go:1470,context.go:135,cluster.go:1467,test_runner.go:803: context canceled

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-20T00:19:56Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@d571624ae4f833c49c717728a74cc7be78a791f0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200319-1816964/scrub/index-only/tpcc/w=100/run_1
	test_runner.go:800: test timed out (10h0m0s)

	scrub.go:79,cluster.go:2344,errgroup.go:57: dial tcp 34.71.89.109:26257: connect: connection refused

	cluster.go:2368,tpcc.go:168,scrub.go:54,test_runner.go:753: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.makeScrubTPCCTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2412
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.init
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2309
		    runtime.doInit
		    	/usr/local/go/src/runtime/proc.go:5222
		    runtime.main
		    	/usr/local/go/src/runtime/proc.go:190
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - Goexit() was called

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1816964-1584603715-84-n5cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: 5603
		3: 5450
		4: 6264
		1: dead
		Error:  1: dead

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

jordanlewis · 2020-03-20T15:41:07Z

Here's a weird SIGSEGV? No goroutine seems to have caused it - I don't understand really.

SIGSEGV: segmentation violation
PC=0x7cc9c3 m=0 sigcode=0

goroutine 50 [syscall]:
runtime.notetsleepg(0x74c16c0, 0x5f402ed, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00047a760 sp=0xc00047a730 pc=0x7761c4
runtime.timerproc(0x74c16a0)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc00047a7d8 sp=0xc00047a760 pc=0x7b96e1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00047a7e0 sp=0xc00047a7d8 pc=0x7caaa1
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

goroutine 1 [select, 599 minutes]:
runtime.gopark(0x44832a8, 0x0, 0x1809, 0x1)
	/usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0xc004e01408 sp=0xc004e013e8 pc=0x79c550
runtime.selectgo(0xc004e019f8, 0xc004e01680, 0x4, 0x75049d8, 0x4135100)
	/usr/local/go/src/runtime/select.go:313 +0xc9b fp=0xc004e01530 sp=0xc004e01408 pc=0x7ac10b
github.com/cockroachdb/cockroach/pkg/cli.runStart(0x6e20320, 0xc0005b1ef0, 0x0, 0x9, 0x42cbf00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:880 +0xc7d fp=0xc004e01aa8 sp=0xc004e01530 pc=0x319c14d
github.com/cockroachdb/cockroach/pkg/cli.runStartJoin(0x6e20320, 0xc0005b1ef0, 0x0, 0x9, 0xc000654400, 0xc000359ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:445 +0x4e fp=0xc004e01af0 sp=0xc004e01aa8 pc=0x319b49e
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0x6e20320, 0xc0005b1ef0, 0x0, 0x9, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:187 +0x79 fp=0xc004e01cc0 sp=0xc004e01af0 pc=0x31b1429
github.com/cockroachdb/cockroach/pkg/cli.maybeShoutError.func1(0x6e20320, 0xc0005b1ef0, 0x0, 0x9, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:351 +0x51 fp=0xc004e01d00 sp=0xc004e01cc0 pc=0x31b23f1
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute(0x6e20320, 0xc0005b1e60, 0x9, 0x9, 0x6e20320, 0xc0005b1e60)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:762 +0x460 fp=0xc004e01dd8 sp=0xc004e01d00 pc=0x2e9f750
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x6e18a00, 0xc00007a110, 0x42c7f5c, 0x6)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852 +0x2ea fp=0xc004e01ea8 sp=0xc004e01dd8 pc=0x2ea018a
github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
github.com/cockroachdb/cockroach/pkg/cli.Main()
	/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67 +0x26b fp=0xc004e01f50 sp=0xc004e01ea8 pc=0x315676b
main.main()
	/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach/main.go:26 +0x20 fp=0xc004e01f60 sp=0xc004e01f50 pc=0x3282c10
runtime.main()
	/usr/local/go/src/runtime/proc.go:203 +0x21e fp=0xc004e01fe0 sp=0xc004e01f60 pc=0x79c17e
runtime.goexit()

irfansharif · 2020-04-23T19:48:10Z

Saw the same in #47808.

cockroach-teamcity · 2020-04-30T14:13:30Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@3b612692db93aa7c87493705e1fad85c9c664f6c:

		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2440
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2488
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2385
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5222
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) Goexit() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

	cluster.go:1481,context.go:135,cluster.go:1470,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1909750-1588231526-85-n5cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: 3797
		1: dead
		4: dead
		2: dead
		Error: UNCLASSIFIED_PROBLEM: 1: dead, 4: dead, 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) 1: dead, 4: dead, 2: dead
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1129
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:272
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1793
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Error types: (1) errors.Unclassified (2) *errors.fundamental

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-02T00:33:33Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@3a03f3843a8cdf04f82c52753c61cf01b0d2ddcd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/scrub/index-only/tpcc/w=100/run_1
	test_runner.go:804: test timed out (10h0m0s)

	cluster.go:2040,tpcc.go:136,tpcc.go:138,tpcc.go:149,scrub.go:54,test_runner.go:757: output in run_143422.232_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2054002-1593584816-69-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_143422.232_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2054002-1593584816-69-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned
		  | stderr:
		  | I200701 14:34:24.166827 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200701 14:34:25.059929 76 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 877.367937ms (100000 rows, 0 index entries, 8.9 MiB)
		  | I200701 14:34:26.090978 75 ccl/workloadccl/fixture.go:586  loaded 11 MiB table new_order in 1.908438462s (900000 rows, 0 index entries, 5.8 MiB)
		  | I200701 14:34:59.291949 71 ccl/workloadccl/fixture.go:586  loaded 99 KiB table district in 35.109073038s (1000 rows, 0 index entries, 2.8 KiB)
		  | I200701 14:35:12.986107 74 ccl/workloadccl/fixture.go:586  loaded 126 MiB table order in 48.803503161s (3000000 rows, 3000000 index entries, 2.6 MiB)
		  | I200701 14:35:13.032698 70 ccl/workloadccl/fixture.go:586  loaded 5.1 KiB table warehouse in 48.850014891s (100 rows, 0 index entries, 107 B)
		  | I200701 14:35:27.733597 73 ccl/workloadccl/fixture.go:586  loaded 421 MiB table history in 1m3.551112059s (3000000 rows, 6000000 index entries, 6.6 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-04T21:31:45Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@456a07cfc1e53b87abc7709052e54efb1450e758:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/scrub/index-only/tpcc/w=100/run_1
	test_runner.go:804: test timed out (10h0m0s)

	cluster.go:2040,tpcc.go:170,scrub.go:54,test_runner.go:757: output in run_125737.220_n5_workload_check_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2062841-1593843322-19-n5cpu4:5 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_125737.220_n5_workload_check_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2062841-1593843322-19-n5cpu4:5 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-06T00:04:17Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@3e0de239121813ea4d47873388a2828a66d9edf7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/scrub/index-only/tpcc/w=100/run_1
	test_runner.go:804: test timed out (10h0m0s)

	cluster.go:2040,tpcc.go:136,tpcc.go:138,tpcc.go:149,scrub.go:54,test_runner.go:757: output in run_140507.038_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2063962-1593929497-71-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_140507.038_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2063962-1593929497-71-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned
		  | stderr:
		  | I200705 14:05:08.868479 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200705 14:05:09.257801 67 ccl/workloadccl/fixture.go:586  loaded 99 KiB table district in 379.053091ms (1000 rows, 0 index entries, 261 KiB)
		  | I200705 14:05:09.904458 72 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 1.025697262s (100000 rows, 0 index entries, 7.6 MiB)
		  | I200705 14:05:11.011904 71 ccl/workloadccl/fixture.go:586  loaded 11 MiB table new_order in 2.133173166s (900000 rows, 0 index entries, 5.2 MiB)
		  | I200705 14:05:12.011906 66 ccl/workloadccl/fixture.go:586  loaded 5.1 KiB table warehouse in 3.13294295s (100 rows, 0 index entries, 1.6 KiB)
		  | I200705 14:06:00.256850 69 ccl/workloadccl/fixture.go:586  loaded 421 MiB table history in 51.378138843s (3000000 rows, 6000000 index entries, 8.2 MiB)
		  | I200705 14:06:07.155478 70 ccl/workloadccl/fixture.go:586  loaded 126 MiB table order in 58.276590188s (3000000 rows, 3000000 index entries, 2.2 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /scrub/index-only/tpcc/w=100

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-26T10:06:14Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@ef55609797c92e46eb9efd08facc9db49558291d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/scrub/index-only/tpcc/w=100/run_1
	cluster.go:2167,tpcc.go:147,tpcc.go:157,tpcc.go:168,scrub.go:54,test_runner.go:754: output in run_100552.392_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2223072-1598421936-81-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2165
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_100552.392_n5_workload_fixtures_load_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2223072-1598421936-81-n5cpu4:5 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned
		  | stderr:
		  | Error: finding fixture: fixture not found: workload/tpcc/version=2.1.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=100
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=100  {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /scrub/index-only/tpcc/w=100
Related:

roachtest: scrub/index-only/tpcc/w=100 failed #51342 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: scrub/index-only/tpcc/w=100 failed #51157 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-provisional_202007071743_v20.2.0-alpha.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-13T12:04:26Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@03797b17417ae34451537f8f76d66ac69dba2d07:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:118
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:134
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:145
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_120236.929_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2578434-1610521418-75-n5cpu4:1 -- ./cockroach workload fixtures import tpcc --warehouses=100  returned
		  | stderr:
		  | I210113 12:02:37.698636 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210113 12:02:41.623653 70 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 3.916014231s, 1.99 MiB/s)
		  | I210113 12:02:42.827121 16 ccl/workloadccl/fixture.go:472  imported 99 KiB in district table (1000 rows, 0 index entries, took 5.120218434s, 0.02 MiB/s)
		  | I210113 12:02:44.309987 69 ccl/workloadccl/fixture.go:472  imported 11 MiB in new_order table (900000 rows, 0 index entries, took 6.602385878s, 1.69 MiB/s)
		  | I210113 12:02:55.741428 67 ccl/workloadccl/fixture.go:472  imported 216 MiB in history table (3000000 rows, 0 index entries, took 18.034261936s, 11.95 MiB/s)
		  | I210113 12:03:00.225791 15 ccl/workloadccl/fixture.go:472  imported 5.1 KiB in warehouse table (100 rows, 0 index entries, took 22.519063775s, 0.00 MiB/s)
		  | I210113 12:03:01.938756 68 ccl/workloadccl/fixture.go:472  imported 150 MiB in order table (3000000 rows, 3000000 index entries, took 24.231477083s, 6.20 MiB/s)
		  | I210113 12:03:30.532588 66 ccl/workloadccl/fixture.go:472  imported 1.7 GiB in customer table (3000000 rows, 3000000 index entries, took 52.825551748s, 33.24 MiB/s)
		  | I210113 12:03:33.327623 71 ccl/workloadccl/fixture.go:472  imported 3.0 GiB in stock table (10000000 rows, 0 index entries, took 55.619608315s, 55.00 MiB/s)
		  | I210113 12:03:47.242395 72 ccl/workloadccl/fixture.go:472  imported 1.6 GiB in order_line table (30005985 rows, 0 index entries, took 1m9.535736938s, 23.86 MiB/s)
		  | I210113 12:03:47.282843 1 ccl/workloadccl/fixture.go:351  imported 6.7 GiB bytes in 9 tables (took 1m9.583746731s, 98.57 MiB/s)
		  | I210113 12:03:48.376939 1 ccl/workloadccl/cliccl/fixtures.go:355  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I210113 12:03:48.405359 1 workload/tpcc/tpcc.go:389  check 3.3.2.1 took 28.348646ms
		  | I210113 12:03:50.072917 1 workload/tpcc/tpcc.go:389  check 3.3.2.2 took 1.667498241s
		  | I210113 12:03:50.480897 1 workload/tpcc/tpcc.go:389  check 3.3.2.3 took 407.915998ms
		  | I210113 12:04:02.290284 1 workload/tpcc/tpcc.go:389  check 3.3.2.4 took 11.80933049s
		  | I210113 12:04:05.120779 1 workload/tpcc/tpcc.go:389  check 3.3.2.5 took 2.830432107s
		  | Error: check failed: 3.3.2.5: right EXCEPT left returned nonzero results.
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=100
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /scrub/index-only/tpcc/w=100
Related:

roachtest: scrub/index-only/tpcc/w=100 failed #58832 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-14T12:00:20Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@f9b44f59d0451dc9efbb771baac3e88540485629:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2292
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2212
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:118
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:134
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:145
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_115823.106_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2582294-1610607747-78-n5cpu4:1 -- ./cockroach workload fixtures import tpcc --warehouses=100  returned
		  | stderr:
		  | I210114 11:58:23.883145 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210114 11:58:36.536309 14 ccl/workloadccl/fixture.go:472  imported 99 KiB in district table (1000 rows, 0 index entries, took 12.644683021s, 0.01 MiB/s)
		  | I210114 11:58:37.188854 20 ccl/workloadccl/fixture.go:472  imported 11 MiB in new_order table (900000 rows, 0 index entries, took 13.296350844s, 0.84 MiB/s)
		  | I210114 11:58:37.634851 13 ccl/workloadccl/fixture.go:472  imported 5.1 KiB in warehouse table (100 rows, 0 index entries, took 13.743225516s, 0.00 MiB/s)
		  | I210114 11:58:38.515521 21 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 14.622354185s, 0.53 MiB/s)
		  | I210114 11:58:46.803080 16 ccl/workloadccl/fixture.go:472  imported 216 MiB in history table (3000000 rows, 0 index entries, took 22.911190028s, 9.41 MiB/s)
		  | I210114 11:58:46.932052 19 ccl/workloadccl/fixture.go:472  imported 150 MiB in order table (3000000 rows, 3000000 index entries, took 23.039235206s, 6.52 MiB/s)
		  | I210114 11:59:14.978270 15 ccl/workloadccl/fixture.go:472  imported 1.7 GiB in customer table (3000000 rows, 3000000 index entries, took 51.086600324s, 34.38 MiB/s)
		  | I210114 11:59:35.445529 22 ccl/workloadccl/fixture.go:472  imported 3.0 GiB in stock table (10000000 rows, 0 index entries, took 1m11.552055799s, 42.75 MiB/s)
		  | I210114 11:59:39.972311 23 ccl/workloadccl/fixture.go:472  imported 1.6 GiB in order_line table (30005985 rows, 0 index entries, took 1m16.079541177s, 21.81 MiB/s)
		  | I210114 11:59:40.009118 1 ccl/workloadccl/fixture.go:351  imported 6.7 GiB bytes in 9 tables (took 1m16.125772458s, 90.10 MiB/s)
		  | I210114 11:59:41.223897 1 ccl/workloadccl/cliccl/fixtures.go:355  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I210114 11:59:41.246167 1 workload/tpcc/tpcc.go:389  check 3.3.2.1 took 22.216119ms
		  | I210114 11:59:42.413864 1 workload/tpcc/tpcc.go:389  check 3.3.2.2 took 1.167636278s
		  | I210114 11:59:42.815080 1 workload/tpcc/tpcc.go:389  check 3.3.2.3 took 401.157576ms
		  | I210114 11:59:56.604486 1 workload/tpcc/tpcc.go:389  check 3.3.2.4 took 13.789349946s
		  | I210114 11:59:59.243678 1 workload/tpcc/tpcc.go:389  check 3.3.2.5 took 2.639127554s
		  | Error: check failed: 3.3.2.5: right EXCEPT left returned nonzero results.
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=100
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /scrub/index-only/tpcc/w=100
Related:

roachtest: scrub/index-only/tpcc/w=100 failed #58832 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-01-15T12:10:41Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@a786c51627fe66e47b4a4445c67b2a9077ae2a93:

		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2305
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2225
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:120
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:760
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_120833.633_n1_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2586231-1610694347-76-n5cpu4:1 -- ./cockroach workload fixtures import tpcc --warehouses=100  returned
		  | stderr:
		  | I210115 12:08:34.461934 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I210115 12:08:34.626659 25 ccl/workloadccl/fixture.go:472  imported 99 KiB in district table (1000 rows, 0 index entries, took 156.821415ms, 0.62 MiB/s)
		  | I210115 12:08:34.738594 24 ccl/workloadccl/fixture.go:472  imported 5.1 KiB in warehouse table (100 rows, 0 index entries, took 269.072457ms, 0.02 MiB/s)
		  | I210115 12:08:40.716038 30 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 6.246018632s, 1.24 MiB/s)
		  | I210115 12:08:41.476815 29 ccl/workloadccl/fixture.go:472  imported 11 MiB in new_order table (900000 rows, 0 index entries, took 7.006729225s, 1.59 MiB/s)
		  | I210115 12:08:56.998917 27 ccl/workloadccl/fixture.go:472  imported 216 MiB in history table (3000000 rows, 0 index entries, took 22.529252852s, 9.57 MiB/s)
		  | I210115 12:08:58.372919 28 ccl/workloadccl/fixture.go:472  imported 150 MiB in order table (3000000 rows, 3000000 index entries, took 23.903012816s, 6.28 MiB/s)
		  | I210115 12:09:33.810189 26 ccl/workloadccl/fixture.go:472  imported 1.7 GiB in customer table (3000000 rows, 3000000 index entries, took 59.340575644s, 29.59 MiB/s)
		  | I210115 12:09:45.975806 31 ccl/workloadccl/fixture.go:472  imported 3.0 GiB in stock table (10000000 rows, 0 index entries, took 1m11.505772481s, 42.78 MiB/s)
		  | I210115 12:09:50.148074 32 ccl/workloadccl/fixture.go:472  imported 1.6 GiB in order_line table (30005985 rows, 0 index entries, took 1m15.678268837s, 21.92 MiB/s)
		  | I210115 12:09:50.185605 1 ccl/workloadccl/fixture.go:351  imported 6.7 GiB bytes in 9 tables (took 1m15.723438934s, 90.58 MiB/s)
		  | I210115 12:09:51.242083 1 ccl/workloadccl/cliccl/fixtures.go:355  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I210115 12:09:51.266620 1 workload/tpcc/tpcc.go:389  check 3.3.2.1 took 24.482896ms
		  | I210115 12:09:53.253100 1 workload/tpcc/tpcc.go:389  check 3.3.2.2 took 1.986422977s
		  | I210115 12:09:53.645559 1 workload/tpcc/tpcc.go:389  check 3.3.2.3 took 392.398829ms
		  | I210115 12:10:02.550038 1 workload/tpcc/tpcc.go:389  check 3.3.2.4 took 8.904424523s
		  | I210115 12:10:05.500259 1 workload/tpcc/tpcc.go:389  check 3.3.2.5 took 2.950163092s
		  | I210115 12:10:18.194343 1 workload/tpcc/tpcc.go:389  check 3.3.2.7 took 12.693995582s
		  | Error: check failed: 3.3.2.7: 6277 rows returned, expected zero
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=100
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /scrub/index-only/tpcc/w=100
Related:

roachtest: scrub/index-only/tpcc/w=100 failed #58832 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-18T12:38:28Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@3c223f5f5162103110a790743b687ef2bf952489:

		  |   799.0s       28            5.0            8.2   2415.9   3623.9   3623.9   3623.9 stockLevel
		  |   800.0s       28            8.0            8.2    385.9   5637.1   5637.1   5637.1 delivery
		  |   800.0s       28          103.0           81.8   5100.3  18253.6  23622.3  25769.8 newOrder
		  |   800.0s       28            4.0            8.3    130.0    184.5    184.5    184.5 orderStatus
		  |   800.0s       28           42.0           81.7   6442.5  13958.6  26843.5  26843.5 payment
		  |   800.0s       28            3.0            8.2   2684.4   2952.8   2952.8   2952.8 stockLevel
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |   801.0s       28            5.0            8.2    671.1   2281.7   2281.7   2281.7 delivery
		  |   801.0s       28           54.0           81.8   4295.0  23622.3  30064.8  51539.6 newOrder
		  |   801.0s       28            9.0            8.3    251.7   2013.3   2013.3   2013.3 orderStatus
		  |   801.0s       28           32.0           81.6   6979.3  13958.6  19327.4  19327.4 payment
		  |   801.0s       28            4.0            8.2    704.6   2281.7   2281.7   2281.7 stockLevel
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2687,tpcc.go:162,scrub.go:54,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /scrub/index-only/tpcc/w=100
Related:

roachtest: scrub/index-only/tpcc/w=100 failed #60410 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2021-02-19T12:03:37Z

(roachtest).scrub/index-only/tpcc/w=100 failed on master@83e70ce84b740e27e721c3b73c38a4b8b515094a:

		  |   473.0s        3           87.0           88.2   5100.3  16106.1  20401.1  21474.8 payment
		  |   473.0s        3            8.0            8.9    335.5   3489.7   3489.7   3489.7 stockLevel
		  |   474.0s        3            6.0            8.9    704.6  12348.0  12348.0  12348.0 delivery
		  |   474.0s        3           83.9           88.9   2818.6  10200.5  20401.1  24696.1 newOrder
		  |   474.0s        3            4.0            8.9    176.2    973.1    973.1    973.1 orderStatus
		  |   474.0s        3           62.9           88.2   5368.7  14495.5  19327.4  38654.7 payment
		  |   474.0s        3            5.0            8.9    604.0   2415.9   2415.9   2415.9 stockLevel
		  |   475.0s        3           14.0            8.9   1476.4   2684.4   4026.5   4026.5 delivery
		  |   475.0s        3           93.1           88.9   2550.1  12348.0  15569.3  24696.1 newOrder
		  |   475.0s        3           10.0            8.9    570.4   1409.3   1409.3   1409.3 orderStatus
		  |   475.0s        3           65.1           88.1   5100.3  11274.3  11811.2  13421.8 payment
		  |   475.0s        3            6.0            8.9    201.3   1342.2   1342.2   1342.2 stockLevel
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2687,tpcc.go:162,scrub.go:54,test_runner.go:767: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2675
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2683
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:162
		  | main.makeScrubTPCCTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/scrub.go:54
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2731
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2645
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5652
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:191
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /scrub/index-only/tpcc/w=100
Related:

roachtest: scrub/index-only/tpcc/w=100 failed #60410 roachtest: scrub/index-only/tpcc/w=100 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

rytaft · 2021-08-17T18:51:13Z

Closing since we haven't seen this since February, and artifacts have been deleted. Presumably it will be reopened if it happens again.

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Jan 16, 2020

cockroach-teamcity added this to the 20.1 milestone Jan 16, 2020

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

nvanbenschoten mentioned this issue Mar 24, 2020

roachtest: restart/down-for-2m failed #46326

Closed

nvanbenschoten mentioned this issue Apr 4, 2020

roachtest: tpccbench/nodes=12/cpu=16 failed #47004

Closed

irfansharif mentioned this issue Apr 23, 2020

roachtest: acceptance/gossip/restart-node-one failed #47808

Closed

This was referenced Jul 8, 2020

roachtest: scrub/index-only/tpcc/w=100 failed #51157

Closed

roachtest: scrub/index-only/tpcc/w=100 failed #51342

Closed

This was referenced Sep 29, 2020

roachtest: scrub/index-only/tpcc/w=100 failed #54922

Closed

roachtest: scrub/index-only/tpcc/w=100 failed #55022

Closed

roachtest: scrub/index-only/tpcc/w=100 failed #55039

Closed

This was referenced Jan 7, 2021

roachtest: scrub/index-only/tpcc/w=100 failed #58598

Closed

roachtest: scrub/index-only/tpcc/w=100 failed #58832

Closed

This was referenced Feb 9, 2021

roachtest: scrub/index-only/tpcc/w=100 failed #60125

Closed

roachtest: scrub/index-only/tpcc/w=100 failed #60410

Closed

blathers-crl bot added the T-sql-queries SQL Queries Team label Aug 12, 2021

rytaft closed this as completed Aug 17, 2021

mgartner added this to SQL Queries Jul 24, 2023

mgartner moved this to Done in SQL Queries Jul 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: scrub/index-only/tpcc/w=100 failed #44069

roachtest: scrub/index-only/tpcc/w=100 failed #44069

cockroach-teamcity commented Jan 16, 2020

cockroach-teamcity commented Feb 13, 2020

cockroach-teamcity commented Mar 4, 2020

cockroach-teamcity commented Mar 5, 2020

cockroach-teamcity commented Mar 6, 2020

cockroach-teamcity commented Mar 7, 2020

cockroach-teamcity commented Mar 8, 2020

cockroach-teamcity commented Mar 10, 2020

cockroach-teamcity commented Mar 10, 2020

cockroach-teamcity commented Mar 17, 2020

cockroach-teamcity commented Mar 20, 2020

jordanlewis commented Mar 20, 2020

irfansharif commented Apr 23, 2020

cockroach-teamcity commented Apr 30, 2020

cockroach-teamcity commented Jul 2, 2020

cockroach-teamcity commented Jul 4, 2020

cockroach-teamcity commented Jul 6, 2020

cockroach-teamcity commented Aug 26, 2020

cockroach-teamcity commented Jan 13, 2021

cockroach-teamcity commented Jan 14, 2021

cockroach-teamcity commented Jan 15, 2021

cockroach-teamcity commented Feb 18, 2021

cockroach-teamcity commented Feb 19, 2021

rytaft commented Aug 17, 2021

roachtest: scrub/index-only/tpcc/w=100 failed #44069

roachtest: scrub/index-only/tpcc/w=100 failed #44069

Comments

cockroach-teamcity commented Jan 16, 2020

cockroach-teamcity commented Feb 13, 2020

cockroach-teamcity commented Mar 4, 2020

cockroach-teamcity commented Mar 5, 2020

cockroach-teamcity commented Mar 6, 2020

cockroach-teamcity commented Mar 7, 2020

cockroach-teamcity commented Mar 8, 2020

cockroach-teamcity commented Mar 10, 2020

cockroach-teamcity commented Mar 10, 2020

cockroach-teamcity commented Mar 17, 2020

cockroach-teamcity commented Mar 20, 2020

jordanlewis commented Mar 20, 2020

irfansharif commented Apr 23, 2020

cockroach-teamcity commented Apr 30, 2020

cockroach-teamcity commented Jul 2, 2020

cockroach-teamcity commented Jul 4, 2020

cockroach-teamcity commented Jul 6, 2020

cockroach-teamcity commented Aug 26, 2020

cockroach-teamcity commented Jan 13, 2021

cockroach-teamcity commented Jan 14, 2021

cockroach-teamcity commented Jan 15, 2021

cockroach-teamcity commented Feb 18, 2021

cockroach-teamcity commented Feb 19, 2021

rytaft commented Aug 17, 2021