roachtest: schemachange/index/tpcc/w=1000 failed #44071

cockroach-teamcity · 2020-01-16T14:56:22Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1693614-1579160767-76-n5cpu16:5 -- ./workload run tpcc --warehouses=1000 --histograms=perf/stats.json --wait=false --tolerate-errors --workers=1000 --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		            0           56.0           56.2     35.7    436.2    520.1    520.1 orderStatus
		       21.0s        0          547.1          552.7    318.8    604.0    805.3   1006.6 payment
		       21.0s        0           56.0           55.8    121.6    453.0    503.3    503.3 stockLevel
		       22.0s        0          220.1           42.4   2684.4  11811.2  13421.8  15569.3 delivery
		       22.0s        0          340.1          538.7    335.5    738.2   1006.6   1208.0 newOrder
		       22.0s        0           47.0           55.8     71.3    570.4    906.0    906.0 orderStatus
		       22.0s        0          291.1          540.8    318.8    738.2   1073.7   1610.6 payment
		       22.0s        0           41.0           55.1    109.1    469.8    570.4    570.4 stockLevel
		       23.0s        0          360.7           56.2   5637.1  10200.5  11274.3  11811.2 delivery
		       23.0s        0          361.7          531.0    604.0   1140.9   1476.4   1811.9 newOrder
		       23.0s        0           43.0           55.2     65.0    285.2   1140.9   1140.9 orderStatus
		       23.0s        0          385.7          534.0    671.1   1208.0   1409.3   1946.2 payment
		       23.0s        0           40.0           54.5    142.6    805.3   1140.9   1140.9 stockLevel
		       24.0s        0          170.8           61.0   5637.1  11811.2  11811.2  12348.0 delivery
		       24.0s        0          508.5          530.1    738.2   1275.1   1543.5   2684.4 newOrder
		       24.0s        0           47.0           54.9     92.3    436.2    872.4    872.4 orderStatus
		       24.0s        0          425.6          529.5    805.3   1409.3   1744.8   2415.9 payment
		       24.0s        0           63.9           54.9    243.3    704.6    771.8   1275.1 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		       25.0s        0            4.0           58.7   1744.8   2415.9   2415.9   2415.9 delivery
		       25.0s        0           22.1          509.7    570.4   1140.9   1543.5   1543.5 newOrder
		       25.0s        0            0.0           52.7      0.0      0.0      0.0      0.0 orderStatus
		       25.0s        0            0.0          508.3      0.0      0.0      0.0      0.0 payment
		       25.0s        0            4.0           52.9     71.3   1040.2   1040.2   1040.2 stockLevel
		    E200116 14:43:30.173448 1 workload/cli/run.go:444  error in payment: EOF
		       26.0s    13514            0.0           56.5      0.0      0.0      0.0      0.0 delivery
		       26.0s    13514            0.0          490.1      0.0      0.0      0.0      0.0 newOrder
		       26.0s    13514            3.0           50.8     27.3     31.5     31.5     31.5 orderStatus
		       26.0s    13514            0.0          488.8      0.0      0.0      0.0      0.0 payment
		       26.0s    13514            4.0           51.0     50.3   1610.6   1610.6   1610.6 stockLevel
		    E200116 14:43:31.194892 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.22:26257: connect: connection refused
		       27.0s    33132            0.0           54.4      0.0      0.0      0.0      0.0 delivery
		       27.0s    33132            0.0          472.0      0.0      0.0      0.0      0.0 newOrder
		       27.0s    33132            0.0           48.9      0.0      0.0      0.0      0.0 orderStatus
		       27.0s    33132            0.0          470.7      0.0      0.0      0.0      0.0 payment
		       27.0s    33132            0.0           49.1      0.0      0.0      0.0      0.0 stockLevel
		    E200116 14:43:32.194834 1 workload/cli/run.go:444  error in orderStatus: dial tcp 10.128.0.22:26257: connect: connection refused
		       28.0s    44694            0.0           52.4      0.0      0.0      0.0      0.0 delivery
		       28.0s    44694            0.0          455.1      0.0      0.0      0.0      0.0 newOrder
		       28.0s    44694            0.0           47.2      0.0      0.0      0.0      0.0 orderStatus
		       28.0s    44694            0.0          453.9      0.0      0.0      0.0      0.0 payment
		       28.0s    44694            0.0           47.3      0.0      0.0      0.0      0.0 stockLevel
		    E200116 14:43:33.194878 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.22:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /schemachange/index/tpcc/w=1000

make stressrace TESTS=schemachange/index/tpcc/w=1000 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-01-22T06:51:31Z

(roachtest).schemachange/index/tpcc/w=1000 failed on provisional_202001212141_v19.1.7@a8444e28d3e7bf07c8ee0c6d5809e7466b57f6f9:

		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1702949-1579652818-77-n5cpu16:5 -- ./workload run tpcc --warehouses=1000 --histograms=perf/stats.json --wait=false --tolerate-errors --workers=1000 --ramp=5m0s --duration=2h0m0s {pgurl:1-4} returned:
		    stderr:
		    
		    stdout:
		           534.0    704.6   1342.2   1677.7   2080.4 payment
		      269.0s        0           49.0           53.6    167.8   1275.1   1409.3   1409.3 stockLevel
		      270.0s        0           58.0           53.5    536.9    771.8    771.8    872.4 delivery
		      270.0s        0          560.4          534.0    906.0   1677.7   2013.3   2415.9 newOrder
		      270.0s        0           33.0           53.5     52.4    234.9    318.8    318.8 orderStatus
		      270.0s        0          471.3          533.8    872.4   1208.0   1946.2   2147.5 payment
		      270.0s        0           49.0           53.6    167.8   1140.9   1476.4   1476.4 stockLevel
		      271.0s        0           45.0           53.4    570.4    939.5   1040.2   1040.2 delivery
		      271.0s        0          417.8          533.6    805.3   1342.2   1879.0   2550.1 newOrder
		      271.0s        0           49.0           53.5     71.3    184.5    201.3    201.3 orderStatus
		      271.0s        0          583.7          534.0    973.1   1543.5   2147.5   2550.1 payment
		      271.0s        0           55.0           53.6    184.5    872.4   1342.2   1610.6 stockLevel
		      272.0s        0           63.9           53.5    671.1   3758.1   4026.5   4831.8 delivery
		      272.0s        0          430.6          533.2   1140.9   1610.6   2281.7   2550.1 newOrder
		      272.0s        0           51.9           53.5     41.9    243.3    251.7    260.0 orderStatus
		      272.0s        0          617.4          534.3    738.2   1140.9   1946.2   2147.5 payment
		      272.0s        0           49.0           53.5    159.4   1476.4   1610.6   1610.6 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      273.0s        0           60.0           53.5    469.8   1073.7   1208.0   1275.1 delivery
		      273.0s        0          501.4          533.1    939.5   1342.2   1744.8   2550.1 newOrder
		      273.0s        0           60.0           53.5     44.0    218.1    260.0    637.5 orderStatus
		      273.0s        0          649.5          534.7    536.9    939.5   1811.9   1946.2 payment
		      273.0s        0           62.0           53.6    184.5   1409.3   1476.4   1476.4 stockLevel
		      274.0s        0           44.0           53.5    637.5   1275.1   1543.5   1543.5 delivery
		      274.0s        0          606.1          533.4   1342.2   1946.2   2684.4   3221.2 newOrder
		      274.0s        0           54.0           53.5     46.1    285.2    385.9    385.9 orderStatus
		      274.0s        0          288.0          533.8    738.2   1140.9   1543.5   1744.8 payment
		      274.0s        0           28.0           53.5    218.1    771.8    939.5    939.5 stockLevel
		      275.0s        0           54.0           53.5    520.1   1140.9   1208.0   1543.5 delivery
		      275.0s        0          562.0          533.5   1140.9   2147.5   2281.7   2684.4 newOrder
		      275.0s        0           63.0           53.5     75.5    234.9    234.9    285.2 orderStatus
		      275.0s        0          483.0          533.6    738.2    973.1   1208.0   1879.0 payment
		      275.0s        0           54.0           53.5    159.4    436.2    436.2    939.5 stockLevel
		      276.0s        0           38.0           53.4    637.5    939.5   1073.7   1073.7 delivery
		      276.0s        0          544.6          533.5    973.1   1409.3   1879.0   2080.4 newOrder
		      276.0s        0           56.0           53.5     48.2    167.8    192.9    302.0 orderStatus
		      276.0s        0          495.6          533.5    771.8   1040.2   1275.1   1879.0 payment
		      276.0s        0           54.0           53.5    159.4    570.4    771.8    973.1 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      277.0s        0           46.0           53.4    671.1   1006.6   1140.9   1140.9 delivery
		      277.0s        0          436.4          533.2    939.5   1677.7   2281.7   2684.4 newOrder
		      277.0s        0           54.0           53.5     39.8    167.8    176.2    243.3 orderStatus
		      277.0s        0          642.5          533.9    738.2   1275.1   2147.5   2550.1 payment
		      277.0s        0           57.0           53.5    142.6    570.4   1140.9   1409.3 stockLevel
		    Error:  exit status 255:
		  - exit status 1

Repro

Artifacts: /schemachange/index/tpcc/w=1000
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-13T13:20:43Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@2739821b911d777fa2a927295d699b559360a802:

		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_124932.593_n5_workload_fixtures_load_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.setupTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-96-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned:
		    stderr:
		    I200213 12:49:34.708973 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7
		    Error:  exit status 1
		    
		    stdout::
		  - exit status 1

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-03T20:37:53Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@5f9a71adb995837bcff27b9456188018434be4b8:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_171708.015_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1780216-1583223197-76-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    I200303 17:17:10.532490 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 677.455161ms
		    I200303 17:17:43.771268 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 33.238724661s
		    I200303 17:17:49.190301 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 5.418938479s
		    I200303 17:23:33.090416 1 workload/tpcc/tpcc.go:372  check 3.3.2.4 took 5m43.900066494s
		    I200303 17:27:00.555571 1 workload/tpcc/tpcc.go:372  check 3.3.2.5 took 3m27.465090263s
		    I200303 17:31:26.571744 1 workload/tpcc/tpcc.go:372  check 3.3.2.7 took 4m26.016122587s
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-05T19:53:31Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@c9b189b3a2171c8da78c75fe80dd02933e18a5e5:

	cluster.go:1935,tpcc.go:170,schemachange.go:302,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_163335.934_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1786843-1583394619-77-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    I200305 16:33:38.676649 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 901.505252ms
		    I200305 16:34:02.660995 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 23.984302685s
		    I200305 16:34:06.277152 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 3.61611037s
		    I200305 16:38:06.516583 1 workload/tpcc/tpcc.go:372  check 3.3.2.4 took 4m0.239386726s
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-06T21:56:03Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@954fe69d554162aec0fbc001aad1fe5103d8df13:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200306-1790595/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:756: test timed out (6h0m0s)

	cluster.go:1935,tpcc.go:170,schemachange.go:302,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_183757.412_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1790595-1583481796-86-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-07T21:02:46Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@752dea867f3aeb142e98c22f8d320ce19041aa8d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200307-1793930/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:756: test timed out (6h0m0s)

	cluster.go:1935,tpcc.go:170,schemachange.go:302,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_174617.990_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1793930-1583568484-78-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-08T20:41:05Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@dfa5bd527ae7d7373dd03c62118df87a87a77130:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200308-1795062/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:756: test timed out (6h0m0s)

	cluster.go:1935,tpcc.go:170,schemachange.go:302,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_172131.277_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1795062-1583655253-80-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-09T21:27:31Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200309-1796240/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:756: test timed out (6h0m0s)

	cluster.go:1935,tpcc.go:170,schemachange.go:302,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_180824.568_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1796240-1583738442-79-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-10T20:11:26Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@72c4a1bd411f2f82bf9aaa22883821a946614148:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200310-1799071/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:779: test timed out (6h0m0s)

	cluster.go:1935,tpcc.go:170,schemachange.go:302,test_runner.go:747: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_165637.898_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1799071-1583824469-86-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-18T20:48:14Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@b5f030223fbcf22e806c48a3c46e74a73a54f50f:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_173520.921_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1814553-1584515366-82-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    I200318 17:35:23.340383 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 520.961342ms
		    I200318 17:35:40.865824 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 17.525391917s
		    I200318 17:35:45.037600 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 4.171713461s
		    I200318 17:38:58.240006 1 workload/tpcc/tpcc.go:372  check 3.3.2.4 took 3m13.202361913s
		    I200318 17:42:21.734758 1 workload/tpcc/tpcc.go:372  check 3.3.2.5 took 3m23.494665709s
		    I200318 17:48:32.001645 1 workload/tpcc/tpcc.go:372  check 3.3.2.7 took 6m10.266803415s
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-19T19:46:17Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@d571624ae4f833c49c717728a74cc7be78a791f0:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_162247.932_n5_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.makeIndexAddTpccTest.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1816964-1584603715-82-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned:
		    stderr:
		    I200319 16:22:51.233774 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 654.596801ms
		    I200319 16:23:25.314890 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 34.081001771s
		    I200319 16:23:31.423672 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 6.108735312s
		    I200319 16:30:09.569003 1 workload/tpcc/tpcc.go:372  check 3.3.2.4 took 6m38.145233049s
		    I200319 16:33:16.068699 1 workload/tpcc/tpcc.go:372  check 3.3.2.5 took 3m6.499594505s
		    I200319 16:36:59.883819 1 workload/tpcc/tpcc.go:372  check 3.3.2.7 took 3m43.815031179s
		    
		    stdout::
		  - context canceled

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #44527 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-06-20T17:01:57Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@eb69237e5ee14728f32bbb667695ebb7472d9535:

		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_150032.298_n5_workload_check_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2025369-1592636061-79-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned
		  | stderr:
		  | o/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:147
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).pghosts.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1557
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1667
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		  | Wraps: (5) 5 safe details enclosed
		  | Wraps: (6) GetInternalIP: failed to execute hostname on teamcity-2025369-1592636061-79-n5cpu16:4:
		  |   | (stdout) 
		  |   | (stderr) 
		  | Wraps: (7) exit status 255
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *withstack.withStack (5) *safedetails.withSafeDetails (6) *errutil.withMessage (7) *exec.ExitError: 
		  | I200620 17:01:22.258158 10 cluster_synced.go:1749  command failed
		  |
		  | stdout:
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2025369-1592636061-79-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		2: 4244
		3: 4360
		1: 4282
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:766
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:852
		  | github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:800
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-06-25T12:56:21Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@4acd262796288cc84cb2b104bd77c1fc8cac73db:


	cluster.go:2484,tpcc.go:168,schemachange.go:302,test_runner.go:753: monitor failure: unexpected node event: 2: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2472
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2480
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1512,context.go:135,cluster.go:1501,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2037483-1593067161-82-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: dead
		3: 4268
		4: 4265
		1: 5256
		Error: UNCLASSIFIED_PROBLEM: 2: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 2: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString


Failed to find issue assignee: 
couldn't find GitHub commits for user email [email protected]

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-01T20:21:55Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@3a03f3843a8cdf04f82c52753c61cf01b0d2ddcd:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:804: test timed out (6h0m0s)

	cluster.go:2040,tpcc.go:136,tpcc.go:138,tpcc.go:149,schemachange.go:302,test_runner.go:757: output in run_142241.385_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2054002-1593584816-77-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_142241.385_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2054002-1593584816-77-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200701 14:22:43.300228 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200701 14:22:43.649499 60 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 336.760642ms (1000 rows, 0 index entries, 157 KiB)
		  | I200701 14:22:46.962069 61 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 3.649457018s (10000 rows, 0 index entries, 276 KiB)
		  | I200701 14:22:47.121899 130 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 3.80917722s (100000 rows, 0 index entries, 2.0 MiB)
		  | I200701 14:25:18.313939 65 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 2m35.001121176s (9000000 rows, 0 index entries, 830 KiB)
		  | I200701 14:29:02.393454 64 ccl/workloadccl/fixture.go:586  loaded 1.3 GiB table order in 6m19.080610901s (30000000 rows, 30000000 index entries, 3.6 MiB)
		  | I200701 14:31:26.888403 63 ccl/workloadccl/fixture.go:586  loaded 4.3 GiB table history in 8m43.575646455s (30000000 rows, 60000000 index entries, 8.4 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-04T19:56:04Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@456a07cfc1e53b87abc7709052e54efb1450e758:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:804: test timed out (6h0m0s)

	cluster.go:2040,tpcc.go:136,tpcc.go:138,tpcc.go:149,schemachange.go:302,test_runner.go:757: output in run_135654.927_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2062841-1593843322-80-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_135654.927_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2062841-1593843322-80-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200704 13:56:56.836257 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200704 13:56:57.604077 99 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 756.911423ms (100000 rows, 0 index entries, 10 MiB)
		  | I200704 13:57:00.999857 77 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 4.152609019s (1000 rows, 0 index entries, 13 KiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-05T20:03:08Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@3e0de239121813ea4d47873388a2828a66d9edf7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:804: test timed out (6h0m0s)

	cluster.go:2040,tpcc.go:136,tpcc.go:138,tpcc.go:149,schemachange.go:302,test_runner.go:757: output in run_140356.433_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2063962-1593929497-84-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2115
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2038
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_140356.433_n5_workload_fixtures_load_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2063962-1593929497-84-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I200705 14:03:58.364721 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		  | I200705 14:03:58.788441 45 ccl/workloadccl/fixture.go:586  loaded 53 KiB table warehouse in 413.106577ms (1000 rows, 0 index entries, 128 KiB)
		  | I200705 14:03:58.819023 46 ccl/workloadccl/fixture.go:586  loaded 1006 KiB table district in 443.718579ms (10000 rows, 0 index entries, 2.2 MiB)
		  | I200705 14:03:59.385207 99 ccl/workloadccl/fixture.go:586  loaded 7.8 MiB table item in 1.009982154s (100000 rows, 0 index entries, 7.7 MiB)
		  | I200705 14:06:20.327371 98 ccl/workloadccl/fixture.go:586  loaded 126 MiB table new_order in 2m21.95213191s (9000000 rows, 0 index entries, 906 KiB)
		  | I200705 14:09:16.791382 49 ccl/workloadccl/fixture.go:586  loaded 1.3 GiB table order in 5m18.41597215s (30000000 rows, 30000000 index entries, 4.3 MiB)
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-07T14:27:43Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@9304ecd70e9f3ba4cb16b5443a10b4e17d7baee0:

		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2515
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2429
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5420
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *withstack.withStack (4) *errutil.withMessage (5) *withstack.withStack (6) *errors.errorString

	cluster.go:1516,context.go:135,cluster.go:1505,test_runner.go:829: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2068260-1594103442-85-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		1: dead
		3: 4237
		2: 4220
		4: 4814
		Error: UNCLASSIFIED_PROBLEM: 1: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 1: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString


Failed to find issue assignee: 
couldn't find GitHub commits for user email [email protected]

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-08T18:34:29Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@e3fb5aa18d0f5064f7ba5d4df3864e94b3abb96d:

		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_163306.596_n5_workload_check_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2072412-1594189844-83-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned
		  | stderr:
		  | go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:127
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).pghosts.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1534
		  |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1644
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		  | Wraps: (5) 5 safe details enclosed
		  | Wraps: (6) GetInternalIP: failed to execute hostname on teamcity-2072412-1594189844-83-n5cpu16:4:
		  |   | (stdout) 
		  |   | (stderr) 
		  | Wraps: (7) exit status 255
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *withstack.withStack (5) *safedetails.withSafeDetails (6) *errutil.withMessage (7) *exec.ExitError: 
		  | I200708 18:33:56.813445 8 cluster_synced.go:1726  command failed
		  |
		  | stdout:
		Wraps: (5) exit status 1
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *exec.ExitError

	cluster.go:1516,context.go:135,cluster.go:1505,test_runner.go:829: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2072412-1594189844-83-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: 4245
		4: dead
		1: 4334
		2: 4366
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-07-10T16:23:34Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@542c83e7db1694208e09c7fea7d350db4993b897:


	cluster.go:1516,context.go:135,cluster.go:1505,test_runner.go:826: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2079204-1594361493-97-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: 4314
		4: dead
		1: 4268
		2: dead
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) secondary error attachment
		  | 2: dead
		  | (1) attached stack trace
		  |   | main.glob..func13
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:203
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		  | Wraps: (2) 3 safe details enclosed
		  | Wraps: (3) 2: dead
		  | Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errors.errorString
		Wraps: (3) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1789
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (4) 3 safe details enclosed
		Wraps: (5) 4: dead
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *withstack.withStack (4) *safedetails.withSafeDetails (5) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-07-13T15:47:13Z

I don't know that test has an owner, so sending it over to @lucy-zhang for triage.

thoszhang · 2020-07-13T18:49:02Z

First of all, I think the issue assigner isn't working because I have a different email on my commits. I just reconfigured my email address in git, so hopefully it'll start working. I'll take a look at the actual issue shortly.

cockroach-teamcity · 2020-07-24T12:10:13Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@bfa6307c292ef4dfed4a53cb99f506e6dab26533:

		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

	cluster.go:2553,tpcc.go:187,schemachange.go:302,test_runner.go:757: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2541
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2549
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:757
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *errutil.withMessage (3) *errors.errorString

	cluster.go:1571,context.go:135,cluster.go:1560,test_runner.go:826: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2114210-1595571129-79-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: dead
		2: 4240
		4: 4255
		1: 4470
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-07T18:27:32Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@d310eb8eb6da5df4ac4407a2c90d898acdfdddf3:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:801: test timed out (6h0m0s)

	cluster.go:2126,tpcc.go:189,schemachange.go:302,test_runner.go:754: output in run_145827.393_n5_workload_check_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2159276-1596781348-77-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2204
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2124
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:189
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1373
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_145827.393_n5_workload_check_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2159276-1596781348-77-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-10T18:13:01Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@fdf54ebc69c7d975f8f1dfbf28ced79ddeb83ee4:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	test_runner.go:801: test timed out (6h0m0s)

	cluster.go:2126,tpcc.go:189,schemachange.go:302,test_runner.go:754: output in run_144312.142_n5_workload_check_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2164877-1597040336-76-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned: context canceled
		(1) attached stack trace
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2204
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2124
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:189
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) 2 safe details enclosed
		Wraps: (3) output in run_144312.142_n5_workload_check_tpcc
		Wraps: (4) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2164877-1597040336-76-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned
		  | stderr:
		  |
		  | stdout:
		Wraps: (5) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-19T12:50:42Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@ebd5c732f83009acc9c6f5859ca95e74e5453a1c:

		  | Error types: (1) *exec.ExitError
		Wraps: (6) context canceled
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *main.withCommandDetails (5) *secondary.withSecondaryError (6) *errors.errorString

	cluster.go:2597,tpcc.go:187,schemachange.go:302,test_runner.go:754: monitor failure: unexpected node event: 3: dead
		(1) attached stack trace
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2585
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2593
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) 1 safe detail enclosed
		Wraps: (3) monitor failure
		Wraps: (4) unexpected node event: 3: dead
		Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *errutil.withMessage (4) *errors.errorString

	cluster.go:1612,context.go:135,cluster.go:1601,test_runner.go:823: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2195516-1597818221-75-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		3: dead
		2: 5278
		4: 4271
		1: 5072
		Error: UNCLASSIFIED_PROBLEM: 3: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  | main.glob..func13
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1115
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:266
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1808
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 3 safe details enclosed
		Wraps: (4) 3: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *safedetails.withSafeDetails (4) *errors.errorString

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-26T10:05:43Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@ef55609797c92e46eb9efd08facc9db49558291d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	cluster.go:2167,tpcc.go:147,tpcc.go:157,tpcc.go:168,schemachange.go:302,test_runner.go:754: output in run_100521.964_n5_workload_fixtures_load_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2223072-1598421936-78-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2165
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:147
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_100521.964_n5_workload_fixtures_load_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2223072-1598421936-78-n5cpu16:5 -- ./workload fixtures load tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | Error: finding fixture: fixture not found: workload/tpcc/version=2.1.0,deprecated-fk-indexes=false,fks=true,interleaved=false,seed=1,warehouses=1000
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=1000  {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-27T13:52:25Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@fb25dbe53c2bc99e673c292b71d73d30247a7450:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/index/tpcc/w=1000/run_1
	cluster.go:2183,tpcc.go:189,schemachange.go:302,test_runner.go:754: output in run_134035.623_n5_workload_check_tpcc: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2226493-1598508864-81-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2261
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2181
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:189
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_134035.623_n5_workload_check_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2226493-1598508864-81-n5cpu16:5 -- ./workload check tpcc --warehouses=1000 {pgurl:1} returned
		  | stderr:
		  | I200827 13:40:38.175442 1 workload/tpcc/tpcc.go:387  check 3.3.2.1 took 308.708468ms
		  | I200827 13:41:10.562804 1 workload/tpcc/tpcc.go:387  check 3.3.2.2 took 32.38714367s
		  | I200827 13:41:13.539904 1 workload/tpcc/tpcc.go:387  check 3.3.2.3 took 2.976990489s
		  | I200827 13:45:51.083499 1 workload/tpcc/tpcc.go:387  check 3.3.2.4 took 4m37.543512859s
		  | I200827 13:47:33.815073 1 workload/tpcc/tpcc.go:387  check 3.3.2.5 took 1m42.73141341s
		  | I200827 13:51:30.305278 1 workload/tpcc/tpcc.go:387  check 3.3.2.7 took 3m56.490080384s
		  | Error: check failed: 3.3.2.7: 4064 rows returned, expected zero
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./workload check tpcc --warehouses=1000 {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-08-28T11:43:56Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@123ca0782f423bc3d225a5ca798ece60c3185935:

		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2613,tpcc.go:187,schemachange.go:302,test_runner.go:754: monitor failure: unexpected node event: 4: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2601
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2609
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:187
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 4: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1628,context.go:135,cluster.go:1617,test_runner.go:823: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2229351-1598594772-81-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		2: 4296
		3: 4359
		1: 4535
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-03T11:33:27Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@11a82264044ed8edc7d1fca2b912296d6f576a1a:

		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2639,tpcc.go:190,schemachange.go:302,test_runner.go:754: monitor failure: unexpected node event: 4: dead
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2627
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2635
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:190
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 4: dead
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1642,context.go:135,cluster.go:1631,test_runner.go:823: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-2243794-1599113210-80-n5cpu16 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		2: 4345
		1: 4520
		3: 4329
		Error: UNCLASSIFIED_PROBLEM: 4: dead
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1143
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:267
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:830
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:914
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/pkg/mod/github.com/spf13/[email protected]/command.go:864
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1839
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:203
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (3) 4: dead
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #53615 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #51224 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-provisional_202007081918_v20.2.0-alpha.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #47559 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-29T11:30:01Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@0c12d8b101a853f441926ad758c4c2882ecd33ee:

		  |   113.0s        0          721.2          565.5    192.9    318.8    352.3    469.8 payment
		  |   113.0s        0           63.0           56.6     83.9    268.4    285.2    285.2 stockLevel
		  |   114.0s        0           65.0           56.7    218.1    486.5    570.4    604.0 delivery
		  |   114.0s        0          667.2          566.2    302.0    469.8    570.4    637.5 newOrder
		  |   114.0s        0           60.0           56.7     30.4     96.5    142.6    209.7 orderStatus
		  |   114.0s        0          714.2          566.8    201.3    369.1    469.8    570.4 payment
		  |   114.0s        0           60.0           56.6     71.3    352.3    352.3    453.0 stockLevel
		  |   115.0s        0           71.0           56.8    260.0    520.1    570.4    604.0 delivery
		  |   115.0s        0          643.8          566.9    318.8    503.3    604.0    838.9 newOrder
		  |   115.0s        0           76.0           56.9     24.1    125.8    142.6    142.6 orderStatus
		  |   115.0s        0          622.8          567.3    209.7    352.3    453.0    671.1 payment
		  |   115.0s        0           73.0           56.7     96.5    335.5    436.2    453.0 stockLevel
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2651,tpcc.go:168,schemachange.go:302,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2639
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2647
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2695
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2609
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #54925 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #54304 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-09-30T11:24:04Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@5a7f32fd41cb84d35562db09761912ff9c903700:

		  |   125.0s        0          743.8          595.9    218.1    352.3    419.4    570.4 payment
		  |   125.0s        0           66.1           59.6     83.9    218.1    251.7    369.1 stockLevel
		  |   126.0s        0           70.0           59.5    318.8    671.1    704.6    738.2 delivery
		  |   126.0s        0          665.0          596.3    318.8    486.5    536.9    637.5 newOrder
		  |   126.0s        0           87.0           59.8     32.5    109.1    113.2    113.2 orderStatus
		  |   126.0s        0          708.0          596.8    226.5    385.9    453.0    604.0 payment
		  |   126.0s        0           59.0           59.6    117.4    218.1    385.9    436.2 stockLevel
		  |   127.0s        0           74.0           59.7    385.9    637.5    771.8    805.3 delivery
		  |   127.0s        0          693.0          597.1    302.0    503.3    570.4    973.1 newOrder
		  |   127.0s        0           73.0           59.9     39.8     96.5    142.6    184.5 orderStatus
		  |   127.0s        0          747.0          598.0    218.1    369.1    436.2    604.0 payment
		  |   127.0s        0           76.0           59.7     96.5    285.2    385.9    503.3 stockLevel
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

	cluster.go:2651,tpcc.go:168,schemachange.go:302,test_runner.go:755: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2639
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2647
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2695
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2609
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:5228
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:190
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #55038 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #55018 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker
roachtest: schemachange/index/tpcc/w=1000 failed #54304 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-10-06T11:29:43Z

(roachtest).schemachange/index/tpcc/w=1000 failed on master@203415533cc0400e6d2e0095aed3453f3cad7cb7:

		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2287
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2207
		  | main.setupTPCC.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:124
		  | main.setupTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:140
		  | main.runTPCC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:151
		  | main.makeIndexAddTpccTest.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/schemachange.go:302
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:755
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (2) output in run_111928.106_n5_cockroach_workload_fixtures_import_tpcc
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2343043-1601965108-71-n5cpu16:5 -- ./cockroach workload fixtures import tpcc --warehouses=1000  {pgurl:1} returned
		  | stderr:
		  | I201006 11:19:29.586094 1 ccl/workloadccl/fixture.go:342  starting import of 9 tables
		  | I201006 11:19:35.514067 33 ccl/workloadccl/fixture.go:472  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 5.911968238s, 1.32 MiB/s)
		  | I201006 11:19:36.778714 27 ccl/workloadccl/fixture.go:472  imported 53 KiB in warehouse table (1000 rows, 0 index entries, took 7.176757819s, 0.01 MiB/s)
		  | I201006 11:19:37.912764 28 ccl/workloadccl/fixture.go:472  imported 1006 KiB in district table (10000 rows, 0 index entries, took 8.310785727s, 0.12 MiB/s)
		  | I201006 11:19:44.588913 32 ccl/workloadccl/fixture.go:472  imported 126 MiB in new_order table (9000000 rows, 0 index entries, took 14.986851405s, 8.38 MiB/s)
		  | I201006 11:20:36.078791 31 ccl/workloadccl/fixture.go:472  imported 1.6 GiB in order table (30000000 rows, 30000000 index entries, took 1m6.476662032s, 24.00 MiB/s)
		  | I201006 11:20:41.362935 30 ccl/workloadccl/fixture.go:472  imported 2.2 GiB in history table (30000000 rows, 0 index entries, took 1m11.760860614s, 30.74 MiB/s)
		  | I201006 11:22:55.075511 29 ccl/workloadccl/fixture.go:472  imported 17 GiB in customer table (30000000 rows, 30000000 index entries, took 3m25.473371151s, 85.93 MiB/s)
		  | I201006 11:24:53.719790 83 ccl/workloadccl/fixture.go:472  imported 17 GiB in order_line table (300011520 rows, 0 index entries, took 5m24.117556954s, 53.13 MiB/s)
		  | I201006 11:25:07.132577 82 ccl/workloadccl/fixture.go:472  imported 31 GiB in stock table (100000000 rows, 100000000 index entries, took 5m37.530340185s, 95.04 MiB/s)
		  | I201006 11:25:07.241807 1 ccl/workloadccl/fixture.go:351  imported 69 GiB bytes in 9 tables (took 5m37.65539348s, 209.94 MiB/s)
		  | I201006 11:25:09.027142 1 ccl/workloadccl/cliccl/fixtures.go:355  fixture is restored; now running consistency checks (ctrl-c to abort)
		  | I201006 11:25:09.176870 1 workload/tpcc/tpcc.go:384  check 3.3.2.1 took 149.6658ms
		  | I201006 11:25:29.508682 1 workload/tpcc/tpcc.go:384  check 3.3.2.2 took 20.33172753s
		  | I201006 11:25:31.399631 1 workload/tpcc/tpcc.go:384  check 3.3.2.3 took 1.890813407s
		  | I201006 11:27:11.690783 1 workload/tpcc/tpcc.go:384  check 3.3.2.4 took 1m40.291105974s
		  | I201006 11:28:00.988272 1 workload/tpcc/tpcc.go:384  check 3.3.2.5 took 49.297309547s
		  | I201006 11:29:14.846254 1 workload/tpcc/tpcc.go:384  check 3.3.2.7 took 1m13.857785253s
		  | Error: check failed: 3.3.2.7: pq: root: memory budget exceeded: 40960 bytes requested, 3781222044 currently allocated, 3781235712 bytes in budget
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 5. Command with error:
		  |   | ```
		  |   | ./cockroach workload fixtures import tpcc --warehouses=1000  {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

More

Artifacts: /schemachange/index/tpcc/w=1000
Related:

roachtest: schemachange/index/tpcc/w=1000 failed #54304 roachtest: schemachange/index/tpcc/w=1000 failed C-test-failure O-roachtest O-robot branch-release-20.2

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

jordanlewis · 2020-10-06T13:26:21Z

This is new - we're failing to finish check 3.3.2.7 due to a memory budget exceeded error.

During the checks, are we also running an index build? If the checks are running by themselves, this could be fallout from the change to monitor KV memory via SQL.

yuzefovich · 2020-10-06T15:15:21Z

The memory budget error actually comes from the auto stats

teamcity-2343043-1601965108-71-n5cpu16-0001> I201006 11:27:04.400788 112513 jobs/registry.go:1063 â‹® [n1] AUTO CREATE STATS job 596093084952199169: stepping through state reverting with error: â€¹Ã—â€º: memory budget exceeded: 10240 bytes requested, 3781232640 currently allocated, 3781235712 bytes in budget
(1) tags: [n1,received-error]
Wraps: (2) tags: [nâ€¹Ã—â€º,sent-error=â€¹Ã—â€º]
Wraps: (3)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:682
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:690
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:690
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).increaseBudget
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:690
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BytesMonitor).reserveBytes
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:615
  | github.com/cockroachdb/cockroach/pkg/util/mon.(*BoundAccount).Grow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:570
  | github.com/cockroachdb/cockroach/pkg/sql/rowenc.(*EncDatum).Fingerprint
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowenc/encoded_datum.go:328
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*sketchInfo).addRow
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:550
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).mainLoop
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:334
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).Run
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:234
  | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).startInternal.func1
  | 	/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:338
  | runtime.goexit
  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
Wraps: (4) â€¹Ã—â€º
Wraps: (5) candidate pg code: 53200

and occurs because of my recent addition in #54770. I realized that it was a bad idea, so I already have #55199 to remove that addition.

yuzefovich · 2020-10-06T17:04:47Z

I'm closing this issue because the latest failure is due to a known issue that I've recently introduced, the second latest is the index backfiller memory accounting problem which has been fixed, and for all previous issues the logs are no longer there. @lucy-zhang feel free to reopen if you think there is some value in this issue.

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Jan 16, 2020

cockroach-teamcity added this to the 20.1 milestone Jan 16, 2020

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

cockroach-teamcity mentioned this issue Jan 30, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #44527

Closed

This was referenced Apr 16, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #47559

Closed

roachtest: schemachange/index/tpcc/w=1000 failed #47650

Closed

nvanbenschoten mentioned this issue Jun 30, 2020

roachtest: tpccbench/nodes=6/cpu=16/multi-az failed #50763

Closed

cockroach-teamcity mentioned this issue Jul 9, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #51224

Closed

irfansharif assigned thoszhang Jul 13, 2020

cockroach-teamcity mentioned this issue Aug 29, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #53615

Closed

cockroach-teamcity mentioned this issue Sep 12, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #54304

Closed

thoszhang mentioned this issue Sep 21, 2020

roachtest: schemachange/during/tpcc failed #47614

Closed

cockroach-teamcity mentioned this issue Sep 29, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #54925

Closed

This was referenced Sep 30, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #55018

Closed

roachtest: schemachange/index/tpcc/w=1000 failed #55038

Closed

yuzefovich mentioned this issue Oct 6, 2020

rowexec: remove recently added memory accounting in sampler #55199

Merged

yuzefovich closed this as completed Oct 6, 2020

roachtest: schemachange/index/tpcc/w=1000 failed #44071

roachtest: schemachange/index/tpcc/w=1000 failed #44071

Comments

cockroach-teamcity commented Jan 16, 2020

cockroach-teamcity commented Jan 22, 2020

cockroach-teamcity commented Feb 13, 2020

cockroach-teamcity commented Mar 3, 2020

cockroach-teamcity commented Mar 5, 2020

cockroach-teamcity commented Mar 6, 2020

cockroach-teamcity commented Mar 7, 2020

cockroach-teamcity commented Mar 8, 2020

cockroach-teamcity commented Mar 9, 2020

cockroach-teamcity commented Mar 10, 2020

cockroach-teamcity commented Mar 18, 2020

cockroach-teamcity commented Mar 19, 2020

cockroach-teamcity commented Jun 20, 2020

cockroach-teamcity commented Jun 25, 2020

cockroach-teamcity commented Jul 1, 2020

cockroach-teamcity commented Jul 4, 2020

cockroach-teamcity commented Jul 5, 2020

cockroach-teamcity commented Jul 7, 2020

cockroach-teamcity commented Jul 8, 2020

cockroach-teamcity commented Jul 10, 2020

irfansharif commented Jul 13, 2020

thoszhang commented Jul 13, 2020

cockroach-teamcity commented Jul 24, 2020

cockroach-teamcity commented Aug 7, 2020

cockroach-teamcity commented Aug 10, 2020

cockroach-teamcity commented Aug 19, 2020

cockroach-teamcity commented Aug 26, 2020

cockroach-teamcity commented Aug 27, 2020

cockroach-teamcity commented Aug 28, 2020

cockroach-teamcity commented Sep 3, 2020

cockroach-teamcity commented Sep 29, 2020

cockroach-teamcity commented Sep 30, 2020

cockroach-teamcity commented Oct 6, 2020

jordanlewis commented Oct 6, 2020

yuzefovich commented Oct 6, 2020

yuzefovich commented Oct 6, 2020