roachtest: tpcc/w=100/nodes=3/chaos=true failed #41584

cockroach-teamcity · 2019-10-15T05:17:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/a40cefbb5bd5e9d34f5db930334a385d0934d2d0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1538863&tab=artifacts#/tpcc/w=100/nodes=3/chaos=true

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191015-1538863/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1766,tpcc.go:136,tpcc.go:137,tpcc.go:148,tpcc.go:304,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1571116260-07-n4cpu4:4 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned:
		stderr:
		
		stdout:
		I191015 05:15:54.976238 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		I191015 05:16:00.332226 73 ccl/workloadccl/fixture.go:547  loaded 99 KiB table district in 5.347238265s (1000 rows, 0 index entries, 18 KiB)
		I191015 05:16:18.484318 77 ccl/workloadccl/fixture.go:547  loaded 11 MiB table new_order in 23.499486894s (900000 rows, 0 index entries, 486 KiB)
		I191015 05:16:30.923538 76 ccl/workloadccl/fixture.go:547  loaded 126 MiB table order in 35.938681328s (3000000 rows, 3000000 index entries, 3.5 MiB)
		Error:  exit status 255
		: exit status 1

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-11-22T08:28:36Z

SHA: https://github.com/cockroachdb/cockroach/commits/c128b5b9a0b1d98181bf124f775154df97948fd2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1604949&tab=artifacts#/tpcc/w=100/nodes=3/chaos=true

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191122-1604949/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1783,tpcc.go:157,cluster.go:2142,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1604949-1574406222-03-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		stderr:
		
		stdout:
		l
		  388.0s  2162230            7.0            5.6   1275.1   2281.7   2281.7   2281.7 delivery
		  388.0s  2162230           56.0           55.0   2281.7  19327.4  20401.1  22548.6 newOrder
		  388.0s  2162230            3.0            5.8     71.3    352.3    352.3    352.3 orderStatus
		  388.0s  2162230           61.0           52.9   4295.0  49392.1  51539.6  53687.1 payment
		  388.0s  2162230            8.0            5.7    234.9   4026.5   4026.5   4026.5 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  389.0s  2162230            8.0            5.7   1208.0   3623.9   3623.9   3623.9 delivery
		  389.0s  2162230           70.0           55.1   3623.9  17179.9  19327.4  21474.8 newOrder
		  389.0s  2162230           13.0            5.8     88.1   2550.1   2818.6   2818.6 orderStatus
		  389.0s  2162230           66.0           53.0   3758.1  42949.7  53687.1  53687.1 payment
		  389.0s  2162230            5.0            5.7    285.2   1275.1   1275.1   1275.1 stockLevel
		: context canceled

cockroach-teamcity · 2019-12-21T08:45:14Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@dd9b1c1f40dda59ee9d446416106d311ae5ce1e6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191221-1658785/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1658785-1576914291-08-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    0      0.0 newOrder
		     1307.0s  7407299            0.0            8.5      0.0      0.0      0.0      0.0 orderStatus
		     1307.0s  7407299            1.0           82.9  24696.1  24696.1  24696.1  24696.1 payment
		     1307.0s  7407299            0.0            8.5      0.0      0.0      0.0      0.0 stockLevel
		    E191221 08:44:47.606044 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.1.27:26257: connect: connection refused
		     1308.0s  7435318            0.0            8.4      0.0      0.0      0.0      0.0 delivery
		     1308.0s  7435318            0.0           84.6      0.0      0.0      0.0      0.0 newOrder
		     1308.0s  7435318            0.0            8.5      0.0      0.0      0.0      0.0 orderStatus
		     1308.0s  7435318            0.0           82.8      0.0      0.0      0.0      0.0 payment
		     1308.0s  7435318            0.0            8.5      0.0      0.0      0.0      0.0 stockLevel
		    E191221 08:44:48.606586 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.1.27:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-25T08:21:30Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@c172b37ec3316ec982e1ee92d4414a76823680e6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191225-1662271/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1786,tpcc.go:157,cluster.go:2145,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1820
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1784
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2145
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1662271-1577259744-05-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		         6.8    872.4   2415.9   2415.9   2415.9 delivery
		      281.0s  2544604           66.1           67.8   1208.0  31138.5  33286.0  42949.7 newOrder
		      281.0s  2544604           14.0            7.0    142.6   4831.8  30064.8  30064.8 orderStatus
		      281.0s  2544604           52.1           65.2   1208.0  30064.8  49392.1  57982.1 payment
		      281.0s  2544604            7.0            7.0    419.4    671.1    671.1    671.1 stockLevel
		    E191225 08:21:04.931039 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.11:26257: connect: connection refused
		      282.0s  2567100            9.0            6.8   1040.2   5100.3   5100.3   5100.3 delivery
		      282.0s  2567100           67.9           67.8   1476.4  12884.9  24696.1  31138.5 newOrder
		      282.0s  2567100            5.0            7.0   2281.7  28991.0  28991.0  28991.0 orderStatus
		      282.0s  2567100           66.9           65.2   1409.3  28991.0  30064.8  32212.3 payment
		      282.0s  2567100            4.0            7.0   1476.4   9663.7   9663.7   9663.7 stockLevel:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-31T08:21:20Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@f1f15250871d124178f40bf340ef48719b50e7e5:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191231-1668189/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1668189-1577777590-15-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    32212.3 newOrder
		      502.0s  3880751            5.9            7.4     83.9   2080.4   2080.4   2080.4 orderStatus
		      502.0s  3880751           69.3           70.2   3221.2  40802.2  53687.1  57982.1 payment
		      502.0s  3880751            5.9            7.3    176.2   1409.3   1409.3   1409.3 stockLevel
		    E191231 08:20:55.458834 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.15.204:26257: connect: connection refused
		      503.0s  3907274            5.1            7.2   1040.2   5100.3   5100.3   5100.3 delivery
		      503.0s  3907274           50.6           72.6   1342.2  22548.6  42949.7  42949.7 newOrder
		      503.0s  3907274            9.1            7.4    234.9   7247.8   7247.8   7247.8 orderStatus
		      503.0s  3907274           80.0           70.2   3489.7  36507.2  53687.1  90194.3 payment
		      503.0s  3907274            7.1            7.3   1677.7  16643.0  16643.0  16643.0 stockLevel
		    E191231 08:20:56.458966 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.15.204:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-03T08:45:58Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@8facb469d66426a3f372a1aed4ea6f34e287b501:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200103-1671733/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1671733-1578038067-07-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		       7.1     96.5   1543.5   1543.5   1543.5 delivery
		      787.0s  6453599            1.0           70.7     60.8     60.8     60.8     60.8 newOrder
		      787.0s  6453599            5.0            7.1     19.9     75.5     75.5     75.5 orderStatus
		      787.0s  6453599           47.0           69.1   3221.2  34359.7  85899.3  85899.3 payment
		      787.0s  6453599            0.0            7.2      0.0      0.0      0.0      0.0 stockLevel
		    E200103 08:45:35.525963 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.15.233:26257: connect: connection refused
		      788.0s  6458847            0.0            7.1      0.0      0.0      0.0      0.0 delivery
		      788.0s  6458847            0.0           70.6      0.0      0.0      0.0      0.0 newOrder
		      788.0s  6458847            0.0            7.1      0.0      0.0      0.0      0.0 orderStatus
		      788.0s  6458847            0.0           69.1      0.0      0.0      0.0      0.0 payment
		      788.0s  6458847            0.0            7.2      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-06T08:26:41Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@f180501b7c2ccb946d7f7c8944d5509b87cb0394:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200106-1674751/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1674751-1578297236-02-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    1  20401.1 newOrder
		      121.0s   663684            6.0            9.5     96.5    234.9    234.9    234.9 orderStatus
		      121.0s   663684           75.8           93.2    738.2   3758.1   5100.3   5100.3 payment
		      121.0s   663684            8.0            9.5     92.3   1409.3   1409.3   1409.3 stockLevel
		    E200106 08:26:15.895964 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.133:26257: connect: connection refused
		      122.0s   678211           18.0            9.4   1342.2   2415.9   4160.7   4160.7 delivery
		      122.0s   678211            1.0           93.2    125.8    125.8    125.8    125.8 newOrder
		      122.0s   678211            4.0            9.5     33.6     56.6     56.6     56.6 orderStatus
		      122.0s   678211           76.2           93.1    453.0   4295.0   5100.3   9663.7 payment
		      122.0s   678211            4.0            9.5     25.2    184.5    184.5    184.5 stockLevel
		    E200106 08:26:16.895864 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.133:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-06T22:09:52Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on 43008@b555d16b4455f7122f4211082512820742f8116f:

The test failed on branch=43008, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200106-1676232/tpcc/w=100/nodes=3/chaos=true/run_1
	cluster.go:1809,tpcc.go:157,cluster.go:2168,errgroup.go:57: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1843
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1807
		    main.runTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:157
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2168
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1676232-1578345768-03-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    3  25769.8 newOrder
		      625.0s  4189814           11.1            7.0    130.0   1040.2   3221.2   3221.2 orderStatus
		      625.0s  4189814           86.6           66.4   1476.4  34359.7  47244.6  53687.1 payment
		      625.0s  4189814           10.1            6.9    243.3   2550.1   2550.1   2550.1 stockLevel
		    E200106 22:09:31.892824 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.140:26257: connect: connection refused
		      626.0s  4213020            8.1            6.8    872.4   1879.0   1879.0   1879.0 delivery
		      626.0s  4213020           80.7           68.3   1476.4  10200.5  26843.5  27917.3 newOrder
		      626.0s  4213020            8.1            7.0    176.2   1476.4   1476.4   1476.4 orderStatus
		      626.0s  4213020           58.5           66.4   1811.9  33286.0  40802.2  42949.7 payment
		      626.0s  4213020            4.0            6.9    520.1   1409.3   1409.3   1409.3 stockLevel
		    E200106 22:09:32.892961 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.140:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-15T08:44:49Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@0046d62b00aa9e74489d1f44125948c9af729e2b:

		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1690426-1579074735-12-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		           8.0            5.5    369.1  24696.1  24696.1  24696.1 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      333.0s  2835146            3.0            5.4    771.8   1744.8   1744.8   1744.8 delivery
		      333.0s  2835146           47.0           54.7   2550.1  32212.3  55834.6  55834.6 newOrder
		      333.0s  2835146            7.0            5.5    453.0  21474.8  21474.8  21474.8 orderStatus
		      333.0s  2835146           50.0           53.0   1140.9  53687.1  77309.4  77309.4 payment
		      333.0s  2835146            3.0            5.5    939.5   3355.4   3355.4   3355.4 stockLevel
		      334.0s  2835146            7.0            5.4   1543.5  19327.4  19327.4  19327.4 delivery
		      334.0s  2835146           88.0           54.8   2684.4  23622.3  30064.8  31138.5 newOrder
		      334.0s  2835146            7.0            5.5    402.7  32212.3  32212.3  32212.3 orderStatus
		      334.0s  2835146           43.0           52.9  20401.1  42949.7  98784.2  98784.2 payment
		      334.0s  2835146            7.0            5.5    486.5   4831.8   4831.8   4831.8 stockLevel
		      335.0s  2835146            6.0            5.4   1543.5   6442.5   6442.5   6442.5 delivery
		      335.0s  2835146           57.0           54.8   1610.6  25769.8  30064.8  33286.0 newOrder
		      335.0s  2835146           21.0            5.6  20401.1  20401.1  20401.1  20401.1 orderStatus
		      335.0s  2835146           60.0           53.0   6710.9  42949.7  51539.6  60129.5 payment
		      335.0s  2835146           13.0            5.5   1275.1  20401.1  20401.1  20401.1 stockLevel
		    E200115 08:44:21.691469 1 workload/cli/run.go:444  error in payment: EOF
		      336.0s  2846556            6.0            5.4   1543.5  21474.8  21474.8  21474.8 delivery
		      336.0s  2846556           54.9           54.8   2080.4  21474.8  23622.3  38654.7 newOrder
		      336.0s  2846556            5.0            5.6    176.2    318.8    318.8    318.8 orderStatus
		      336.0s  2846556           55.9           53.0   2415.9  36507.2  40802.2  81604.4 payment
		      336.0s  2846556            8.0            5.6    805.3  28991.0  28991.0  28991.0 stockLevel
		    E200115 08:44:22.691469 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.1.4:26257: connect: connection refused
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      337.0s  2870761            6.0            5.4   1946.2   6174.0   6174.0   6174.0 delivery
		      337.0s  2870761           26.0           54.7   1610.6  23622.3  24696.1  24696.1 newOrder
		      337.0s  2870761            6.0            5.6    104.9   9126.8   9126.8   9126.8 orderStatus
		      337.0s  2870761           51.9           53.0   2952.8  31138.5  38654.7  62277.0 payment
		      337.0s  2870761            8.0            5.6    453.0  10737.4  10737.4  10737.4 stockLevel
		    E200115 08:44:23.691570 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.1.4:26257: connect: connection refused
		      338.0s  2895829            8.0            5.4   2281.7  11274.3  11274.3  11274.3 delivery
		      338.0s  2895829           15.0           54.6   3355.4   9126.8  11274.3  11274.3 newOrder
		      338.0s  2895829            4.0            5.6    104.9    167.8    167.8    167.8 orderStatus
		      338.0s  2895829           71.9           53.0   3355.4  38654.7  45097.2  62277.0 payment
		      338.0s  2895829           10.0            5.6    671.1  11274.3  11274.3  11274.3 stockLevel
		    E200115 08:44:24.692087 1 workload/cli/run.go:444  error in delivery: dial tcp 10.128.1.4:26257: connect: connection refused
		      339.0s  2919981            9.9            5.4   1879.0   6979.3   6979.3   6979.3 delivery
		      339.0s  2919981           14.9           54.5   2281.7   5368.7  10200.5  10200.5 newOrder
		      339.0s  2919981            4.0            5.6    142.6    335.5    335.5    335.5 orderStatus
		      339.0s  2919981           75.4           53.1   4563.4  40802.2  53687.1  62277.0 payment
		      339.0s  2919981            3.0            5.6    176.2    226.5    226.5    226.5 stockLevel
		    E200115 08:44:25.691884 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.1.4:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-16T08:57:45Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1693614-1579160767-11-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    atus
		       71.0s   337725          164.9          105.2    469.8   2013.3   3355.4   3623.9 payment
		       71.0s   337725           18.0           10.7    151.0    385.9    704.6    704.6 stockLevel
		       72.0s   337725           13.0           10.5    704.6   1543.5   2013.3   2013.3 delivery
		       72.0s   337725          128.0          104.8    637.5   1543.5   1677.7   2550.1 newOrder
		       72.0s   337725            7.0           10.7    167.8    260.0    260.0    260.0 orderStatus
		       72.0s   337725          134.0          105.6    436.2   2147.5   3221.2   5100.3 payment
		       72.0s   337725           16.0           10.7    251.7   1073.7   1543.5   1543.5 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		       73.0s   337725           17.0           10.6    805.3   2147.5   3221.2   3221.2 delivery
		       73.0s   337725          120.1          105.0    771.8   1677.7   2818.6   3087.0 newOrder
		       73.0s   337725           11.0           10.7    104.9    167.8    352.3    352.3 orderStatus
		       73.0s   337725          116.1          105.8    469.8   1946.2   2684.4   2952.8 payment
		       73.0s   337725            9.0           10.7    192.9   1208.0   1208.0   1208.0 stockLevel
		       74.0s   337725           17.0           10.7    671.1   1409.3   1744.8   1744.8 delivery
		       74.0s   337725          142.0          105.5    738.2   1677.7   3221.2   4026.5 newOrder
		       74.0s   337725           14.0           10.8     92.3    201.3    234.9    234.9 orderStatus
		       74.0s   337725          146.0          106.3    453.0   3087.0   6442.5   8053.1 payment
		       74.0s   337725           11.0           10.7    226.5    570.4   1409.3   1409.3 stockLevel
		       75.0s   337725           14.0           10.7    570.4    872.4   1543.5   1543.5 delivery
		       75.0s   337725          164.0          106.3    771.8   1946.2   3892.3   5368.7 newOrder
		       75.0s   337725            8.0           10.7    209.7    318.8    318.8    318.8 orderStatus
		       75.0s   337725          120.0          106.5    536.9   2013.3   3087.0   4160.7 payment
		       75.0s   337725            8.0           10.7    130.0    260.0    260.0    260.0 stockLevel
		    E200116 08:57:18.232574 1 workload/cli/run.go:444  error in payment: EOF
		       76.0s   355347            0.0           10.6      0.0      0.0      0.0      0.0 delivery
		       76.0s   355347            0.0          104.9      0.0      0.0      0.0      0.0 newOrder
		       76.0s   355347            0.0           10.6      0.0      0.0      0.0      0.0 orderStatus
		       76.0s   355347            0.0          105.1      0.0      0.0      0.0      0.0 payment
		       76.0s   355347            2.0           10.6    184.5    302.0    302.0    302.0 stockLevel
		    E200116 08:57:19.232735 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.170:26257: connect: connection refused
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		       77.0s   375535            0.0           10.5      0.0      0.0      0.0      0.0 delivery
		       77.0s   375535            0.0          103.5      0.0      0.0      0.0      0.0 newOrder
		       77.0s   375535            0.0           10.5      0.0      0.0      0.0      0.0 orderStatus
		       77.0s   375535            0.0          103.7      0.0      0.0      0.0      0.0 payment
		       77.0s   375535            0.0           10.4      0.0      0.0      0.0      0.0 stockLevel
		    E200116 08:57:20.232763 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.170:26257: connect: connection refused
		       78.0s   396986            0.0           10.3      0.0      0.0      0.0      0.0 delivery
		       78.0s   396986            0.0          102.2      0.0      0.0      0.0      0.0 newOrder
		       78.0s   396986            0.0           10.3      0.0      0.0      0.0      0.0 orderStatus
		       78.0s   396986            0.0          102.4      0.0      0.0      0.0      0.0 payment
		       78.0s   396986            0.0           10.3      0.0      0.0      0.0      0.0 stockLevel
		    E200116 08:57:21.232831 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.170:26257: connect: connection refused:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true

make stressrace TESTS=tpcc/w=100/nodes=3/chaos=true PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-21T20:54:53Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@a5b42be47cf47206d2343f40bbaf445f387f3890:

		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1702345-1579636752-12-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    rStatus
		      498.0s  3625037           54.0           61.7  16643.0  51539.6  57982.1  68719.5 payment
		      498.0s  3625037            3.0            6.4    260.0  12348.0  12348.0  12348.0 stockLevel
		      499.0s  3629317            3.0            6.4  12348.0  25769.8  25769.8  25769.8 delivery
		      499.0s  3629317           21.0           63.6   5100.3  16643.0  49392.1  49392.1 newOrder
		      499.0s  3629317            1.0            6.4    192.9    192.9    192.9    192.9 orderStatus
		      499.0s  3629317           10.0           61.6    151.0  40802.2  40802.2  40802.2 payment
		      499.0s  3629317            2.0            6.4    142.6   3623.9   3623.9   3623.9 stockLevel
		      500.0s  3629317            0.0            6.4      0.0      0.0      0.0      0.0 delivery
		      500.0s  3629317           40.0           63.6  14495.5  20401.1  27917.3  27917.3 newOrder
		      500.0s  3629317            2.0            6.4    209.7    704.6    704.6    704.6 orderStatus
		      500.0s  3629317           38.0           61.6  16643.0  62277.0  90194.3  90194.3 payment
		      500.0s  3629317            3.0            6.4    939.5   4831.8   4831.8   4831.8 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      501.0s  3629317            1.0            6.4   5637.1   5637.1   5637.1   5637.1 delivery
		      501.0s  3629317           66.0           63.6   6442.5  34359.7  62277.0  62277.0 newOrder
		      501.0s  3629317            6.0            6.4    268.4    352.3    352.3    352.3 orderStatus
		      501.0s  3629317           45.0           61.5  17179.9  45097.2  66572.0  66572.0 payment
		      501.0s  3629317            2.0            6.4    704.6   5905.6   5905.6   5905.6 stockLevel
		    E200121 20:54:17.500764 1 workload/cli/run.go:444  error in payment: EOF
		      502.0s  3633115            1.0            6.4   1140.9   1140.9   1140.9   1140.9 delivery
		      502.0s  3633115           49.9           63.5   3758.1  23622.3  25769.8  25769.8 newOrder
		      502.0s  3633115            7.0            6.4    192.9  15569.3  15569.3  15569.3 orderStatus
		      502.0s  3633115           48.9           61.5   4295.0  36507.2  45097.2  45097.2 payment
		      502.0s  3633115            5.0            6.4    704.6  25769.8  25769.8  25769.8 stockLevel
		    E200121 20:54:18.500852 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.14:26257: connect: connection refused
		      503.0s  3657182            7.0            6.4   1610.6   2550.1   2550.1   2550.1 delivery
		      503.0s  3657182           36.9           63.5   9126.8  33286.0  77309.4  77309.4 newOrder
		      503.0s  3657182            4.0            6.4    167.8  19327.4  19327.4  19327.4 orderStatus
		      503.0s  3657182           46.9           61.5   9663.7  45097.2  81604.4  81604.4 payment
		      503.0s  3657182            6.0            6.4   2415.9  24696.1  24696.1  24696.1 stockLevel
		    E200121 20:54:19.500889 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.14:26257: connect: connection refused
		      504.0s  3682619            3.9            6.4   3355.4  24696.1  24696.1  24696.1 delivery
		      504.0s  3682619            1.0           63.4    385.9    385.9    385.9    385.9 newOrder
		      504.0s  3682619            2.0            6.4    159.4    176.2    176.2    176.2 orderStatus
		      504.0s  3682619           55.8           61.5   3221.2  47244.6  49392.1  60129.5 payment
		      504.0s  3682619            2.0            6.4    251.7   2147.5   2147.5   2147.5 stockLevel
		    E200121 20:54:20.500930 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.14:26257: connect: connection refused
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      505.0s  3708187            6.1            6.4   1073.7  11274.3  11274.3  11274.3 delivery
		      505.0s  3708187            1.0           63.2   3221.2   3221.2   3221.2   3221.2 newOrder
		      505.0s  3708187            3.0            6.4    121.6    184.5    184.5    184.5 orderStatus
		      505.0s  3708187           40.5           61.4   4563.4  45097.2  77309.4  77309.4 payment
		      505.0s  3708187            0.0            6.4      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

Repro

Artifacts: /tpcc/w=100/nodes=3/chaos=true
roachdash

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-23T08:48:32Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@545c96573d36519c27115278187fdd441f54a64d:

		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1705428-1579766323-07-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		    87.0s  1477888            6.0            9.9   1040.2  21474.8  21474.8  21474.8 delivery
		      287.0s  1477888           93.0          100.2   1275.1  15569.3  17179.9  20401.1 newOrder
		      287.0s  1477888           12.0           10.1    151.0   8321.5  13958.6  13958.6 orderStatus
		      287.0s  1477888           86.0           98.2   1677.7  21474.8  33286.0  47244.6 payment
		      287.0s  1477888            8.0            9.9    302.0   8589.9   8589.9   8589.9 stockLevel
		      288.0s  1477888            5.0            9.9    738.2   2550.1   2550.1   2550.1 delivery
		      288.0s  1477888           63.1          100.0   1543.5  16643.0  18253.6  18253.6 newOrder
		      288.0s  1477888            2.0           10.0  15569.3  15569.3  15569.3  15569.3 orderStatus
		      288.0s  1477888           60.1           98.1   1946.2  33286.0  40802.2  64424.5 payment
		      288.0s  1477888            2.0            9.9    436.2  15569.3  15569.3  15569.3 stockLevel
		    E200123 08:47:51.437537 1 workload/cli/run.go:444  error in payment: read tcp 10.128.1.21:34416->10.128.1.31:26257: read: connection reset by peer
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      289.0s  1487445            6.9            9.9   1342.2   5637.1   5637.1   5637.1 delivery
		      289.0s  1487445           63.7           99.9   2080.4  10200.5  17179.9  18253.6 newOrder
		      289.0s  1487445            3.9           10.0    268.4   2147.5   2147.5   2147.5 orderStatus
		      289.0s  1487445           50.9           97.9   2550.1  27917.3  31138.5  42949.7 payment
		      289.0s  1487445            8.8            9.9    318.8   2080.4   2080.4   2080.4 stockLevel
		    E200123 08:47:52.438048 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.1.31:26257: connect: connection refused
		      290.0s  1509828            2.0            9.9    704.6   1677.7   1677.7   1677.7 delivery
		      290.0s  1509828           64.0           99.8   2147.5  11811.2  27917.3  31138.5 newOrder
		      290.0s  1509828           10.2           10.0    335.5   6979.3   6979.3   6979.3 orderStatus
		      290.0s  1509828           57.9           97.8   2952.8  23622.3  26843.5  30064.8 payment
		      290.0s  1509828            8.1            9.9    318.8   2281.7   2281.7   2281.7 stockLevel
		    E200123 08:47:53.438201 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.1.31:26257: connect: connection refused
		      291.0s  1532982           11.8            9.9   1811.9   5368.7   7516.2   7516.2 delivery
		      291.0s  1532982           83.7           99.7   2281.7   6174.0   9663.7  22548.6 newOrder
		      291.0s  1532982            4.9           10.0    176.2    234.9    234.9    234.9 orderStatus
		      291.0s  1532982           74.9           97.7   2550.1  27917.3  45097.2  60129.5 payment
		      291.0s  1532982            3.9            9.9    218.1   3087.0   3087.0   3087.0 stockLevel
		    E200123 08:47:54.438344 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.1.31:26257: connect: connection refused
		      292.0s  1557717           10.0            9.9   1275.1   4563.4   4563.4   4563.4 delivery
		      292.0s  1557717           67.2           99.6   1744.8   8053.1   9663.7  13958.6 newOrder
		      292.0s  1557717           13.0           10.0    192.9   1879.0   1946.2   1946.2 orderStatus
		      292.0s  1557717           90.2           97.7   4026.5  47244.6  77309.4  81604.4 payment
		      292.0s  1557717            7.0            9.9    226.5   2080.4   2080.4   2080.4 stockLevel
		    E200123 08:47:55.438442 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.1.31:26257: connect: connection refused
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      293.0s  1580149            7.1            9.9   2147.5   7247.8   7247.8   7247.8 delivery
		      293.0s  1580149           97.1           99.6   1610.6   6442.5  22548.6  26843.5 newOrder
		      293.0s  1580149           11.1           10.0    176.2    218.1    234.9    234.9 orderStatus
		      293.0s  1580149           99.1           97.7   1610.6  22548.6  30064.8  45097.2 payment
		      293.0s  1580149           14.2            9.9    243.3   1476.4   2952.8   2952.8 stockLevel:
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-25T09:01:31Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@72ce65798f55af352c6a38c306542f63dfba465f:

		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1710860-1579939958-01-n4cpu4:4 -- ./workload run tpcc --warehouses=100 --histograms=perf/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=30m0s {pgurl:1-3} returned:
		    stderr:
		    
		    stdout:
		                6.6    637.5   3758.1   3758.1   3758.1 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      505.0s  3716909            5.0            6.6   1879.0   7784.6   7784.6   7784.6 delivery
		      505.0s  3716909           63.0           66.4   2415.9  20401.1  21474.8  24696.1 newOrder
		      505.0s  3716909            4.0            6.7    771.8   2818.6   2818.6   2818.6 orderStatus
		      505.0s  3716909           66.0           64.1   2818.6  49392.1  77309.4 103079.2 payment
		      505.0s  3716909            4.0            6.6    285.2   1677.7   1677.7   1677.7 stockLevel
		      506.0s  3716909            8.0            6.6   2281.7  20401.1  20401.1  20401.1 delivery
		      506.0s  3716909           71.0           66.4   2281.7  10737.4  20401.1  21474.8 newOrder
		      506.0s  3716909           19.0            6.7  20401.1  20401.1  20401.1  20401.1 orderStatus
		      506.0s  3716909           49.0           64.1   3355.4  34359.7  53687.1  53687.1 payment
		      506.0s  3716909           12.0            6.7   4831.8  20401.1  20401.1  20401.1 stockLevel
		      507.0s  3716909            4.0            6.6   1744.8   2013.3   2013.3   2013.3 delivery
		      507.0s  3716909           63.0           66.4   3221.2  24696.1  27917.3  33286.0 newOrder
		      507.0s  3716909            9.0            6.7    385.9   4026.5   4026.5   4026.5 orderStatus
		      507.0s  3716909           54.0           64.0   3489.7  25769.8  32212.3  47244.6 payment
		      507.0s  3716909            2.0            6.6    268.4   4563.4   4563.4   4563.4 stockLevel
		    E200125 09:00:58.257943 1 workload/cli/run.go:444  error in payment: EOF
		      508.0s  3735579            6.0            6.6   1744.8   5100.3   5100.3   5100.3 delivery
		      508.0s  3735579           77.0           66.4   1946.2   7516.2  12348.0  31138.5 newOrder
		      508.0s  3735579            4.0            6.7    176.2   2952.8   2952.8   2952.8 orderStatus
		      508.0s  3735579           81.0           64.1   3355.4  38654.7  51539.6  51539.6 payment
		      508.0s  3735579            8.0            6.6    385.9   3892.3   3892.3   3892.3 stockLevel
		    E200125 09:00:59.258707 1 workload/cli/run.go:444  error in payment: dial tcp 10.128.0.153:26257: connect: connection refused
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      509.0s  3760141            9.9            6.6   1476.4   5100.3   5100.3   5100.3 delivery
		      509.0s  3760141           94.6           66.5   1744.8  12884.9  23622.3  24696.1 newOrder
		      509.0s  3760141           12.8            6.7    104.9   1677.7   5100.3   5100.3 orderStatus
		      509.0s  3760141           70.9           64.1   1040.2  28991.0  47244.6  49392.1 payment
		      509.0s  3760141            9.9            6.7    335.5   5637.1   5637.1   5637.1 stockLevel
		    E200125 09:01:00.258803 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.153:26257: connect: connection refused
		      510.0s  3783711           11.1            6.6   1342.2   5100.3   5905.6   5905.6 delivery
		      510.0s  3783711            9.1           66.4   2013.3  13421.8  13421.8  13421.8 newOrder
		      510.0s  3783711            6.0            6.7    113.2   4831.8   4831.8   4831.8 orderStatus
		      510.0s  3783711           87.6           64.1   3623.9  38654.7  60129.5  60129.5 payment
		      510.0s  3783711            3.0            6.6    151.0    285.2    285.2    285.2 stockLevel
		    E200125 09:01:01.258920 1 workload/cli/run.go:444  error in newOrder: dial tcp 10.128.0.153:26257: connect: connection refused
		      511.0s  3809004            6.1            6.6   2684.4   5100.3   5100.3   5100.3 delivery
		      511.0s  3809004            0.0           66.2      0.0      0.0      0.0      0.0 newOrder
		      511.0s  3809004            4.0            6.7     14.2   2684.4   2684.4   2684.4 orderStatus
		      511.0s  3809004           32.3           64.1   3221.2  30064.8  81604.4  81604.4 payment
		      511.0s  3809004            0.0            6.6      0.0      0.0      0.0      0.0 stockLevel
		    E200125 09:01:02.259032 1 workload/cli/run.go:444  error in stockLevel: dial tcp 10.128.0.153:26257: connect: connection refused:
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-03T21:57:34Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@4ec5929a08687b82b05ef80231fddd0d53421973:

		     1214.0s  5947623            4.0            7.2    159.4   1610.6   1610.6   1610.6 stockLevel
		     1215.0s  5947623            9.0            7.2   1543.5   6174.0   6174.0   6174.0 delivery
		     1215.0s  5947623          109.9           71.7   1744.8  10737.4  18253.6  25769.8 newOrder
		     1215.0s  5947623           12.0            7.3    142.6   7516.2   9126.8   9126.8 orderStatus
		     1215.0s  5947623          115.9           69.9   1610.6  25769.8  49392.1  51539.6 payment
		     1215.0s  5947623           11.0            7.2    302.0   4563.4   6979.3   6979.3 stockLevel
		     1216.0s  5947623           10.0            7.2   1744.8  31138.5  31138.5  31138.5 delivery
		     1216.0s  5947623          110.1           71.7   1543.5  24696.1  49392.1  49392.1 newOrder
		     1216.0s  5947623            4.0            7.3    167.8    285.2    285.2    285.2 orderStatus
		     1216.0s  5947623          115.1           70.0   2952.8  26843.5  49392.1  66572.0 payment
		     1216.0s  5947623            9.0            7.2    234.9   7247.8   7247.8   7247.8 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1217.0s  5947623           12.0            7.2   1677.7   3892.3  20401.1  20401.1 delivery
		     1217.0s  5947623          124.0           71.8   1543.5  10737.4  21474.8  24696.1 newOrder
		     1217.0s  5947623           10.0            7.3    159.4   8321.5   8321.5   8321.5 orderStatus
		     1217.0s  5947623          100.0           70.0   2818.6  30064.8  51539.6  77309.4 payment
		     1217.0s  5947623           21.0            7.3    671.1   9126.8   9126.8   9126.8 stockLevel
		     1218.0s  5947623           11.0            7.2   1342.2   3623.9   5368.7   5368.7 delivery
		     1218.0s  5947623           99.9           71.8   1879.0   9126.8  12884.9  20401.1 newOrder:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func3
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2416
		    main.(*monitor).wait.func4
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2445
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor command failure:
		  - signal: broken pipe

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1725352-1580762820-06-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 12239
		4: skipped
		1: dead
		3: 11160
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T10:20:12Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@b67d7d14e22f227caa4a2bc3727e3baf815fb25e:

		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      121.0s   644121            7.8           11.3   1409.3   2952.8   2952.8   2952.8 delivery
		      121.0s   644121          103.0          112.6    872.4   3221.2   4026.5   4563.4 newOrder
		      121.0s   644121            6.9           11.5    151.0    268.4    268.4    268.4 orderStatus
		      121.0s   644121          104.0          113.1    704.6   4026.5   5637.1   5905.6 payment
		      121.0s   644121            8.8           11.3    176.2    906.0    906.0    906.0 stockLevel
		      122.0s   661704           18.4           11.4    805.3   2952.8   4831.8   4831.8 delivery
		      122.0s   661704          104.1          112.5    771.8   2147.5   2684.4   3489.7 newOrder
		      122.0s   661704            5.1           11.4     92.3    192.9    192.9    192.9 orderStatus
		      122.0s   661704          100.0          113.0    604.0   3489.7   5905.6   8321.5 payment
		      122.0s   661704            3.1           11.2    104.9    318.8    318.8    318.8 stockLevel
		      123.0s   681845           10.0           11.4    838.9   3087.0   3087.0   3087.0 delivery
		      123.0s   681845          106.0          112.4    738.2   1811.9   2013.3   2415.9 newOrder
		      123.0s   681845            5.0           11.3    134.2    151.0    151.0    151.0 orderStatus
		      123.0s   681845          108.0          112.9    838.9   5905.6   7516.2   7516.2 payment
		      123.0s   681845           15.0           11.3    352.3    906.0   1744.8   1744.8 stockLevel
		      124.0s   703436           13.0           11.4    906.0   2684.4   3758.1   3758.1 delivery
		      124.0s   703436          115.9          112.5    704.6   1543.5   3221.2   3221.2 newOrder
		      124.0s   703436           14.0           11.4     79.7    117.4    151.0    151.0 orderStatus
		      124.0s   703436          111.9          112.9    637.5   3892.3   5368.7   5368.7 payment
		      124.0s   703436           10.0           11.3    260.0    872.4    872.4    872.4 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      125.0s   724786           10.0           11.4    838.9   1946.2   1946.2   1946.2 delivery
		      125.0s   724786           95.8          112.3    671.1   1476.4   1677.7   1946.2 newOrder
		      125.0s   724786           12.0           11.4     79.7    167.8    192.9    192.9 orderStatus
		      125.0s   724786          106.8          112.9    671.1   5637.1   7247.8   9126.8 payment
		      125.0s   724786           12.0           11.3    226.5    369.1   1610.6   1610.6 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 2: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736162-1581070194-02-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: 3459
		2: dead
		3: 3351
		Error:  2: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T10:28:09Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@b67d7d14e22f227caa4a2bc3727e3baf815fb25e:

		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1694
		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:127
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - cluster.Put:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).PutE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1708
		    main.(*cluster).Put
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1694
		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:127
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1736186-1581071065-03-n4cpu4:4 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach returned:
		    stderr:
		    
		    stdout:
		    teamcity-1736186-1581071065-03-n4cpu4: putting (dist) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 ./cockroach
		    .........
		       4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 [email protected]:./cockroach
		    Warning: Permanently added '34.67.238.213' (ECDSA) to the list of known hosts.
		    packet_write_wait: Connection to 34.67.238.213 port 22: Broken pipe
		    lost connection
		    : exit status 1
		    I200207 10:27:55.370728 1 cluster_synced.go:1100  put /home/agent/work/.go/src/github.com/cockroachdb/cockroach/cockroach.linux-2.6.32-gnu-amd64 failed:
		  - exit status 1

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T10:44:46Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@b67d7d14e22f227caa4a2bc3727e3baf815fb25e:

		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1245.0s  7047321            6.0            8.8   1275.1   1811.9   1811.9   1811.9 delivery
		     1245.0s  7047321          105.0           87.4   1946.2  23622.3  26843.5  27917.3 newOrder
		     1245.0s  7047321            9.0            8.8    125.8    285.2    285.2    285.2 orderStatus
		     1245.0s  7047321           83.0           85.3   1744.8  32212.3  42949.7  45097.2 payment
		     1245.0s  7047321           11.0            8.8    486.5   1610.6   5637.1   5637.1 stockLevel
		     1246.0s  7054658           11.9            8.8   1140.9   6979.3  12348.0  12348.0 delivery
		     1246.0s  7054658          116.9           87.4   1476.4  21474.8  24696.1  25769.8 newOrder
		     1246.0s  7054658            7.9            8.8     96.5  20401.1  20401.1  20401.1 orderStatus
		     1246.0s  7054658           64.4           85.2   2147.5  27917.3  34359.7  38654.7 payment
		     1246.0s  7054658            7.9            8.8    402.7   5905.6   5905.6   5905.6 stockLevel
		     1247.0s  7078210           10.1            8.8    906.0   5368.7   5368.7   5368.7 delivery
		     1247.0s  7078210           85.7           87.4   1140.9   9126.8  23622.3  24696.1 newOrder
		     1247.0s  7078210            7.1            8.8     96.5   5637.1   5637.1   5637.1 orderStatus
		     1247.0s  7078210           87.7           85.2   1208.0  24696.1  31138.5  60129.5 payment
		     1247.0s  7078210           10.1            8.8    184.5  22548.6  22548.6  22548.6 stockLevel
		     1248.0s  7100765            9.0            8.8    771.8   6174.0   6174.0   6174.0 delivery
		     1248.0s  7100765          118.5           87.5    671.1   5637.1  22548.6  24696.1 newOrder
		     1248.0s  7100765            6.0            8.8     71.3    151.0    151.0    151.0 orderStatus
		     1248.0s  7100765          102.6           85.3    805.3  22548.6  28991.0  55834.6 payment
		     1248.0s  7100765           10.0            8.8    167.8   1543.5   1543.5   1543.5 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1249.0s  7124329            7.0            8.8    671.1   1811.9   1811.9   1811.9 delivery
		     1249.0s  7124329           73.3           87.4    469.8   3623.9   4831.8  27917.3 newOrder
		     1249.0s  7124329            8.0            8.8    117.4    184.5    184.5    184.5 orderStatus
		     1249.0s  7124329           78.3           85.3    771.8  10737.4  49392.1  81604.4 payment
		     1249.0s  7124329           10.0            8.8    142.6   3892.3   3892.3   3892.3 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 1: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736162-1581070194-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: dead
		3: 8584
		2: 6869
		4: skipped
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T10:57:33Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@b67d7d14e22f227caa4a2bc3727e3baf815fb25e:

		     1129.0s  5967981           11.0            6.7    226.5    570.4  23622.3  23622.3 stockLevel
		     1130.0s  5973210            9.7            6.6   1677.7  18253.6  18253.6  18253.6 delivery
		     1130.0s  5973210           56.1           66.0   1006.6  30064.8  33286.0  36507.2 newOrder
		     1130.0s  5973210            3.9            6.7    176.2    218.1    218.1    218.1 orderStatus
		     1130.0s  5973210           60.9           64.2   1476.4  36507.2  45097.2  73014.4 payment
		     1130.0s  5973210            4.8            6.7    469.8  17179.9  17179.9  17179.9 stockLevel
		     1131.0s  5994363            4.1            6.6   1543.5  19327.4  19327.4  19327.4 delivery
		     1131.0s  5994363           91.9           66.0   1677.7  19327.4  26843.5  31138.5 newOrder
		     1131.0s  5994363            6.2            6.7    125.8  40802.2  40802.2  40802.2 orderStatus
		     1131.0s  5994363           84.7           64.2   1275.1  47244.6  64424.5  77309.4 payment
		     1131.0s  5994363            6.2            6.7    268.4   1879.0   1879.0   1879.0 stockLevel
		     1132.0s  6015778            7.9            6.6   1677.7  11274.3  11274.3  11274.3 delivery
		     1132.0s  6015778           62.6           66.0   1040.2  27917.3  28991.0  60129.5 newOrder
		     1132.0s  6015778            8.9            6.7    209.7    385.9    385.9    385.9 orderStatus
		     1132.0s  6015778           55.6           64.2   1610.6  22548.6  64424.5  64424.5 payment
		     1132.0s  6015778            7.0            6.7    218.1    604.0    604.0    604.0 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1133.0s  6037359           10.0            6.6   1946.2  12884.9  12884.9  12884.9 delivery
		     1133.0s  6037359           77.4           66.0   1744.8  40802.2  45097.2  45097.2 newOrder
		     1133.0s  6037359            3.0            6.7    192.9   8589.9   8589.9   8589.9 orderStatus
		     1133.0s  6037359           56.3           64.2   1073.7  42949.7  57982.1  60129.5 payment
		     1133.0s  6037359           11.1            6.7    318.8    637.5   1677.7   1677.7 stockLevel
		     1134.0s  6059480            7.0            6.6   1409.3   3623.9   3623.9   3623.9 delivery
		     1134.0s  6059480           62.8           66.0   1275.1  26843.5  33286.0  49392.1 newOrder
		     1134.0s  6059480            4.0            6.7    260.0    302.0    302.0    302.0 orderStatus
		     1134.0s  6059480           73.7           64.2   3355.4  42949.7  57982.1 103079.2 payment
		     1134.0s  6059480            7.0            6.7    369.1   2952.8   2952.8   2952.8 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 2: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736186-1581071065-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: 6448
		1: 7951
		4: skipped
		2: dead
		Error:  2: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T11:00:46Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@b67d7d14e22f227caa4a2bc3727e3baf815fb25e:

		     1302.0s  6865864            5.0            7.0  18253.6  18253.6  18253.6  18253.6 stockLevel
		     1303.0s  6868250            5.0            6.9   2147.5   2415.9   2415.9   2415.9 delivery
		     1303.0s  6868250           63.7           69.3   2080.4  23622.3  26843.5  27917.3 newOrder
		     1303.0s  6868250            5.0            7.1  19327.4  19327.4  19327.4  19327.4 orderStatus
		     1303.0s  6868250           53.8           67.6   2415.9  25769.8  27917.3  33286.0 payment
		     1303.0s  6868250            3.0            7.0  18253.6  19327.4  19327.4  19327.4 stockLevel
		     1304.0s  6892147            9.0            6.9   1610.6  16643.0  16643.0  16643.0 delivery
		     1304.0s  6892147           82.3           69.3   1409.3  18253.6  38654.7  40802.2 newOrder
		     1304.0s  6892147            8.0            7.1    151.0    302.0    302.0    302.0 orderStatus
		     1304.0s  6892147           87.3           67.6   2080.4  38654.7  47244.6  51539.6 payment
		     1304.0s  6892147            7.0            7.0    226.5   2013.3   2013.3   2013.3 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     1305.0s  6915460            4.9            6.9   1409.3   1677.7   1677.7   1677.7 delivery
		     1305.0s  6915460           63.1           69.3   1543.5  12884.9  18253.6  20401.1 newOrder
		     1305.0s  6915460            8.9            7.1    184.5  12884.9  12884.9  12884.9 orderStatus
		     1305.0s  6915460           64.1           67.6   2952.8  23622.3  32212.3  66572.0 payment
		     1305.0s  6915460            4.9            7.0    285.2    352.3    352.3    352.3 stockLevel
		     1306.0s  6939186            4.0            6.9   2415.9  26843.5  26843.5  26843.5 delivery
		     1306.0s  6939186           75.1           69.3   1610.6  18253.6  22548.6  24696.1 newOrder
		     1306.0s  6939186           13.0            7.1    184.5  11811.2  12348.0  12348.0 orderStatus
		     1306.0s  6939186           67.1           67.6   1275.1  27917.3  33286.0  45097.2 payment
		     1306.0s  6939186            5.0            6.9    402.7   7247.8   7247.8   7247.8 stockLevel
		     1307.0s  6962003            2.0            6.9    486.5    872.4    872.4    872.4 delivery
		     1307.0s  6962003           97.1           69.3   1342.2  11274.3  13421.8  14495.5 newOrder
		     1307.0s  6962003            4.0            7.1     83.9    109.1    109.1    109.1 orderStatus
		     1307.0s  6962003           73.8           67.6   2415.9  21474.8  23622.3  26843.5 payment
		     1307.0s  6962003            8.1            7.0    486.5   1208.0   1208.0   1208.0 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 1: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736186-1581071065-04-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		3: 8550
		2: 5428
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T16:25:08Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@47b425fe9385f110e4b29b79208a638cc8f59f4d:

		      793.0s  3979439           13.0            7.1  21474.8  22548.6  22548.6  22548.6 stockLevel
		      794.0s  3989481            1.0            7.0   1006.6   1006.6   1006.6   1006.6 delivery
		      794.0s  3989481           56.9           70.3   2281.7   9126.8   9663.7  23622.3 newOrder
		      794.0s  3989481            7.0            7.2    209.7   6710.9   6710.9   6710.9 orderStatus
		      794.0s  3989481           60.9           68.6   2415.9  30064.8  42949.7  81604.4 payment
		      794.0s  3989481           11.0            7.1   1342.2   7784.6  14495.5  14495.5 stockLevel
		      795.0s  3993238           11.0            7.0   1409.3   9126.8  25769.8  25769.8 delivery
		      795.0s  3993238           62.2           70.3   2281.7  11274.3  16643.0  24696.1 newOrder
		      795.0s  3993238            7.0            7.2     96.5    369.1    369.1    369.1 orderStatus
		      795.0s  3993238           46.1           68.6   3489.7  53687.1  60129.5  60129.5 payment
		      795.0s  3993238            4.0            7.1    159.4   2281.7   2281.7   2281.7 stockLevel
		      796.0s  3993238            6.0            7.0   1409.3  25769.8  25769.8  25769.8 delivery
		      796.0s  3993238           62.9           70.3   1744.8   9663.7  17179.9  25769.8 newOrder
		      796.0s  3993238            9.0            7.2    109.1  15032.4  15032.4  15032.4 orderStatus
		      796.0s  3993238           71.9           68.6   2415.9  17179.9  31138.5  33286.0 payment
		      796.0s  3993238            6.0            7.1    117.4   6979.3   6979.3   6979.3 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      797.0s  3993238           10.0            7.0   1073.7   8053.1   8053.1   8053.1 delivery
		      797.0s  3993238           73.2           70.3   1677.7   7247.8  12884.9  13421.8 newOrder
		      797.0s  3993238            6.0            7.2     92.3    218.1    218.1    218.1 orderStatus
		      797.0s  3993238           70.1           68.6   1275.1  36507.2  55834.6  64424.5 payment
		      797.0s  3993238            5.0            7.1    226.5   8589.9   8589.9   8589.9 stockLevel
		      798.0s  3993238            8.0            7.0    838.9  13421.8  13421.8  13421.8 delivery
		      798.0s  3993238           66.0           70.3   1476.4   8321.5   9663.7   9663.7 newOrder
		      798.0s  3993238            7.0            7.2    113.2    201.3    201.3    201.3 orderStatus
		      798.0s  3993238           72.0           68.6   2147.5  32212.3  40802.2  62277.0 payment
		      798.0s  3993238            4.0            7.1    243.3   2013.3   2013.3   2013.3 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 1: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736338-1581091138-03-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		2: 6666
		3: 6527
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T18:01:41Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@f9537b15c70f6e005cc003b852f39415db7cea64:

		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      289.0s  1084510            9.0            9.1   1811.9  18253.6  18253.6  18253.6 delivery
		      289.0s  1084510           85.0           90.0   3221.2  22548.6  24696.1  26843.5 newOrder
		      289.0s  1084510            5.0            9.1    159.4   1208.0   1208.0   1208.0 orderStatus
		      289.0s  1084510           56.0           88.3   7516.2  42949.7  47244.6  77309.4 payment
		      289.0s  1084510            6.0            9.1    268.4   3087.0   3087.0   3087.0 stockLevel
		      290.0s  1101814            4.9            9.1   1744.8   2550.1   2550.1   2550.1 delivery
		      290.0s  1101814           57.1           89.9   2080.4  18253.6  20401.1  21474.8 newOrder
		      290.0s  1101814            6.9            9.1    352.3  18253.6  18253.6  18253.6 orderStatus
		      290.0s  1101814           70.9           88.2   2147.5  45097.2  55834.6  77309.4 payment
		      290.0s  1101814            6.9            9.1    285.2  13421.8  13421.8  13421.8 stockLevel
		      291.0s  1126162            6.0            9.1   1677.7   5905.6   5905.6   5905.6 delivery
		      291.0s  1126162           65.7           89.8   2013.3  20401.1  21474.8  23622.3 newOrder
		      291.0s  1126162            5.0            9.1    369.1   1610.6   1610.6   1610.6 orderStatus
		      291.0s  1126162           51.7           88.1   2013.3  36507.2  40802.2  53687.1 payment
		      291.0s  1126162            2.0            9.1   1275.1  19327.4  19327.4  19327.4 stockLevel
		      292.0s  1150052            5.0            9.0   2684.4   2952.8   2952.8   2952.8 delivery
		      292.0s  1150052           50.3           89.7   2147.5  21474.8  30064.8  30064.8 newOrder
		      292.0s  1150052           10.1            9.1    285.2   1610.6   1610.6   1610.6 orderStatus
		      292.0s  1150052           49.3           88.0   6174.0  57982.1  60129.5  60129.5 payment
		      292.0s  1150052            8.0            9.1    906.0   1543.5   1543.5   1543.5 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      293.0s  1173661            7.9            9.0   1610.6   3623.9   3623.9   3623.9 delivery
		      293.0s  1173661           72.5           89.6   2147.5   6442.5  20401.1  25769.8 newOrder
		      293.0s  1173661            8.9            9.1    159.4   3355.4   3355.4   3355.4 orderStatus
		      293.0s  1173661           60.6           87.9   2080.4  45097.2  53687.1  60129.5 payment
		      293.0s  1173661            7.9            9.1    486.5   3489.7   3489.7   3489.7 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 1: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736443-1581097734-02-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: 3528
		1: dead
		2: 4774
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-07T20:10:19Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@f9537b15c70f6e005cc003b852f39415db7cea64:

		      157.0s  2330679           11.0            7.1    637.5   4563.4  18253.6  18253.6 stockLevel
		      158.0s  2339826            3.9            6.9   1073.7   1744.8   1744.8   1744.8 delivery
		      158.0s  2339826           75.9           67.4   1811.9  18253.6  19327.4  31138.5 newOrder
		      158.0s  2339826            3.9            7.2    159.4  16643.0  16643.0  16643.0 orderStatus
		      158.0s  2339826           66.1           64.7   1744.8  45097.2  53687.1  73014.4 payment
		      158.0s  2339826            9.9            7.1    604.0  14495.5  14495.5  14495.5 stockLevel
		      159.0s  2362993           11.1            6.9   1342.2   5100.3  24696.1  24696.1 delivery
		      159.0s  2362993           85.0           67.5   1879.0  18253.6  24696.1  32212.3 newOrder
		      159.0s  2362993            2.0            7.2    142.6    302.0    302.0    302.0 orderStatus
		      159.0s  2362993           67.8           64.7   1476.4  36507.2  81604.4  85899.3 payment
		      159.0s  2362993            4.0            7.1    260.0    771.8    771.8    771.8 stockLevel
		      160.0s  2385218            9.0            6.9   1140.9   3355.4   3355.4   3355.4 delivery
		      160.0s  2385218           86.1           67.6   1342.2  24696.1  32212.3  34359.7 newOrder
		      160.0s  2385218            7.0            7.2    184.5   9663.7   9663.7   9663.7 orderStatus
		      160.0s  2385218           72.1           64.7   1073.7  21474.8  25769.8  42949.7 payment
		      160.0s  2385218            5.0            7.1    260.0    536.9    536.9    536.9 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      161.0s  2408709            4.0            6.9    973.1   5637.1   5637.1   5637.1 delivery
		      161.0s  2408709           55.5           67.6   1879.0  19327.4  20401.1  30064.8 newOrder
		      161.0s  2408709            6.0            7.2    142.6    234.9    234.9    234.9 orderStatus
		      161.0s  2408709           66.5           64.7   1543.5  33286.0  51539.6  81604.4 payment
		      161.0s  2408709            4.0            7.0    385.9   1879.0   1879.0   1879.0 stockLevel
		      162.0s  2433125            8.0            6.9   2415.9   4160.7   4160.7   4160.7 delivery
		      162.0s  2433125            3.0           67.2   4026.5   7247.8   7247.8   7247.8 newOrder
		      162.0s  2433125            8.0            7.2    176.2   9126.8   9126.8   9126.8 orderStatus
		      162.0s  2433125           83.6           64.9   1811.9  40802.2  77309.4  81604.4 payment
		      162.0s  2433125            3.0            7.0    226.5    637.5    637.5    637.5 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 3: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1736827-1581105243-05-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		3: dead
		1: 4265
		2: 5179
		Error:  3: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-13T08:45:17Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@2739821b911d777fa2a927295d699b559360a802:

		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_084135.006_n4_workload_fixtures_load_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.setupTPCC.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:136
		    main.setupTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:138
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:149
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-06-n4cpu4:4 -- ./workload fixtures load tpcc --warehouses=100  {pgurl:1} returned:
		    stderr:
		    I200213 08:41:37.023783 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7
		    Error:  exit status 1
		    
		    stdout::
		  - exit status 1

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-14T09:17:31Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@e671a4ef97cbc6cf5d22f8f322fd45733d302094:

		      229.0s  1171845            0.0            8.9      0.0      0.0      0.0      0.0 stockLevel
		      230.0s  1171845            0.0            8.7      0.0      0.0      0.0      0.0 delivery
		      230.0s  1171845            1.0           87.6  18253.6  18253.6  18253.6  18253.6 newOrder
		      230.0s  1171845            0.0            8.9      0.0      0.0      0.0      0.0 orderStatus
		      230.0s  1171845            0.0           87.4      0.0      0.0      0.0      0.0 payment
		      230.0s  1171845            2.0            8.9  14495.5  15569.3  15569.3  15569.3 stockLevel
		      231.0s  1187393            4.0            8.7  16643.0  17179.9  17179.9  17179.9 delivery
		      231.0s  1187393            0.0           87.2      0.0      0.0      0.0      0.0 newOrder
		      231.0s  1187393            0.0            8.8      0.0      0.0      0.0      0.0 orderStatus
		      231.0s  1187393           28.0           87.2  18253.6  22548.6  23622.3  23622.3 payment
		      231.0s  1187393            0.0            8.9      0.0      0.0      0.0      0.0 stockLevel
		      232.0s  1199601            1.0            8.7  18253.6  18253.6  18253.6  18253.6 delivery
		      232.0s  1199601            0.0           86.9      0.0      0.0      0.0      0.0 newOrder
		      232.0s  1199601            0.0            8.8      0.0      0.0      0.0      0.0 orderStatus
		      232.0s  1199601            0.0           86.8      0.0      0.0      0.0      0.0 payment
		      232.0s  1199601            1.0            8.8     12.1     12.1     12.1     12.1 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      233.0s  1212135            2.0            8.6     56.6     92.3     92.3     92.3 delivery
		      233.0s  1212135            0.0           86.5      0.0      0.0      0.0      0.0 newOrder
		      233.0s  1212135            0.0            8.7      0.0      0.0      0.0      0.0 orderStatus
		      233.0s  1212135            4.0           86.4  19327.4  22548.6  22548.6  22548.6 payment
		      233.0s  1212135            0.0            8.8      0.0      0.0      0.0      0.0 stockLevel
		      234.0s  1227234            0.0            8.6      0.0      0.0      0.0      0.0 delivery
		      234.0s  1227234           21.0           86.2  20401.1  24696.1  26843.5  26843.5 newOrder
		      234.0s  1227234            2.0            8.7  19327.4  19327.4  19327.4  19327.4 orderStatus
		      234.0s  1227234           30.0           86.2  20401.1  26843.5  27917.3  27917.3 payment
		      234.0s  1227234            4.0            8.8     54.5  21474.8  21474.8  21474.8 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 3: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1749385-1581667748-04-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: 9923
		3: dead
		1: 9796
		Error:  3: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-03T18:42:58Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@5f9a71adb995837bcff27b9456188018434be4b8:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200303-1780216/tpcc/w=100/nodes=3/chaos=true/run_1
	test_runner.go:756: test timed out (10h0m0s)

	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_092330.412_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1780216-1583223197-15-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200303 09:23:32.046722 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 24.060931ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-06T18:41:34Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@954fe69d554162aec0fbc001aad1fe5103d8df13:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200306-1790595/tpcc/w=100/nodes=3/chaos=true/run_1
	test_runner.go:756: test timed out (10h0m0s)

	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_092121.305_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1790595-1583481796-06-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-07T08:50:02Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@752dea867f3aeb142e98c22f8d320ce19041aa8d:

		      203.0s   883386            8.0            7.7    385.9    805.3    805.3    805.3 stockLevel
		      204.0s   883386            5.0            7.5   1543.5   2684.4   2684.4   2684.4 delivery
		      204.0s   883386           89.0           75.6   1879.0   5368.7  24696.1  26843.5 newOrder
		      204.0s   883386           10.0            7.7    125.8    536.9    536.9    536.9 orderStatus
		      204.0s   883386           73.0           73.9   2281.7   7247.8  16643.0  25769.8 payment
		      204.0s   883386            7.0            7.7   1208.0   3355.4   3355.4   3355.4 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      205.0s   896493            0.0            7.5      0.0      0.0      0.0      0.0 delivery
		      205.0s   896493            2.0           75.3   1275.1   5100.3   5100.3   5100.3 newOrder
		      205.0s   896493            0.0            7.7      0.0      0.0      0.0      0.0 orderStatus
		      205.0s   896493            3.0           73.5   2952.8   3489.7   3489.7   3489.7 payment
		      205.0s   896493            3.0            7.6  28991.0  28991.0  28991.0  28991.0 stockLevel
		      206.0s   909162            0.0            7.4      0.0      0.0      0.0      0.0 delivery
		      206.0s   909162            0.0           74.9      0.0      0.0      0.0      0.0 newOrder
		      206.0s   909162            0.0            7.6      0.0      0.0      0.0      0.0 orderStatus
		      206.0s   909162            0.0           73.2      0.0      0.0      0.0      0.0 payment
		      206.0s   909162            0.0            7.6      0.0      0.0      0.0      0.0 stockLevel
		      207.0s   918101            0.0            7.4      0.0      0.0      0.0      0.0 delivery
		      207.0s   918101            0.0           74.6      0.0      0.0      0.0      0.0 newOrder
		      207.0s   918101            0.0            7.6      0.0      0.0      0.0      0.0 orderStatus
		      207.0s   918101            0.0           72.8      0.0      0.0      0.0      0.0 payment
		      207.0s   918101            0.0            7.6      0.0      0.0      0.0      0.0 stockLevel
		      208.0s   931195            0.0            7.4      0.0      0.0      0.0      0.0 delivery
		      208.0s   931195            0.0           74.2      0.0      0.0      0.0      0.0 newOrder
		      208.0s   931195            0.0            7.6      0.0      0.0      0.0      0.0 orderStatus
		      208.0s   931195            0.0           72.5      0.0      0.0      0.0      0.0 payment
		      208.0s   931195            0.0            7.5      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 1: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1793930-1583568484-08-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: dead
		2: 7923
		3: 6592
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-08T08:53:03Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@dfa5bd527ae7d7373dd03c62118df87a87a77130:

		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		       57.0s  1243878           13.0            6.4   1275.1   8321.5  10200.5  10200.5 delivery
		       57.0s  1243878          105.0           60.8   1006.6   7784.6  10737.4  11274.3 newOrder
		       57.0s  1243878           15.0            6.3    142.6   3221.2   7516.2   7516.2 orderStatus
		       57.0s  1243878           88.0           54.3   1677.7  27917.3  31138.5  36507.2 payment
		       57.0s  1243878            5.0            6.3   2684.4   8321.5   8321.5   8321.5 stockLevel
		       58.0s  1243878            1.0            6.3    520.1    520.1    520.1    520.1 delivery
		       58.0s  1243878           44.0           60.5   1342.2   8321.5  28991.0  28991.0 newOrder
		       58.0s  1243878            4.0            6.3    121.6    176.2    176.2    176.2 orderStatus
		       58.0s  1243878           44.0           54.2   1275.1  33286.0  51539.6  51539.6 payment
		       58.0s  1243878            9.0            6.3    352.3   9126.8   9126.8   9126.8 stockLevel
		       59.0s  1257939            0.0            6.2      0.0      0.0      0.0      0.0 delivery
		       59.0s  1257939            0.0           59.5      0.0      0.0      0.0      0.0 newOrder
		       59.0s  1257939            0.0            6.2      0.0      0.0      0.0      0.0 orderStatus
		       59.0s  1257939            0.0           53.2      0.0      0.0      0.0      0.0 payment
		       59.0s  1257939            0.0            6.2      0.0      0.0      0.0      0.0 stockLevel
		       60.0s  1257939            0.0            6.1      0.0      0.0      0.0      0.0 delivery
		       60.0s  1257939            0.0           58.5      0.0      0.0      0.0      0.0 newOrder
		       60.0s  1257939            0.0            6.1      0.0      0.0      0.0      0.0 orderStatus
		       60.0s  1257939            0.0           52.4      0.0      0.0      0.0      0.0 payment
		       60.0s  1257939            0.0            6.1      0.0      0.0      0.0      0.0 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		       61.0s  1257939            0.0            6.0      0.0      0.0      0.0      0.0 delivery
		       61.0s  1257939            0.0           57.6      0.0      0.0      0.0      0.0 newOrder
		       61.0s  1257939            0.0            6.0      0.0      0.0      0.0      0.0 orderStatus
		       61.0s  1257939            0.0           51.5      0.0      0.0      0.0      0.0 payment
		       61.0s  1257939            0.0            6.0      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 2: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1795062-1583655253-01-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: dead
		3: 9191
		1: 9731
		Error:  2: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-09T07:56:00Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      133.0s   645708           11.0            9.8    604.0   1006.6   1006.6   1006.6 delivery
		      133.0s   645708          108.1           96.3    738.2   1744.8   2281.7   2684.4 newOrder
		      133.0s   645708            9.0            9.8    130.0    192.9    192.9    192.9 orderStatus
		      133.0s   645708           75.0           95.4   1006.6   2818.6   4026.5   6710.9 payment
		      133.0s   645708            9.0            9.7     96.5   2147.5   2147.5   2147.5 stockLevel
		      134.0s   645708           19.0            9.9    704.6   1610.6   2952.8   2952.8 delivery
		      134.0s   645708          134.0           96.6    704.6   2013.3   3221.2   3623.9 newOrder
		      134.0s   645708            9.0            9.8     67.1    234.9    234.9    234.9 orderStatus
		      134.0s   645708          116.0           95.6    771.8   4563.4   7516.2   8321.5 payment
		      134.0s   645708           17.0            9.8    604.0   1275.1   1476.4   1476.4 stockLevel
		      135.0s   660046            0.0            9.8      0.0      0.0      0.0      0.0 delivery
		      135.0s   660046            0.0           95.9      0.0      0.0      0.0      0.0 newOrder
		      135.0s   660046            0.0            9.7      0.0      0.0      0.0      0.0 orderStatus
		      135.0s   660046            0.0           94.9      0.0      0.0      0.0      0.0 payment
		      135.0s   660046            2.0            9.7     75.5    176.2    176.2    176.2 stockLevel
		      136.0s   674982            0.0            9.7      0.0      0.0      0.0      0.0 delivery
		      136.0s   674982            0.0           95.2      0.0      0.0      0.0      0.0 newOrder
		      136.0s   674982            0.0            9.6      0.0      0.0      0.0      0.0 orderStatus
		      136.0s   674982            0.0           94.2      0.0      0.0      0.0      0.0 payment
		      136.0s   674982            1.0            9.7     33.6     33.6     33.6     33.6 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      137.0s   689717            0.0            9.7      0.0      0.0      0.0      0.0 delivery
		      137.0s   689717            0.0           94.5      0.0      0.0      0.0      0.0 newOrder
		      137.0s   689717            0.0            9.6      0.0      0.0      0.0      0.0 orderStatus
		      137.0s   689717            0.0           93.5      0.0      0.0      0.0      0.0 payment
		      137.0s   689717            0.0            9.6      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 2: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1796240-1583738442-14-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: 6091
		2: dead
		3: 6693
		Error:  2: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-10T07:51:08Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@72c4a1bd411f2f82bf9aaa22883821a946614148:

		      134.0s   504247            7.0            9.0    671.1   3355.4   3355.4   3355.4 stockLevel
		      135.0s   504247           11.0            9.0   2147.5   2952.8   6442.5   6442.5 delivery
		      135.0s   504247          111.9           90.2   1342.2   2952.8   4563.4   7247.8 newOrder
		      135.0s   504247           11.0            9.2     52.4    134.2    209.7    209.7 orderStatus
		      135.0s   504247          101.0           89.6   1409.3   6174.0  11274.3  13421.8 payment
		      135.0s   504247           10.0            9.0   1811.9   4563.4   4563.4   4563.4 stockLevel
		      136.0s   508282            5.0            9.0    838.9   2952.8   2952.8   2952.8 delivery
		      136.0s   508282           53.9           90.0   1040.2   2818.6   3623.9   4563.4 newOrder
		      136.0s   508282            2.0            9.1     79.7    167.8    167.8    167.8 orderStatus
		      136.0s   508282           56.9           89.4   1342.2   5100.3  10200.5  16643.0 payment
		      136.0s   508282            8.0            9.0    260.0   3355.4   3355.4   3355.4 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      137.0s   532097            0.0            8.9      0.0      0.0      0.0      0.0 delivery
		      137.0s   532097            0.0           89.3      0.0      0.0      0.0      0.0 newOrder
		      137.0s   532097            0.0            9.1      0.0      0.0      0.0      0.0 orderStatus
		      137.0s   532097            0.0           88.7      0.0      0.0      0.0      0.0 payment
		      137.0s   532097            0.0            8.9      0.0      0.0      0.0      0.0 stockLevel
		      138.0s   557228            0.0            8.9      0.0      0.0      0.0      0.0 delivery
		      138.0s   557228            0.0           88.7      0.0      0.0      0.0      0.0 newOrder
		      138.0s   557228            0.0            9.0      0.0      0.0      0.0      0.0 orderStatus
		      138.0s   557228            0.0           88.1      0.0      0.0      0.0      0.0 payment
		      138.0s   557228            0.0            8.9      0.0      0.0      0.0      0.0 stockLevel
		      139.0s   581669            0.0            8.8      0.0      0.0      0.0      0.0 delivery
		      139.0s   581669            0.0           88.0      0.0      0.0      0.0      0.0 newOrder
		      139.0s   581669            0.0            8.9      0.0      0.0      0.0      0.0 orderStatus
		      139.0s   581669            0.0           87.4      0.0      0.0      0.0      0.0 payment
		      139.0s   581669            0.0            8.8      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

	cluster.go:2368,tpcc.go:168,tpcc.go:310,test_runner.go:747: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:168
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 1: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:801: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1799071-1583824469-13-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: dead
		4: skipped
		3: 7025
		2: 8170
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-11T17:51:59Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@5570c01402796edb7cd06eb8ce7f615371f22d42:


	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:747: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_083239.474_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1801614-1583911542-03-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200311 08:32:41.363521 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 90.499006ms
		    I200311 08:33:22.775969 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 41.412285007s
		    I200311 08:33:23.342252 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 566.193197ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-12T18:00:31Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@793a9200c16693aff32aa6a4dd9d8bbcbddb30aa:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200312-1804460/tpcc/w=100/nodes=3/chaos=true/run_1
	test_runner.go:779: test timed out (10h0m0s)

	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:747: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_084049.309_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:747
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1804460-1583998462-03-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200312 08:40:51.006540 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 73.802081ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-13T17:46:47Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@69dc87d68addedf2fabfb2b14c098cfb35b5f3d0:


	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:753: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_082620.217_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1806460-1584083876-13-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200313 08:26:21.961773 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 79.445594ms
		    I200313 08:27:19.171601 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 57.209702891s
		    I200313 08:27:19.674095 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 502.41856ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-14T17:54:54Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@f5585e933a097b53242d8d5800127b821a9a4d41:


	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:753: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_083544.267_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1807675-1584170966-14-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200314 08:35:45.896823 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 95.459149ms
		    I200314 08:36:22.739687 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 36.842685328s
		    I200314 08:36:23.403329 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 663.587081ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-15T17:47:17Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@33d71472dc01cbc5064b3c5e1fcd666a33f606de:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200315-1808695/tpcc/w=100/nodes=3/chaos=true/run_1
	test_runner.go:785: test timed out (10h0m0s)

	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:753: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_082751.059_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1808695-1584256913-15-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200315 08:27:53.015966 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 167.733977ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-16T18:16:43Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@d8bdc938177ea4b5226b95ee6cc51af0951054fc:

		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_085631.727_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1809758-1584345030-09-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200316 08:56:33.559467 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 177.663631ms
		    I200316 08:57:19.725076 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 46.165368348s
		    I200316 08:57:20.160822 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 435.654687ms
		    I200316 09:03:49.724056 1 workload/tpcc/tpcc.go:372  check 3.3.2.4 took 6m29.563007517s
		    I200316 09:04:11.847747 1 workload/tpcc/tpcc.go:372  check 3.3.2.5 took 22.123478288s
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true
Related:

roachtest: tpcc/w=100/nodes=3/chaos=true failed #46142 roachtest: tpcc/w=100/nodes=3/chaos=true failed C-test-failure O-roachtest O-robot branch-provisional_202003112118_v20.1.0-beta.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-17T17:53:59Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@5a3d0c9539a671f0e55b680d3021b18dde9d190d:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200317-1811809/tpcc/w=100/nodes=3/chaos=true/run_1
	test_runner.go:785: test timed out (10h0m0s)

	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:753: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_083437.348_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1811809-1584430043-06-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200317 08:34:39.007928 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 73.946366ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true
Related:

roachtest: tpcc/w=100/nodes=3/chaos=true failed #46142 roachtest: tpcc/w=100/nodes=3/chaos=true failed C-test-failure O-roachtest O-robot branch-provisional_202003112118_v20.1.0-beta.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-18T17:40:31Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@b5f030223fbcf22e806c48a3c46e74a73a54f50f:


	cluster.go:1935,tpcc.go:170,tpcc.go:310,test_runner.go:753: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_082050.223_n4_workload_check_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:170
		    main.registerTPCC.func5
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:310
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:753
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1814553-1584515366-05-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    I200318 08:20:51.975249 1 workload/tpcc/tpcc.go:372  check 3.3.2.1 took 39.812479ms
		    I200318 08:44:43.406717 1 workload/tpcc/tpcc.go:372  check 3.3.2.2 took 23m51.43141092s
		    I200318 08:44:44.139290 1 workload/tpcc/tpcc.go:372  check 3.3.2.3 took 732.526098ms
		    
		    stdout::
		  - context canceled

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true
Related:

roachtest: tpcc/w=100/nodes=3/chaos=true failed #46142 roachtest: tpcc/w=100/nodes=3/chaos=true failed C-test-failure O-roachtest O-robot branch-provisional_202003112118_v20.1.0-beta.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-19T18:13:00Z

(roachtest).tpcc/w=100/nodes=3/chaos=true failed on master@d571624ae4f833c49c717728a74cc7be78a791f0:

		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1816964-1584603715-06-n4cpu4:4 -- ./workload check tpcc --warehouses=100 {pgurl:1} returned:
		    stderr:
		    gs     0x0
		    
		    -----
		    
		    SIGQUIT: quit
		    PC=0x761ce3 m=13 sigcode=0
		    
		    goroutine 50 [syscall]:
		    runtime.notetsleepg(0x6f2cfc0, 0x3b98c470, 0x0)
		    	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc00002e760 sp=0xc00002e730 pc=0x70b5f4
		    runtime.timerproc(0x6f2cfa0)
		    	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc00002e7d8 sp=0xc00002e760 pc=0x74ea01
		    runtime.goexit()
		    	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00002e7e0 sp=0xc00002e7d8 pc=0x75fdc1
		    created by runtime.(*timersBucket).addtimerLocked
		    	/usr/local/go/src/runtime/time.go:169 +0x10e
		    rax    0xfffffffffffffffc
		    rbx    0x3b98c470
		    rcx    0x761ce3
		    rdx    0x0
		    rdi    0x6f2cfc0
		    rsi    0x80
		    rbp    0xc00002e6e8
		    rsp    0xc00002e6a0
		    r8     0x0
		    r9     0x0
		    r10    0xc00002e6d8
		    r11    0x202
		    r12    0xa
		    r13    0x1
		    r14    0x0
		    r15    0xc0000bc960
		    rip    0x761ce3
		    rflags 0x202
		    cs     0x33
		    fs     0x0
		    gs     0x0
		    bash: line 1: 12090 Aborted                 (core dumped) bash -c "./workload check tpcc --warehouses=100 'postgres://[email protected]:26257?sslmode=disable'"
		    
		    stdout::
		  - context canceled

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:825: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1816964-1584603715-06-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		2: 12655
		3: 12009
		1: dead
		Error:  1: dead

More

Artifacts: /tpcc/w=100/nodes=3/chaos=true
Related:

roachtest: tpcc/w=100/nodes=3/chaos=true failed #46318 roachtest: tpcc/w=100/nodes=3/chaos=true failed C-test-failure O-roachtest O-robot branch-provisional_202003181957_v20.1.0-beta.3 release-blocker
roachtest: tpcc/w=100/nodes=3/chaos=true failed #46142 roachtest: tpcc/w=100/nodes=3/chaos=true failed C-test-failure O-roachtest O-robot branch-provisional_202003112118_v20.1.0-beta.3 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

jordanlewis · 2020-03-19T18:39:22Z

Wow, I haven't seen this before:

SIGSEGV: segmentation violation
PC=0x7cc9c3 m=0 sigcode=0

goroutine 50 [syscall]:
runtime.notetsleepg(0x74c16c0, 0x23a1bc9, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x34 fp=0xc000499760 sp=0xc000499730 pc=0x7761c4
runtime.timerproc(0x74c16a0)
	/usr/local/go/src/runtime/time.go:311 +0x2f1 fp=0xc0004997d8 sp=0xc000499760 pc=0x7b96e1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004997e0 sp=0xc0004997d8 pc=0x7caaa1
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:169 +0x10e

nvanbenschoten · 2020-03-23T20:40:22Z

This looks like an instance of golang/go#24790 in the workload process, not a cockroach process.

Interestingly, we see 3 of the 16 OS threads and 7 goroutines in total in runtime.(*timersBucket).addtimerLocked at the time of the crash. This lends weight to the speculation here that this is related to concurrent access to time.Timer instances.

nvanbenschoten · 2020-03-23T20:51:32Z

Before that failure, we see a series of failures where the tpcc checks got stuck and the test eventually timed out. We see warning like the following in the log:

W200317 08:30:09.471229 73149 kv/kvserver/txnwait/queue.go:517  [n1,s1,r35/1:/Table/55{-/1/40/8}] pusher b648f6ce: have been waiting 60.00s for pushee 03141fb0
W200317 08:30:09.490899 72459 kv/kvserver/txnwait/queue.go:517  [n1,s1,r35/1:/Table/55{-/1/40/8}] pusher 99582d0e: have been waiting 60.00s for pushee fa30a706
W200317 08:30:15.309579 96049 kv/kvserver/txnwait/queue.go:517  [n1,s1,r35/1:/Table/55{-/1/40/8}] pusher 219013e8: have been waiting 60.00s for pushee d1d38dab
W200317 08:30:15.442122 96792 kv/kvserver/txnwait/queue.go:517  [n1,s1,r35/1:/Table/55{-/1/40/8}] pusher 6a25af35: have been waiting 60.00s for pushee b7d25568
W200317 08:30:16.545977 1595 kv/kvserver/txnwait/queue.go:517  [n1,s1,r403/1:/Table/5{5/1/40/8-6}] pusher 663fc37a: have been waiting 60.00s for pushee 89b8e693
W200317 08:30:16.612333 2120 kv/kvserver/txnwait/queue.go:517  [n1,s1,r403/1:/Table/5{5/1/40/8-6}] pusher 28514700: have been waiting 60.00s for pushee 89b8e693

This indicates some difficulty with concurrent transactions. Since this hasn't failed in this way in 5 days, it's possible that this was fixed by #46234, which fixed a critical bug in this area.

I'm going to kick off a suite of 10 of these tests on master to confirm that the test has stabilized.

nvanbenschoten · 2020-03-23T22:03:22Z

10/10 passed on master. Looks like this is fixed.

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Oct 15, 2019

cockroach-teamcity added this to the 19.2 milestone Oct 15, 2019

cockroach-teamcity assigned andreimatei Oct 15, 2019

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

irfansharif modified the milestones: 19.2, 20.1 Mar 11, 2020

cockroach-teamcity mentioned this issue Mar 16, 2020

roachtest: tpcc/w=100/nodes=3/chaos=true failed #46142

Closed

nvanbenschoten unassigned irfansharif Mar 17, 2020

cockroach-teamcity mentioned this issue Mar 19, 2020

roachtest: tpcc/w=100/nodes=3/chaos=true failed #46318

Closed

nvanbenschoten self-assigned this Mar 23, 2020

nvanbenschoten closed this as completed Mar 23, 2020

nvanbenschoten mentioned this issue Mar 24, 2020

roachtest: restart/down-for-2m failed #46326

Closed

irfansharif mentioned this issue Mar 25, 2020

roachtest: scaledata/jobcoordinator/nodes=3 failed #41528

Closed

petermattis mentioned this issue Jun 4, 2020

roachtest: addtimerLocked hits segmentation violation #49385

Closed

roachtest: tpcc/w=100/nodes=3/chaos=true failed #41584

roachtest: tpcc/w=100/nodes=3/chaos=true failed #41584

Comments

cockroach-teamcity commented Oct 15, 2019

cockroach-teamcity commented Nov 22, 2019

cockroach-teamcity commented Dec 21, 2019

cockroach-teamcity commented Dec 25, 2019

cockroach-teamcity commented Dec 31, 2019

cockroach-teamcity commented Jan 3, 2020

cockroach-teamcity commented Jan 6, 2020

cockroach-teamcity commented Jan 6, 2020

cockroach-teamcity commented Jan 15, 2020

cockroach-teamcity commented Jan 16, 2020

cockroach-teamcity commented Jan 21, 2020

cockroach-teamcity commented Jan 23, 2020

cockroach-teamcity commented Jan 25, 2020

cockroach-teamcity commented Feb 3, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 7, 2020

cockroach-teamcity commented Feb 13, 2020

cockroach-teamcity commented Feb 14, 2020

cockroach-teamcity commented Mar 3, 2020

cockroach-teamcity commented Mar 6, 2020

cockroach-teamcity commented Mar 7, 2020

cockroach-teamcity commented Mar 8, 2020

cockroach-teamcity commented Mar 9, 2020

cockroach-teamcity commented Mar 10, 2020

cockroach-teamcity commented Mar 11, 2020

cockroach-teamcity commented Mar 12, 2020

cockroach-teamcity commented Mar 13, 2020

cockroach-teamcity commented Mar 14, 2020

cockroach-teamcity commented Mar 15, 2020

cockroach-teamcity commented Mar 16, 2020

cockroach-teamcity commented Mar 17, 2020

cockroach-teamcity commented Mar 18, 2020

cockroach-teamcity commented Mar 19, 2020

jordanlewis commented Mar 19, 2020

nvanbenschoten commented Mar 23, 2020

nvanbenschoten commented Mar 23, 2020

nvanbenschoten commented Mar 23, 2020