roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44302

cockroach-teamcity · 2020-01-23T15:37:49Z

(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on release-19.1@ff46e59a94d5b41e314d15aac2bd403ecb5da777:

		  - monitor failure:
		  - error with attached stack trace:
		    main.(*monitor).wait.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor task failed:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1847
		    main.loadTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:628
		    main.runTPCCBench.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:718
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2180
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1705447-1579763243-76-n12cpu4-geo:4 -- ./workload fixtures load tpcc --warehouses=5000 --scatter --checks=false --partitions=3 --zones="us-east1-b,us-west1-b,europe-west2-b" {pgurl:1} returned:
		    stderr:
		    
		    stdout:
		    I200123 12:46:13.380693 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    I200123 12:46:21.653392 83 ccl/workloadccl/fixture.go:547  loaded 4.9 MiB table district in 7.43819258s (50000 rows, 0 index entries, 678 KiB)
		    I200123 12:46:25.776044 88 ccl/workloadccl/fixture.go:547  loaded 7.8 MiB table item in 11.559788645s (100000 rows, 0 index entries, 689 KiB)
		    I200123 12:47:49.474269 82 ccl/workloadccl/fixture.go:547  loaded 266 KiB table warehouse in 1m35.258854172s (5000 rows, 0 index entries, 2.8 KiB)
		    I200123 12:48:52.351743 87 ccl/workloadccl/fixture.go:547  loaded 641 MiB table new_order in 2m38.135308129s (45000000 rows, 0 index entries, 4.1 MiB)
		    I200123 13:21:28.365212 86 ccl/workloadccl/fixture.go:547  loaded 6.7 GiB table order in 35m14.149949532s (150000000 rows, 150000000 index entries, 3.2 MiB)
		    I200123 13:46:40.272107 85 ccl/workloadccl/fixture.go:547  loaded 21 GiB table history in 1h0m26.056908993s (150000000 rows, 300000000 index entries, 6.0 MiB)
		    I200123 14:17:04.368215 84 ccl/workloadccl/fixture.go:547  loaded 86 GiB table customer in 1h30m50.152972532s (150000000 rows, 150000000 index entries, 16 MiB)
		    I200123 15:06:52.061581 89 ccl/workloadccl/fixture.go:547  loaded 157 GiB table stock in 2h20m37.845243107s (500000000 rows, 500000000 index entries, 19 MiB)
		    I200123 15:36:07.822572 90 ccl/workloadccl/fixture.go:547  loaded 115 GiB table order_line in 2h49m53.607444169s (1500013787 rows, 1500013787 index entries, 12 MiB)
		    I200123 15:36:14.461642 1 ccl/workloadccl/cliccl/fixtures.go:286  restored 387 GiB bytes in 9 tables (took 2h50m1.08079115s, 38.88 MiB/s)
		    Error: Could not postload: could not partition tables: Couldn't exec "\n\t\t\tALTER INDEX item@replicated_idx_2\n\t\t\tCONFIGURE ZONE USING num_replicas = COPY FROM PARENT, constraints = '{\"+zone=europe-west2-b\":1}', lease_preferences = '[[+zone=europe-west2-b]]'": pq: could not validate zone config: at least one replica is required
		    Error:  exit status 1:
		  - exit status 1

More

Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2020-02-04T13:22:46Z

(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on release-19.1@dae56c04a5a357b12697c050599c1ae8394f2827:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2368,tpcc.go:729,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:729
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 7: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1726337-1580800472-76-n12cpu4-geo --oneshot --ignore-empty-nodes: exit status 1 4: skipped
		1: 3476
		2: 4059
		3: 4114
		8: skipped
		7: dead
		5: 3733
		12: skipped
		6: 4030
		11: 3710
		9: 3635
		10: 3674
		Error:  7: dead

More

Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44667 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44541 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-27T23:41:53Z

(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on release-19.1@1fcf7104d19c5c7634cfb52c4302bc9e70c4b9ea:

		          740.0s        0           10.0            6.5    369.1    872.4    872.4    872.4 orderStatus
		          740.0s        0           98.0           64.9   1879.0   7247.8  11811.2  13421.8 payment
		          740.0s        0           10.0            6.5    251.7    906.0    906.0    906.0 stockLevel
		        _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		          741.0s        0            6.0            6.5   2147.5   2952.8   2952.8   2952.8 delivery
		          741.0s        0           58.0           63.9   3355.4   5100.3   6174.0   6442.5 newOrder
		          741.0s        0            9.0            6.5    260.0    704.6    704.6    704.6 orderStatus
		          741.0s        0           58.0           64.9   1879.0   5637.1   7516.2   9126.8 payment
		          741.0s        0            8.0            6.5    369.1    604.0    604.0    604.0 stockLevel
		          742.0s        0            5.0            6.5   1677.7   5100.3   5100.3   5100.3 delivery
		          742.0s        0           76.0           63.9   3758.1   5905.6   7516.2   9126.8 newOrder
		          742.0s        0            5.0            6.5    453.0    704.6    704.6    704.6 orderStatus
		          742.0s        0           68.0           64.9   1744.8   7784.6  11811.2  15569.3 payment
		          742.0s        0            7.0            6.5    520.1   3087.0   3087.0   3087.0 stockLevel
		          743.0s        0            7.0            6.5   3355.4   5100.3   5100.3   5100.3 delivery
		          743.0s        0           76.0           63.9   2952.8   6174.0   6710.9   6710.9 newOrder
		          743.0s        0            4.0            6.5     83.9    142.6    142.6    142.6 orderStatus
		          743.0s        0           80.0           64.9   2080.4   8321.5   9663.7  15032.4 payment
		          743.0s        0            3.0            6.5    176.2    419.4    419.4    419.4 stockLevel
		          744.0s        0            3.0            6.5   2818.6   3355.4   3355.4   3355.4 delivery
		          744.0s        0           46.0           63.9   2952.8   5368.7   5905.6   5905.6 newOrder
		          744.0s        0            3.0            6.5    234.9    419.4    419.4    419.4 orderStatus
		          744.0s        0           61.0           64.9   1744.8  10200.5  11274.3  12348.0 payment
		          744.0s        0            5.0            6.5    318.8    939.5    939.5    939.5 stockLevel
		        _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		          745.0s        0            5.0            6.5   2415.9   3623.9   3623.9   3623.9 delivery
		          745.0s        0           60.0           63.9   2818.6   5100.3   5905.6   8321.5 newOrder
		          745.0s        0           10.0            6.5    285.2    704.6    704.6    704.6 orderStatus
		          745.0s        0           45.0           64.9   2013.3   5100.3   8321.5   8321.5 payment
		          745.0s        0            6.0            6.5    192.9    671.1    671.1    671.1 stockLevel
		          746.0s        0            3.0            6.5   2684.4   3221.2   3221.2   3221.2 delivery
		          746.0s        0           46.0           63.9   2952.8   5368.7   6710.9   6710.9 newOrder
		          746.0s        0            4.0            6.5    167.8    243.3    243.3    243.3 orderStatus
		          746.0s        0           51.0           64.9   2415.9   6710.9   7784.6  12884.9 payment
		          746.0s        0            5.0            6.5    134.2    453.0    453.0    453.0 stockLevel
		          747.0s        0            8.0            6.5   2818.6   4026.5   4026.5   4026.5 delivery
		          747.0s        0           59.0           63.9   3221.2   5905.6   6174.0   6174.0 newOrder
		          747.0s        0            8.0            6.5    121.6    671.1    671.1    671.1 orderStatus
		          747.0s        0           53.0           64.9   2281.7   5368.7   8053.1  10737.4 payment
		          747.0s        0            5.0            6.5    436.2    637.5    637.5    637.5 stockLevel:
		      - context canceled
		    error running tpcc load generator
		    main.runTPCCBench.func3.1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:818
		    main.(*monitor).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
		    github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357

More

Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #45086 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-44941 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44958 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202002062122_v19.2.4 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44667 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44541 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-03-06T23:49:54Z

(roachtest).tpccbench/nodes=9/cpu=4/multi-region failed on release-19.1@5d94cbf2e7ffc65cdf9b63c85fb1507bd37fbb07:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200306-1790614/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2368,tpcc.go:729,tpcc.go:570,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runTPCCBench
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:729
		    main.registerTPCCBenchSpec.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 5: dead

More

Artifacts: /tpccbench/nodes=9/cpu=4/multi-region
Related:

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #45086 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-44941 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44958 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202002062122_v19.2.4 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44667 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44541 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-provisional_202001281357_v19.2.3 release-blocker
roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876 roachtest: tpccbench/nodes=9/cpu=4/multi-region failed C-test-failure O-roachtest O-robot branch-master

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

ajwerner · 2020-03-19T18:04:30Z

Duplicate of #46184

ajwerner · 2020-03-19T18:06:11Z

Err #41876

cockroach-teamcity added branch-release-19.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 23, 2020

cockroach-teamcity added this to the 20.1 milestone Jan 23, 2020

cockroach-teamcity assigned andreimatei Jan 23, 2020

This was referenced Jan 30, 2020

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44541

Closed

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44667

Closed

This was referenced Feb 9, 2020

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #41876

Closed

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44958

Closed

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #45086

Closed

cockroach-teamcity mentioned this issue Mar 17, 2020

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #46184

Closed

ajwerner marked this as a duplicate of #46184 Mar 19, 2020

ajwerner closed this as completed Mar 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44302

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44302

cockroach-teamcity commented Jan 23, 2020

cockroach-teamcity commented Feb 4, 2020

cockroach-teamcity commented Feb 27, 2020

cockroach-teamcity commented Mar 6, 2020

ajwerner commented Mar 19, 2020

ajwerner commented Mar 19, 2020

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44302

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #44302

Comments

cockroach-teamcity commented Jan 23, 2020

cockroach-teamcity commented Feb 4, 2020

cockroach-teamcity commented Feb 27, 2020

cockroach-teamcity commented Mar 6, 2020

ajwerner commented Mar 19, 2020

ajwerner commented Mar 19, 2020