roachtest: tpce/c=5000/nodes=3 failed #74453

cockroach-teamcity · 2022-01-05T14:16:15Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ fb88f951323864fafd104e132cbf1f2f22f2e3ac:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.151
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1 release-blocker]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

yuzefovich · 2022-01-05T20:04:51Z

This is not a release blocker - it's a problem with the test where we try to use a new cluster setting that doesn't exist on 21.2 branch, removing the label.

cockroach-teamcity · 2022-01-06T14:16:43Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ 0c8df44947208da2f6234d2736e8ab6d3b0937c0:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.247
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-07T14:14:18Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ 12c606395dc2cfb35bd82972ab65c4453f33c400:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.1.53
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-08T14:07:56Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ f26513f0b00c95e2e98d5ce3032abd7e65ff0e3d:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.203
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-09T14:39:38Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ aabce20e9f7a7f00323605d976c8d4db799273c9:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.25
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-10T13:57:13Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ 3082c28f2dae9540c4f69ab2ffd6a1d92982b4fe:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.98
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-11T14:00:05Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ f6e5974d563c2700dc68c2f71d4fc05ee1adea6b:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.211
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-12T14:26:07Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ b4a27f3634e3e41dc3966214cf05ff6870c055c5:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.18
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

yuzefovich · 2022-01-12T21:30:28Z

Hey @nvanbenschoten, the first part of the Streamer is about to merged (#68430), and it'll be used by default. At the moment only index joins in some cases are supported, and they probably are slower (for now) when compared against the old path, so I think for TPC-E we should disable the Streamer with SET CLUSTER SETTING sql.distsql.use_streamer.enabled = false;.

cockroach-teamcity · 2022-01-13T13:41:23Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ 66442488fcc9c61af6ce438d1bac91cd47f74529:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.131
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-01-14T13:49:35Z

roachtest.tpce/c=5000/nodes=3 failed with artifacts on release-21.2 @ 4df1305911133426c8c5bb2983048c4486a52cfa:

		  | 48765a6a542d: Pull complete
		  | 1486671a3aac: Pull complete
		  | Digest: sha256:cf6e1dfcab60b992040e08636a0d2c778faf88e146b4a8b2d117ebf7abe00fe1
		  | Status: Downloaded newer image for cockroachdb/tpc-e:latest
		  | Error: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: None, code: SqlState("XXUUU"), message: "unknown cluster setting 'sql.distsql.parallelize_multi_key_lookup_joins.enabled'", detail: None, hint: None, position: None, where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("set_cluster_setting.go"), line: Some(87), routine: Some("SetClusterSetting") }) }
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | sudo docker run cockroachdb/tpc-e:latest --customers=5000 --racks=3 --init --hosts=10.142.0.230
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | Configuring cluster settings...
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,tpce.go:104,tpce.go:121,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:104
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerTPCE.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/tpce.go:121
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6498
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:238
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1581
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

roachtest: tpce/c=5000/nodes=3 failed #74451 roachtest: tpce/c=5000/nodes=3 failed [C-test-failure O-roachtest O-robot branch-release-21.1]

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!}

nvanbenschoten · 2022-01-14T18:18:50Z

This should be resolved.

cockroach-teamcity added branch-release-21.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 5, 2022

yuzefovich removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jan 5, 2022

cockroach-teamcity mentioned this issue Jan 6, 2022

roachtest: tpce/c=5000/nodes=3 failed #74451

Closed

nvanbenschoten self-assigned this Jan 11, 2022

nvanbenschoten closed this as completed Jan 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: tpce/c=5000/nodes=3 failed #74453

roachtest: tpce/c=5000/nodes=3 failed #74453

cockroach-teamcity commented Jan 5, 2022

yuzefovich commented Jan 5, 2022

cockroach-teamcity commented Jan 6, 2022

cockroach-teamcity commented Jan 7, 2022

cockroach-teamcity commented Jan 8, 2022

cockroach-teamcity commented Jan 9, 2022

cockroach-teamcity commented Jan 10, 2022

cockroach-teamcity commented Jan 11, 2022

cockroach-teamcity commented Jan 12, 2022

yuzefovich commented Jan 12, 2022

cockroach-teamcity commented Jan 13, 2022

cockroach-teamcity commented Jan 14, 2022

nvanbenschoten commented Jan 14, 2022

roachtest: tpce/c=5000/nodes=3 failed #74453

roachtest: tpce/c=5000/nodes=3 failed #74453

Comments

cockroach-teamcity commented Jan 5, 2022

yuzefovich commented Jan 5, 2022

cockroach-teamcity commented Jan 6, 2022

cockroach-teamcity commented Jan 7, 2022

cockroach-teamcity commented Jan 8, 2022

cockroach-teamcity commented Jan 9, 2022

cockroach-teamcity commented Jan 10, 2022

cockroach-teamcity commented Jan 11, 2022

cockroach-teamcity commented Jan 12, 2022

yuzefovich commented Jan 12, 2022

cockroach-teamcity commented Jan 13, 2022

cockroach-teamcity commented Jan 14, 2022

nvanbenschoten commented Jan 14, 2022