Search code, repositories, users, issues, pull requests...

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-19T06:34:20Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 4d6d79d8fcba33fc273297472d3c5e373b3c2b6c:

		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) output in run_063359.070933136_n4_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3323450-1629354339-02-n4cpu4:4 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | gx/v4/conn.go:670 +0x627 fp=0xc000c45080 sp=0xc000c44f28 pc=0x17a4c67
		  | github.com/jackc/pgx/v4.(*dbTx).Query(0xc000bbeae0, 0x4a1fd30, 0xc000b223c0, 0x25d828d, 0x7b, 0x0, 0x0, 0x0, 0x0, 0x7ff24d505750, ...)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/tx.go:283 +0x12e fp=0xc000c45100 sp=0xc000c45080 pc=0x17aeaae
		  | github.com/jackc/pgx/v4.(*dbTx).QueryRow(0xc000bbeae0, 0x4a1fd30, 0xc000b223c0, 0x25d828d, 0x7b, 0x0, 0x0, 0x0, 0x1, 0xc00053eaf0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/tx.go:288 +0x87 fp=0xc000c45170 sp=0xc000c45100 pc=0x17aeb87
		  | github.com/jackc/pgx/v4/pgxpool.(*Tx).QueryRow(0xc000bcafd8, 0x4a1fd30, 0xc000b223c0, 0x25d828d, 0x7b, 0x0, 0x0, 0x0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/tx.go:66 +0x89 fp=0xc000c451d0 sp=0xc000c45170 pc=0x17bbb29
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randTable(0xc000b222c0, 0x4a1fd30, 0xc000b223c0, 0x4af57a0, 0xc000bcafd8, 0x5a, 0x0, 0x0, 0x0, 0x0, ...)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2656 +0xcf7 fp=0xc000c457a0 sp=0xc000c451d0 pc=0x1c18b57
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).createTableAs(0xc000b222c0, 0x4a1fd30, 0xc000b223c0, 0x4af57a0, 0xc000bcafd8, 0x554d5f, 0xc000452a00, 0x63c46368bc14d753, 0x7ffffffc00000000)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:1105 +0x467 fp=0xc000c45ad8 sp=0xc000c457a0 pc=0x1c07ce7
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp(0xc000b222c0, 0x4a1fd30, 0xc000b223c0, 0x4af57a0, 0xc000bcafd8, 0xc000c45bb0, 0x49a33b, 0x6b20b26c7a, 0xc023561281)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:238 +0xc7 fp=0xc000c45b58 sp=0xc000c45ad8 pc=0x1bffc87
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn(0xc0003319f0, 0x4a1fd30, 0xc000b223c0, 0x4af57a0, 0xc000bcafd8, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:325 +0x157 fp=0xc000c45c28 sp=0xc000c45b58 pc=0x1c1d137
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run(0xc0003319f0, 0x4a1fd30, 0xc000b223c0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:398 +0x1c5 fp=0xc000c45ea8 sp=0xc000c45c28 pc=0x1c1db05
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run-fm(0x4a1fd30, 0xc000b223c0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:386 +0x47 fp=0xc000c45ee0 sp=0xc000c45ea8 pc=0x1c22187
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.workerRun(0x4a1fd30, 0xc000b223c0, 0xc0004818c0, 0xc000ae5040, 0x0, 0xc000adf120)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:273 +0xaf fp=0xc000c45f58 sp=0xc000c45ee0 pc=0x1835caf
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3.1(0xc0002e8280, 0xc00053c040, 0xc0004818c0, 0x0, 0x4a1fd30, 0xc000b223c0, 0xc000ae5040, 0x4, 0xc000adf120)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:475 +0xec fp=0xc000c45f98 sp=0xc000c45f58 pc=0x183a52c
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000c45fa0 sp=0xc000c45f98 pc=0x49d281
		  | created by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:465 +0x111
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

Reproduce

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-20T06:45:30Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 7897f24246bef3cb94f9f4bfaed474ecaa9fdee6:

		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) output in run_064510.340321223_n4_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3329754-1629441450-01-n4cpu4:4 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | h/pkg/workload/schemachange.(*operationGenerator).randTable(0xc000a7a8c0, 0x4a13ad0, 0xc000a7a9c0, 0x4ae9500, 0xc00034dd40, 0x5a, 0x0, 0x0, 0xc000a7aa00, 0x0, ...)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2656 +0xcf7 fp=0xc000c237a0 sp=0xc000c231d0 pc=0x1c11697
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).createTableAs(0xc000a7a8c0, 0x4a13ad0, 0xc000a7a9c0, 0x4ae9500, 0xc00034dd40, 0x554d5f, 0xc000567500, 0x3e869ad0cce5a2ec, 0x7ffffffc00000000)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:1105 +0x467 fp=0xc000c23ad8 sp=0xc000c237a0 pc=0x1c00827
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp(0xc000a7a8c0, 0x4a13ad0, 0xc000a7a9c0, 0x4ae9500, 0xc00034dd40, 0xc000c23bb0, 0x49a33b, 0x61d0c3187b, 0xc02d5a3464)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:238 +0xc7 fp=0xc000c23b58 sp=0xc000c23ad8 pc=0x1bf87c7
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn(0xc000c10370, 0x4a13ad0, 0xc000a7a9c0, 0x4ae9500, 0xc00034dd40, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:325 +0x157 fp=0xc000c23c28 sp=0xc000c23b58 pc=0x1c15c77
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run(0xc000c10370, 0x4a13ad0, 0xc000a7a9c0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:398 +0x1c5 fp=0xc000c23ea8 sp=0xc000c23c28 pc=0x1c16645
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run-fm(0x4a13ad0, 0xc000a7a9c0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:386 +0x47 fp=0xc000c23ee0 sp=0xc000c23ea8 pc=0x1c1acc7
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.workerRun(0x4a13ad0, 0xc000a7a9c0, 0xc0002a1620, 0xc0004244e0, 0x0, 0xc000a6c910)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:273 +0xaf fp=0xc000c23f58 sp=0xc000c23ee0 pc=0x183adaf
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3.1(0xc00014e850, 0xc00014c040, 0xc0002a1620, 0x0, 0x4a13ad0, 0xc000a7a9c0, 0xc0004244e0, 0x4, 0xc000a6c910)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:475 +0xec fp=0xc000c23f98 sp=0xc000c23f58 pc=0x183f62c
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000c23fa0 sp=0xc000c23f98 pc=0x49d281
		  | created by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:465 +0x111
		  |
		  | goroutine 118 [runnable]:
		  | runtime.gopark(0x3f9baf0, 0x0, 0x1809, 0x1)
		  | 	/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000ac7e18 sp=0xc000ac7df8 pc=0x465dc5
		  | runtime.selectgo(0xc000ac7f98, 0xc000ac7f90, 0x0, 0x0, 0x2, 0x1, 0x0, 0x46d3d6)
		  | 	/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000ac7f50 sp=0xc000ac7e18 pc=0x477037
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1(0x4a13ad0, 0xc000a7a9c0, 0xc000372020)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:41 +0x8b fp=0xc000ac7fc8 sp=0xc000ac7f50 pc=0x16afa8b
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000ac7fd0 sp=0xc000ac7fc8 pc=0x49d281
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 +0x99
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

Reproduce

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-21T06:29:22Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 11e0a4da82124e70e772a009011ca7a4007bff85:

		  | 	goroutine running on other thread; stack unavailable
		  | created by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:465 +0x111
		  |
		  | goroutine 27 [select]:
		  | runtime.gopark(0x3f9e428, 0x0, 0x1809, 0x1)
		  | 	/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002d1618 sp=0xc0002d15f8 pc=0x465dc5
		  | runtime.selectgo(0xc0002d1798, 0xc0002d1790, 0x0, 0x0, 0x2, 0x1, 0x0, 0x100000000000000)
		  | 	/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002d1750 sp=0xc0002d1618 pc=0x477037
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1(0x4a166b0, 0xc00046dac0, 0xc000352020)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:41 +0x8b fp=0xc0002d17c8 sp=0xc0002d1750 pc=0x16afceb
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002d17d0 sp=0xc0002d17c8 pc=0x49d281
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 +0x99
		  | panic: errors.As: *target must be interface or implement error, found *pgconn.PgError
		  |
		  | goroutine 79 [running]:
		  | panic(0x21b84a0, 0xc00027b150)
		  | 	/usr/local/go/src/runtime/panic.go:1065 +0x565 fp=0xc000b89a58 sp=0xc000b89990 pc=0x462d45
		  | github.com/cockroachdb/errors/errutil.As(0x0, 0x0, 0x20e1100, 0xc0006d60f0, 0xc000726e88)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/errors/errutil/as.go:49 +0x691 fp=0xc000b89b58 sp=0xc000b89a58 pc=0x92b711
		  | github.com/cockroachdb/errors.As(...)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/errors/errutil_api.go:194
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn(0xc0005b5090, 0x4a166b0, 0xc00046dac0, 0x4aec0e0, 0xc000726e88, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:327 +0x1e5 fp=0xc000b89c28 sp=0xc000b89b58 pc=0x1c17ec5
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run(0xc0005b5090, 0x4a166b0, 0xc00046dac0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:398 +0x1c5 fp=0xc000b89ea8 sp=0xc000b89c28 pc=0x1c18805
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run-fm(0x4a166b0, 0xc00046dac0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:386 +0x47 fp=0xc000b89ee0 sp=0xc000b89ea8 pc=0x1c1ce87
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.workerRun(0x4a166b0, 0xc00046dac0, 0xc0003a5f80, 0xc0005cb950, 0x0, 0xc00027aee0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:273 +0xaf fp=0xc000b89f58 sp=0xc000b89ee0 pc=0x183b00f
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3.1(0xc0002b48a0, 0xc000700040, 0xc0003a5f80, 0x0, 0x4a166b0, 0xc00046dac0, 0xc0005cb950, 0x2, 0xc00027aee0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:475 +0xec fp=0xc000b89f98 sp=0xc000b89f58 pc=0x183f88c
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b89fa0 sp=0xc000b89f98 pc=0x49d281
		  | created by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:465 +0x111
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 3. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

Reproduce

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-22T06:50:18Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ d18da6c092bf1522e7a6478fe3973817e318c247:

		  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3.1(0xc00035a0f0, 0xc0002c2040, 0xc000456300, 0x0, 0x4a1c090, 0xc000c09a80, 0xc0002caff0, 0x4, 0xc0002c49b0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:475 +0xec fp=0xc000cb1f98 sp=0xc000cb1f58 pc=0x183f88c
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000cb1fa0 sp=0xc000cb1f98 pc=0x49d281
		  | created by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:465 +0x111
		  |
		  | goroutine 62 [runnable]:
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1(0x4a1c090, 0xc000c09a80, 0xc0004f6040)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 fp=0xc000c12fc8 sp=0xc000c12fc0 pc=0x16afc60
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000c12fd0 sp=0xc000c12fc8 pc=0x49d281
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 +0x99
		  |
		  | goroutine 47 [select]:
		  | runtime.gopark(0x3f9ea30, 0x0, 0x1809, 0x1)
		  | 	/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000c18e18 sp=0xc000c18df8 pc=0x465dc5
		  | runtime.selectgo(0xc000c18f98, 0xc000c18f90, 0x0, 0x0, 0x2, 0x1, 0x0, 0x46d3d6)
		  | 	/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000c18f50 sp=0xc000c18e18 pc=0x477037
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1(0x4a1c090, 0xc000c09a80, 0xc0002c1060)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:41 +0x8b fp=0xc000c18fc8 sp=0xc000c18f50 pc=0x16afceb
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000c18fd0 sp=0xc000c18fc8 pc=0x49d281
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 +0x99
		  |
		  | goroutine 77 [select]:
		  | runtime.gopark(0x3f9ea30, 0x0, 0x1809, 0x1)
		  | 	/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00096be18 sp=0xc00096bdf8 pc=0x465dc5
		  | runtime.selectgo(0xc00096bf98, 0xc00096bf90, 0x0, 0x0, 0x2, 0x46d301, 0x0, 0xc00044d8c0)
		  | 	/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc00096bf50 sp=0xc00096be18 pc=0x477037
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1(0x4a1c090, 0xc000c09a80, 0xc000adbe60)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:41 +0x8b fp=0xc00096bfc8 sp=0xc00096bf50 pc=0x16afceb
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00096bfd0 sp=0xc00096bfc8 pc=0x49d281
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 +0x99
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

Reproduce

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-23T06:40:36Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 61bd543ba7288c8f0eed6cddded7b219c9d1fcd4:

		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) output in run_064015.616120217_n4_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-3337780-1629700327-11-n4cpu4:4 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | kroach/pkg/workload/schemachange.(*operationGenerator).randTable(0xc0002fa2c0, 0x4a1c090, 0xc0002fa380, 0x4af1b80, 0xc000662ae0, 0x5a, 0x0, 0x0, 0x10, 0xc0008d0ba0, ...)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:2656 +0xcf7 fp=0xc000b51990 sp=0xc000b513c0 pc=0x1c13917
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).renameTable(0xc0002fa2c0, 0x4a1c090, 0xc0002fa380, 0x4af1b80, 0xc000662ae0, 0x554d5f, 0xc0005e6a00, 0x225ecd6831602d40, 0x7ffffffc00000000)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:1803 +0x92 fp=0xc000b51ad8 sp=0xc000b51990 pc=0x1c09992
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*operationGenerator).randOp(0xc0002fa2c0, 0x4a1c090, 0xc0002fa380, 0x4af1b80, 0xc000662ae0, 0xc000b51bb0, 0x49a33b, 0x691829611c, 0xc0004ba8e3)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:238 +0xc7 fp=0xc000b51b58 sp=0xc000b51ad8 pc=0x1bfaa47
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn(0xc0002fd090, 0x4a1c090, 0xc0002fa380, 0x4af1b80, 0xc000662ae0, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:325 +0x157 fp=0xc000b51c28 sp=0xc000b51b58 pc=0x1c17ef7
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run(0xc0002fd090, 0x4a1c090, 0xc0002fa380, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:398 +0x1c5 fp=0xc000b51ea8 sp=0xc000b51c28 pc=0x1c188c5
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run-fm(0x4a1c090, 0xc0002fa380, 0x0, 0x0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:386 +0x47 fp=0xc000b51ee0 sp=0xc000b51ea8 pc=0x1c1cf47
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.workerRun(0x4a1c090, 0xc0002fa380, 0xc0004e1ce0, 0xc0002aec80, 0x0, 0xc0002aa670)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:273 +0xaf fp=0xc000b51f58 sp=0xc000b51ee0 pc=0x183b00f
		  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3.1(0xc0009f45f0, 0xc0002a8040, 0xc0004e1ce0, 0x0, 0x4a1c090, 0xc0002fa380, 0xc0002aec80, 0x4, 0xc0002aa670)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:475 +0xec fp=0xc000b51f98 sp=0xc000b51f58 pc=0x183f88c
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b51fa0 sp=0xc000b51f98 pc=0x49d281
		  | created by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func3
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:465 +0x111
		  |
		  | goroutine 92 [select]:
		  | runtime.gopark(0x3f9ea30, 0x0, 0x1809, 0x1)
		  | 	/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000b48618 sp=0xc000b485f8 pc=0x465dc5
		  | runtime.selectgo(0xc000b48798, 0xc000b48790, 0x0, 0x0, 0x2, 0x1, 0x0, 0x46d3d6)
		  | 	/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000b48750 sp=0xc000b48618 pc=0x477037
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1(0x4a1c090, 0xc0002fa380, 0xc0009f6de0)
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:41 +0x8b fp=0xc000b487c8 sp=0xc000b48750 pc=0x16afceb
		  | runtime.goexit()
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b487d0 sp=0xc000b487c8 pc=0x49d281
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgconn/internal/ctxwatch/context_watcher.go:40 +0x99
		  | Error: SSH_PROBLEM: exit status 255
		  | (1) SSH_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 255
		  | Error types: (1) errors.SSH (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

Reproduce

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-24T16:47:02Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 46cef2c6f0b36ba2f7d551c8ab017832c1b9d592:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)

Reproduce

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^schemachange/mixed-versions$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-08-31T07:35:50Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ c1ef81f5f435b3cc5bdf8b218532e0779f03a6bf:

		  | W210831 07:35:14.076526 1 workload/cli/run.go:417  [-] 1798  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:16.130636 1 workload/cli/run.go:417  [-] 1799  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:17.894940 1 workload/cli/run.go:417  [-] 1800  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:19.986529 1 workload/cli/run.go:417  [-] 1801  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:22.245471 1 workload/cli/run.go:417  [-] 1802  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:24.308209 1 workload/cli/run.go:417  [-] 1803  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:26.231271 1 workload/cli/run.go:417  [-] 1804  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:28.355425 1 workload/cli/run.go:417  [-] 1805  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | W210831 07:35:30.652258 1 workload/cli/run.go:417  [-] 1806  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | E210831 07:35:31.246582 1 workload/cli/run.go:431  [-] 1807  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: dial tcp 10.142.15.218:26257: connect: connection refused
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.15.218 user=root database=schemachange``: dial error (dial tcp 10.142.15.218:26257: connect: connection refused)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3386181-1630390743-14-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 17425
		1: 13210
		3: 12282
		2: dead (exit status 134)
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 134)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 2: dead (exit status 134)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-09-01T16:52:19Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 15b773c71f92d643795e34c922717fde0447f9cd:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)

Reproduce

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-09-04T07:56:51Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 44c95054cfe0108df872d2cda7952fc92259426b:

		  | I210904 07:56:21.935029 1 workload/pgx_helpers.go:61  [-] 3604  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.99:26257: connect: connection refused]
		  | W210904 07:56:23.996242 1 workload/cli/run.go:417  [-] 3605  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.0.99:26257: connect: connection refused
		  | I210904 07:56:23.997886 1 workload/pgx_helpers.go:61  [-] 3606  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.99:26257: connect: connection refused]
		  | W210904 07:56:25.919489 1 workload/cli/run.go:417  [-] 3607  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.0.99:26257: connect: connection refused
		  | I210904 07:56:25.921120 1 workload/pgx_helpers.go:61  [-] 3608  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.99:26257: connect: connection refused]
		  | W210904 07:56:28.044131 1 workload/cli/run.go:417  [-] 3609  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.0.99:26257: connect: connection refused
		  | I210904 07:56:28.045897 1 workload/pgx_helpers.go:61  [-] 3610  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.99:26257: connect: connection refused]
		  | W210904 07:56:30.341145 1 workload/cli/run.go:417  [-] 3611  retrying after error while creating load: failed to initialize the load generator: dial tcp 10.142.0.99:26257: connect: connection refused
		  | I210904 07:56:30.342752 1 workload/pgx_helpers.go:61  [-] 3612  pgx logger [error]: connect failed logParams=map[err:dial tcp 10.142.0.99:26257: connect: connection refused]
		  | E210904 07:56:31.072492 1 workload/cli/run.go:431  [-] 3613  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: dial tcp 10.142.0.99:26257: connect: connection refused
		  | Error: failed to initialize the load generator: failed to connect to ``host=10.142.0.99 user=root database=schemachange``: dial error (dial tcp 10.142.0.99:26257: connect: connection refused)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3405042-1630737365-11-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 17409
		3: 11985
		2: dead (exit status 134)
		1: 13012
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 134)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 2: dead (exit status 134)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError

Reproduce

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-09-07T16:44:34Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ e369d86b3bac2b9f40d760c3e6e49f55b7e15abe:

The test failed on branch=master, cloud=gce:
test timed out (see artifacts for details)

Reproduce

Same failure on other branches

roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-19T05:26:40Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 4f361d30d788f9614695e687b386aedd783ee3ba:

		  |  "workerId": 0,
		  |  "clientTimestamp": "05:26:24.935546",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42602===========================Executed queries for generating errors: QUERY [SELECT region FROM [SHOW REGIONS FROM DATABASE]] : \nQUERY [SHOW DATABASE] :schemachange\n===========================Previous statements [ALTER DATABASE schemachange SURVIVE ZONE FAILURE]: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:26:24.935354",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-east1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42P12===========================Executed queries for generating errors: QUERY [SELECT region FROM [SHOW REGIONS FROM CLUSTER]] : us-east1,\nQUERY [SELECT region FROM [SHOW REGIONS FROM DATABASE]] : \nQUERY [SHOW DATABASE] :schemachange\n===========================Previous statements [ALTER DATABASE schemachange ADD REGION \"us-east1\"]: ERROR: cannot add region \"us-east1\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:26:24.956545",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE public.enum14 AS ENUM ('d', 'asfsf')"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1647667584.956444178,0 encountered previous write with future timestamp 1647667584.956444178,1 within uncertainty interval ``t \u003c= (local=1647667584.963134766,0, global=1647667585.456444178,0)``; observed timestamps: [{1 1647667584.963134766,0} {3 1647667584.957201685,0} {4 1647667584.956444178,0}]: \"sql txn\" meta={id=2a4d75e6 pri=0.03839941 epo=0 ts=1647667584.956444178,0 min=1647667584.956444178,0 seq=9} lock=true stat=PENDING rts=1647667584.956444178,0 wto=false gul=1647667585.456444178,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:26:24.934924",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema13 AUTHORIZATION root"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "querying for validation errors failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1647667584.934833225,0 encountered previous write with future timestamp 1647667584.935026527,0 within uncertainty interval ``t \u003c= (local=1647667584.935610988,0, global=1647667585.434833225,0)``; observed timestamps: [{1 1647667584.934833225,0} {2 1647667584.992283176,0} {3 1647667584.935610988,0}]: \"sql txn\" meta={id=0cb806ad pri=0.01944085 epo=0 ts=1647667584.956444178,1 min=1647667584.934833225,0 seq=15} lock=true stat=PENDING rts=1647667584.934833225,0 wto=false gul=1647667585.434833225,0 (SQLSTATE 40001)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #77885 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-20T05:28:25Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ b3525c95b176f7520173e7a6d3e4a93dca86b198:

		  |   "BEGIN",
		  |   "CREATE TYPE public.enum1 AS ENUM ('adddf')",
		  |   "CREATE TABLE public.table2 (col2_3 REGPROCEDURE NULL, col2_4 INT2 NOT NULL, col2_5 BYTES NOT NULL, col2_6 CHAR, col2_7 NAME, col2_8 STRING NULL AS (CASE WHEN col2_3 IS NULL THEN ';|,[eA':::STRING ELSE e'\\x07':::STRING END) VIRTUAL, INDEX (col2_5, col2_4 DESC, col2_8, col2_3 DESC, lower(col2_6) ASC, col2_7 DESC) STORING (col2_6), INDEX (lower(col2_7) DESC, col2_4 ASC, (CASE WHEN col2_5 IS NULL THEN e'\\x19\\rB7F\\\\O':::STRING ELSE 'P\"':::STRING END) DESC, col2_3 ASC, col2_8, col2_6 ASC), INDEX (col2_6) STORING (col2_4, col2_5), UNIQUE (col2_6 DESC, col2_4, col2_7, col2_5, col2_8 DESC) STORING (col2_3), INDEX (col2_4, col2_8 DESC, col2_3, col2_6 ASC) STORING (col2_5), INDEX (col2_8 ASC, col2_3 DESC, col2_6 DESC, col2_4 DESC, col2_5 ASC))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:28:08.01875",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER INDEX public.table12@\"IrrelevantConstraintName\" RENAME TO \"OtherConstraintName\""
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42P01===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table12\"]] :false\n===========================Previous statements [ALTER INDEX public.table12@\"IrrelevantConstraintName\" RENAME TO \"OtherConstraintName\"]: ERROR: relation \"public.table12\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:28:08.050024",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:28:08.087516",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError: read at time 1647754088.087378984,0 encountered previous write with future timestamp 1647754088.087378984,1 within uncertainty interval ``t \u003c= (local=1647754088.097086616,0, global=1647754088.587378984,0)``; observed timestamps: [{2 1647754088.097086616,0} {3 1647754088.087378984,0}]: \"sql txn\" meta={id=affc942a pri=0.03711139 epo=0 ts=1647754088.087378984,0 min=1647754088.087378984,0 seq=0} lock=false stat=PENDING rts=1647754088.087378984,0 wto=false gul=1647754088.587378984,0 (SQLSTATE 40001)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #77885 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-22T05:27:38Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 10e0c5d92f8ef953d6b497b448893bb5044cdd31:

		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:27:21.899602",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq122",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=4c9d73a5 key=/Table/SystemConfigSpan/Start pri=0.00925171 epo=0 ts=1647926842.029163356,1 min=1647926841.899531953,0 seq=8} lock=true stat=PENDING rts=1647926841.899531953,0 wto=false gul=1647926842.399531953,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:27:21.907663",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER TABLE public.table123 RENAME TO public.table124"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42P01===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table123\"]] :false\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"public\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table124\"]] :false\nQUERY [\"\\n\\tSELECT EXISTS(\\n        SELECT fd.descriptor_name\\n          FROM crdb_internal.forward_dependencies AS fd\\n         WHERE fd.descriptor_id\\n               = (\\n                    SELECT c.oid\\n                      FROM pg_catalog.pg_class AS c\\n                      JOIN pg_catalog.pg_namespace AS ns ON\\n                            ns.oid = c.relnamespace\\n                     WHERE c.relname = $1 AND ns.nspname = $2\\n                )\\n           AND fd.descriptor_id != fd.dependedonby_id\\n           AND fd.dependedonby_type != 'sequence'\\n       )\\n\\t\" [\"table123\" \"public\"]] :false\n===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() ALTER TABLE public.table123 RENAME TO public.table124]: ERROR: relation \"public.table123\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:27:21.830219",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE public.table111 (col111_112 REGROLE NOT NULL, col111_113 BOOL NOT NULL, col111_114 OID NOT NULL, col111_115 GEOGRAPHY NULL, col111_116 CHAR[] NULL, col111_117 BOX2D NOT NULL, col111_118 GEOGRAPHY NULL, col111_119 STRING NOT NULL AS (lower(CAST(col111_117 AS STRING))) STORED, col111_120 STRING NULL AS (CASE WHEN col111_116 IS NULL THEN e'e\\tm\\x1a':::STRING ELSE '\"':::STRING END) STORED, col111_121 STRING NOT NULL AS (lower(CAST(col111_113 AS STRING))) VIRTUAL, FAMILY (col111_117), FAMILY (col111_116, col111_118, col111_112, col111_113), FAMILY (col111_115), FAMILY (col111_119, col111_114, col111_120))",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #77885 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-23T06:32:32Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ e02793d3e214c5b716149963369aa72f9338ac0d:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*versionUpgradeTest).run
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:206
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_053212.343938023_n2_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 3551 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220323 06:32:10.543551 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220323 06:32:12.628014 1 workload/cli/run.go:423  [-] 3545  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table209" (221): secondary index "table209_col209_210_col209_226_col209_218_col209_224_col209_225_col209_214_col209_216_col209_220_col209_213_col209_211_col209_223_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table209" (221): secondary index "table209_col209_210_col209_226_col209_218_col209_224_col209_225_col209_214_col209_216_col209_220_col209_213_col209_211_col209_223_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +  FROM (
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +    SELECT name
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +      FROM (
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +           ) AS obj (name)
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +       )
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220323 06:32:12.663975 1 workload/pgx_helpers.go:79  [-] 3546 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220323 06:32:13.721382 1 workload/cli/run.go:437  [-] 3547  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table209" (221): secondary index "table209_col209_210_col209_226_col209_218_col209_224_col209_225_col209_214_col209_216_col209_220_col209_213_col209_211_col209_223_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table209" (221): secondary index "table209_col209_210_col209_226_col209_218_col209_224_col209_225_col209_214_col209_216_col209_220_col209_213_col209_211_col209_223_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-24T06:31:42Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 8b367174769c89c0fcfe50986ed68d4650be7750:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*versionUpgradeTest).run
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:206
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_053122.867933933_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +           ) AS obj (name)
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +       )
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220324 06:31:21.753797 1 workload/pgx_helpers.go:79  [-] 3536 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220324 06:31:23.652994 1 workload/cli/run.go:423  [-] 3537  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table159" (213): secondary index "table159_col159_167_col159_172_col159_162_col159_170_col159_171_col159_173_expr_col159_160_expr1_col159_166_col159_174_col159_168_col159_165_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table159" (213): secondary index "table159_col159_167_col159_172_col159_162_col159_170_col159_171_col159_173_expr_col159_160_expr1_col159_166_col159_174_col159_168_col159_165_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +  FROM (
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +    SELECT name
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +      FROM (
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220324 06:31:23.690952 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220324 06:31:24.107944 1 workload/cli/run.go:437  [-] 3539  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table159" (213): secondary index "table159_col159_167_col159_172_col159_162_col159_170_col159_171_col159_173_expr_col159_160_expr1_col159_166_col159_174_col159_168_col159_165_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table159" (213): secondary index "table159_col159_167_col159_172_col159_162_col159_170_col159_171_col159_173_expr_col159_160_expr1_col159_166_col159_174_col159_168_col159_165_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

ajwerner · 2022-03-24T13:12:08Z

@fqazi can you take a look at this when you get a chance?

cockroach-teamcity · 2022-03-25T06:26:49Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ d2a588523715f914906f08b1365f8fc9c8ae2805:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052631.074346486_n4_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3542  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table212" (220): primary index column "col212_221" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +  FROM (
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +    SELECT name
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +      FROM (
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +           ) AS obj (name)
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +       )
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220325 06:26:29.316352 1 workload/pgx_helpers.go:79  [-] 3542 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220325 06:26:31.471804 1 workload/cli/run.go:423  [-] 3543  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table212" (220): primary index column "col212_221" cannot be virtual (SQLSTATE XXUUU)
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table212" (220): primary index column "col212_221" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220325 06:26:31.509826 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220325 06:26:32.495536 1 workload/cli/run.go:437  [-] 3545  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table212" (220): primary index column "col212_221" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table212" (220): primary index column "col212_221" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-26T06:28:07Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 090d481f2ec1d1c3e668274f50e4f152b4596a1b:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*versionUpgradeTest).run
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:206
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052748.215354441_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +           ) AS obj (name)
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +       )
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220326 06:27:46.219212 1 workload/pgx_helpers.go:79  [-] 3540 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220326 06:27:48.042023 1 workload/cli/run.go:423  [-] 3541  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table280" (244): secondary index "table280_col280_286_col280_284_col280_292_col280_294_col280_291_col280_293_col280_295_col280_289_expr_col280_287_col280_283_col280_290_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table280" (244): secondary index "table280_col280_286_col280_284_col280_292_col280_294_col280_291_col280_293_col280_295_col280_289_expr_col280_287_col280_283_col280_290_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +  FROM (
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +    SELECT name
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +      FROM (
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +           ) AS obj (name)
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +       )
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220326 06:27:48.080596 1 workload/pgx_helpers.go:79  [-] 3542 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220326 06:27:49.713902 1 workload/cli/run.go:437  [-] 3543  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table280" (244): secondary index "table280_col280_286_col280_284_col280_292_col280_294_col280_291_col280_293_col280_295_col280_289_expr_col280_287_col280_283_col280_290_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table280" (244): secondary index "table280_col280_286_col280_284_col280_292_col280_294_col280_291_col280_293_col280_295_col280_289_expr_col280_287_col280_283_col280_290_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2 release-blocker]
roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-27T06:30:08Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 834eaa0e83350486830867b5edd6e8809b52aa55:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052947.436975838_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 17668 1 workload/pgx_helpers.go:79  [-] 3542 +  FROM (
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +    SELECT name
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +      FROM (
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +           ) AS obj (name)
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +       )
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220327 06:29:46.417668 1 workload/pgx_helpers.go:79  [-] 3542 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220327 06:29:48.573693 1 workload/cli/run.go:423  [-] 3543  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table446" (292): secondary index "table446_col446_459_col446_456_col446_454_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table446" (292): secondary index "table446_col446_459_col446_456_col446_454_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220327 06:29:48.612080 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220327 06:29:48.681750 1 workload/cli/run.go:437  [-] 3545  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table446" (292): secondary index "table446_col446_459_col446_456_col446_454_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table446" (292): secondary index "table446_col446_459_col446_456_col446_454_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2 release-blocker]
roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-28T06:29:28Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052908.487857284_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  +  FROM (
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +    SELECT name
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +      FROM (
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +           ) AS obj (name)
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +       )
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220328 06:29:06.358238 1 workload/pgx_helpers.go:79  [-] 3550 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220328 06:29:08.463334 1 workload/cli/run.go:423  [-] 3551  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table356" (276): secondary index "table356_col356_365_col356_371_col356_357_col356_370_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table356" (276): secondary index "table356_col356_365_col356_371_col356_357_col356_370_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +  FROM (
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +    SELECT name
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +      FROM (
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +           ) AS obj (name)
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +       )
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220328 06:29:08.637837 1 workload/pgx_helpers.go:79  [-] 3552 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220328 06:29:09.923611 1 workload/cli/run.go:437  [-] 3553  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table356" (276): secondary index "table356_col356_365_col356_371_col356_357_col356_370_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table356" (276): secondary index "table356_col356_365_col356_371_col356_357_col356_370_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-29T06:33:23Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 29716850b181718594663889ddb5f479fef7a305:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1
	cluster.go:1868,versionupgrade.go:421,versionupgrade.go:382,versionupgrade.go:206,mixed_version_schemachange.go:146,mixed_version_schemachange.go:38,test_runner.go:875: one or more parallel execution failure
		(1) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2042
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1923
		  | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:167
		  | github.com/cockroachdb/cockroach/pkg/roachprod.Start
		  | 	github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:660
		  | main.(*clusterImpl).StartE
		  | 	main/pkg/cmd/roachtest/cluster.go:1826
		  | main.(*clusterImpl).Start
		  | 	main/pkg/cmd/roachtest/cluster.go:1867
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.upgradeNodes
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:421
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.binaryUpgradeStep.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:382
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*versionUpgradeTest).run
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/versionupgrade.go:206
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) one or more parallel execution failure
		Error types: (1) *withstack.withStack (2) *errutil.leafError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema blocks-22.1.0-beta.1 branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-29T12:52:46Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 327f886758e64973e9e6ed221688622b6e1bde69:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_115227.147220150_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220329 12:52:24.653230 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220329 12:52:26.737883 1 workload/cli/run.go:423  [-] 3545  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table1" (164): secondary index "table1_expr_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table1" (164): secondary index "table1_expr_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +  FROM (
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +    SELECT name
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +      FROM (
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +           ) AS obj (name)
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +       )
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220329 12:52:26.773902 1 workload/pgx_helpers.go:79  [-] 3546 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220329 12:52:28.482985 1 workload/cli/run.go:437  [-] 3547  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table1" (164): secondary index "table1_expr_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table1" (164): secondary index "table1_expr_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema blocks-22.1.0-beta.1 branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-30T06:30:01Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ dce0a307cd242d8575f59a153026b1af9e6a7936:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052943.332445304_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3536  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table957" (410): primary index column "col957_969" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +  FROM (
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +    SELECT name
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +      FROM (
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +           ) AS obj (name)
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +       )
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220330 06:29:40.664017 1 workload/pgx_helpers.go:79  [-] 3536 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220330 06:29:42.562731 1 workload/cli/run.go:423  [-] 3537  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table957" (410): primary index column "col957_969" cannot be virtual (SQLSTATE XXUUU)
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table957" (410): primary index column "col957_969" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +  FROM (
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +    SELECT name
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +      FROM (
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220330 06:29:42.607101 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220330 06:29:44.600446 1 workload/cli/run.go:437  [-] 3539  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table957" (410): primary index column "col957_969" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table957" (410): primary index column "col957_969" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #78304 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema blocks-22.1.0-beta.1 branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-31T06:28:44Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ e4fcae1625a35917d08a2a569d24529b0cc089d2:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052826.758821740_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3536  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table694" (349): primary index column "col694_705" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +  FROM (
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +    SELECT name
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +      FROM (
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +           ) AS obj (name)
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +       )
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220331 06:28:25.200424 1 workload/pgx_helpers.go:79  [-] 3536 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220331 06:28:27.099148 1 workload/cli/run.go:423  [-] 3537  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table694" (349): primary index column "col694_705" cannot be virtual (SQLSTATE XXUUU)
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table694" (349): primary index column "col694_705" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +  FROM (
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +    SELECT name
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +      FROM (
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220331 06:28:27.141621 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220331 06:28:28.030896 1 workload/cli/run.go:437  [-] 3539  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table694" (349): primary index column "col694_705" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table694" (349): primary index column "col694_705" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #79107 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-03-31T14:30:59Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ e4fcae1625a35917d08a2a569d24529b0cc089d2:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_133039.414195851_n2_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | -] 3538  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table1347" (522): primary index column "col1347_1352" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +  FROM (
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +    SELECT name
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +      FROM (
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220331 14:30:37.280061 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220331 14:30:39.575726 1 workload/cli/run.go:423  [-] 3539  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table1347" (522): primary index column "col1347_1352" cannot be virtual (SQLSTATE XXUUU)
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table1347" (522): primary index column "col1347_1352" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +  FROM (
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +    SELECT name
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +      FROM (
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +           ) AS obj (name)
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +       )
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220331 14:30:39.615220 1 workload/pgx_helpers.go:79  [-] 3540 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220331 14:30:40.816243 1 workload/cli/run.go:437  [-] 3541  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table1347" (522): primary index column "col1347_1352" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table1347" (522): primary index column "col1347_1352" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 2. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #79107 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-02T06:27:17Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 0e6878b6140cb5cc7c799fdf576b3d565a54fd2e:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052700.469948923_n4_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table443" (285): primary index column "col443_454" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220402 06:26:58.899049 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220402 06:27:00.983416 1 workload/cli/run.go:423  [-] 3545  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table443" (285): primary index column "col443_454" cannot be virtual (SQLSTATE XXUUU)
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table443" (285): primary index column "col443_454" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +  FROM (
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +    SELECT name
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +      FROM (
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +           ) AS obj (name)
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +       )
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220402 06:27:01.021779 1 workload/pgx_helpers.go:79  [-] 3546 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220402 06:27:01.740460 1 workload/cli/run.go:437  [-] 3547  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table443" (285): primary index column "col443_454" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table443" (285): primary index column "col443_454" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #79107 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-03T06:29:02Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 71e32a6a5369dabde22043ca6ebbd11651404e5d:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052844.992536197_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table393" (274): primary index column "col393_405" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220403 06:28:42.506333 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220403 06:28:44.589908 1 workload/cli/run.go:423  [-] 3545  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table393" (274): primary index column "col393_405" cannot be virtual (SQLSTATE XXUUU)
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table393" (274): primary index column "col393_405" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +  FROM (
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +    SELECT name
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +      FROM (
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +           ) AS obj (name)
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +       )
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220403 06:28:44.624579 1 workload/pgx_helpers.go:79  [-] 3546 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220403 06:28:46.221472 1 workload/cli/run.go:437  [-] 3547  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table393" (274): primary index column "col393_405" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table393" (274): primary index column "col393_405" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #79107 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-22.1 release-blocker]
roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-04T06:30:34Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 9a66e9ad26bc34250bee89854cbce1999ea70f9f:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_053016.917833142_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table884" (402): primary index column "col884_891" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220404 06:30:14.484575 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220404 06:30:16.568345 1 workload/cli/run.go:423  [-] 3545  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table884" (402): primary index column "col884_891" cannot be virtual (SQLSTATE XXUUU)
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table884" (402): primary index column "col884_891" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +  FROM (
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +    SELECT name
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +      FROM (
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +           ) AS obj (name)
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +       )
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220404 06:30:16.605714 1 workload/pgx_helpers.go:79  [-] 3546 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220404 06:30:18.345129 1 workload/cli/run.go:437  [-] 3547  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table884" (402): primary index column "col884_891" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table884" (402): primary index column "col884_891" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #79107 roachtest: schemachange/mixed-versions failed [C-test-failure GA-blocker O-roachtest O-robot T-sql-schema branch-release-22.1]
roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-05T06:28:54Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 63ea9139e2ca996e38b5fe7c7b43a97e625242f5:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052835.990436034_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | -] 3536  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table1219" (513): primary index column "col1219_1237" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +  FROM (
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +    SELECT name
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +      FROM (
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +           ) AS obj (name)
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +       )
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220405 06:28:34.021160 1 workload/pgx_helpers.go:79  [-] 3536 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220405 06:28:35.920161 1 workload/cli/run.go:423  [-] 3537  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table1219" (513): primary index column "col1219_1237" cannot be virtual (SQLSTATE XXUUU)
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table1219" (513): primary index column "col1219_1237" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +  FROM (
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +    SELECT name
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +      FROM (
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220405 06:28:35.961724 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220405 06:28:37.246103 1 workload/cli/run.go:437  [-] 3539  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table1219" (513): primary index column "col1219_1237" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table1219" (513): primary index column "col1219_1237" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

Same failure on other branches

roachtest: schemachange/mixed-versions failed #79107 roachtest: schemachange/mixed-versions failed [C-test-failure GA-blocker O-roachtest O-robot T-sql-schema branch-release-22.1]
roachtest: schemachange/mixed-versions failed #78500 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/mixed-versions failed #62158 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2022-04-08T06:32:47Z

roachtest.schemachange/mixed-versions failed with artifacts on master @ 7a97bdd02c62787f40d79d5a2cd3439d8c9c8a1b:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:875
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_053229.254766119_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | -] 3536  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table1365" (523): primary index column "col1365_1375" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +  FROM (
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +    SELECT name
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +      FROM (
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +           ) AS obj (name)
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +       )
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220408 06:32:27.682439 1 workload/pgx_helpers.go:79  [-] 3536 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220408 06:32:29.581982 1 workload/cli/run.go:423  [-] 3537  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table1365" (523): primary index column "col1365_1375" cannot be virtual (SQLSTATE XXUUU)
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table1365" (523): primary index column "col1365_1375" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +  FROM (
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +    SELECT name
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +      FROM (
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220408 06:32:29.624334 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220408 06:32:30.537420 1 workload/cli/run.go:437  [-] 3539  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table1365" (523): primary index column "col1365_1375" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table1365" (523): primary index column "col1365_1375" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help