Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions failed #62158

Closed
cockroach-teamcity opened this issue Mar 17, 2021 · 41 comments
Closed

roachtest: schemachange/mixed-versions failed #62158

cockroach-teamcity opened this issue Mar 17, 2021 · 41 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 17, 2021 •
edited by cockroach-jira-scripts
Loading

(roachtest).schemachange/mixed-versions failed on release-21.1@ba54410af79c0481e1e8f8fc45a9b132ee09a2b4:

		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |     0.0 opOk
		  | 34805.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34805.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34805.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 34806.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34806.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34806.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34806.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34807.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34807.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34807.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34807.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34808.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34808.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34808.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34808.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34809.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34809.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34809.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34809.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34810.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34810.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34810.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34810.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 34811.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34811.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34811.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34811.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34812.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34812.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34812.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34812.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34813.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34813.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34813.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34813.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34814.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34814.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34814.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34814.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 34815.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 34815.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 34815.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 34815.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

Jira issue: CRDB-6254
@cockroach-teamcity cockroach-teamcity added branch-release-21.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 17, 2021
@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@8cdd17ff30a89b861a015a8f9d92da77ae781373:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_060354.956_n3_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2795707-1616219693-14-n4cpu4:3 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | le creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:27.784343 1 workload/cli/run.go:370  [-] 1788  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:30.034201 1 workload/cli/run.go:370  [-] 1789  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:31.970496 1 workload/cli/run.go:370  [-] 1790  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:34.040740 1 workload/cli/run.go:370  [-] 1791  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:36.000970 1 workload/cli/run.go:370  [-] 1792  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:37.836927 1 workload/cli/run.go:370  [-] 1793  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:40.109429 1 workload/cli/run.go:370  [-] 1794  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:42.283318 1 workload/cli/run.go:370  [-] 1795  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:44.522619 1 workload/cli/run.go:370  [-] 1796  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:46.609118 1 workload/cli/run.go:370  [-] 1797  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:48.826204 1 workload/cli/run.go:370  [-] 1798  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:50.885372 1 workload/cli/run.go:370  [-] 1799  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:52.655987 1 workload/cli/run.go:370  [-] 1800  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210320 07:03:54.753059 1 workload/cli/run.go:370  [-] 1801  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | E210320 07:03:56.345726 1 workload/cli/run.go:384  [-] 1802  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: type ID 517 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 3. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@JuanLeon1 JuanLeon1 removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Mar 23, 2021
@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@b886d8089334416b026391f2be76bc32e3a09b8a:

		  | 35910.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35910.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35910.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@e6d07fa0b8fb13c447805709e2eadcceb783d661:

		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35842.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35842.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35842.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35843.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35843.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35843.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35844.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35844.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35844.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35845.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35845.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35845.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35846.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35846.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35846.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35847.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35847.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35847.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35848.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35848.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35848.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35849.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35849.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35849.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35850.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35850.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35850.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@3fa241dd0d3aba2547e04ec5ef936315c0c5d3e1:

		  | 35688.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35688.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35689.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35689.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35690.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35690.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35691.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35691.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35692.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35692.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35693.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35693.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35694.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35694.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35695.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35695.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35696.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35696.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35697.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35697.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35698.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35698.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35699.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35699.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35700.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35700.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35701.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35701.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35702.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35702.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35703.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35703.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35704.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35704.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35705.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35705.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35706.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35706.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35707.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35707.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35708.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35708.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@80709778b7c7de7b2b704972197022305bb3ca12:

		  | 35718.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35718.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35718.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35719.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35719.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35719.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35720.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35720.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35720.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35721.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35721.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35721.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35722.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35722.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35722.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35723.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35723.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35723.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35724.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35724.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35724.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35725.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35725.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35725.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35726.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35726.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35726.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35727.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35727.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35727.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35728.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35728.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35728.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35729.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35729.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35729.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35730.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35730.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35730.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35731.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35731.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35731.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 14, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@0eb5ba6e6c433f57ed1851a0c160f7739e7a7725:

		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_191024.218_n1_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2903636-1618945277-05-n4cpu4:1 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | le creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:09:57.561218 1 workload/cli/run.go:370  [-] 1787  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:09:59.513734 1 workload/cli/run.go:370  [-] 1788  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:01.763958 1 workload/cli/run.go:370  [-] 1789  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:03.700708 1 workload/cli/run.go:370  [-] 1790  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:05.771250 1 workload/cli/run.go:370  [-] 1791  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:07.731925 1 workload/cli/run.go:370  [-] 1792  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:09.569823 1 workload/cli/run.go:370  [-] 1793  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:11.843500 1 workload/cli/run.go:370  [-] 1794  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:14.019526 1 workload/cli/run.go:370  [-] 1795  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:16.259541 1 workload/cli/run.go:370  [-] 1796  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:18.347701 1 workload/cli/run.go:370  [-] 1797  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:20.566428 1 workload/cli/run.go:370  [-] 1798  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:22.627959 1 workload/cli/run.go:370  [-] 1799  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | W210420 20:10:24.402373 1 workload/cli/run.go:370  [-] 1800  retrying after error while creating load: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | E210420 20:10:25.615383 1 workload/cli/run.go:384  [-] 1801  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: type ID 421 in descriptor not found: descriptor not found (SQLSTATE XXUUU)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 1. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Apr 25, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

(roachtest).schemachange/mixed-versions failed on release-21.1@affd857ffc05591f5898701690b8a9136859ab1b:

		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2332
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2252
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_192118.836_n4_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2921836-1619377807-07-n4cpu4:4 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | hile creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:20:55.009741 1 workload/cli/run.go:370  [-] 1773  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:20:56.949148 1 workload/cli/run.go:370  [-] 1774  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:20:59.264451 1 workload/cli/run.go:370  [-] 1775  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:01.397806 1 workload/cli/run.go:370  [-] 1776  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:03.515835 1 workload/cli/run.go:370  [-] 1777  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:05.841856 1 workload/cli/run.go:370  [-] 1778  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:07.668529 1 workload/cli/run.go:370  [-] 1779  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:09.538655 1 workload/cli/run.go:370  [-] 1780  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:11.425881 1 workload/cli/run.go:370  [-] 1781  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:13.364834 1 workload/cli/run.go:370  [-] 1782  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:15.434255 1 workload/cli/run.go:370  [-] 1783  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:17.368359 1 workload/cli/run.go:370  [-] 1784  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | W210425 20:21:19.442547 1 workload/cli/run.go:370  [-] 1785  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | E210425 20:21:20.306413 1 workload/cli/run.go:384  [-] 1786  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table919" (368): referenced type ID 214: descriptor not found (SQLSTATE XXUUU)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
More

Artifacts: /schemachange/mixed-versions
Related:

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 654f6311c0b3d6f172a0ecc49c7e37340ad1f9e8:

		  | 35834.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35834.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35835.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35835.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35835.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35835.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35836.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35836.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35836.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35836.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35837.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35837.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35837.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35837.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35838.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35838.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35838.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35838.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35839.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35839.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35839.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35839.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35840.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35840.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35840.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35840.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35841.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35841.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35841.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35841.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35842.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35842.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35842.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35842.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35843.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35843.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35843.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35843.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35844.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35844.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35844.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35844.0s       12            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 4, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ aedf03967e1d9179943f403567433c9c1c6d6f23:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/schemachange/mixed-versions/run_1
	cluster.go:2234,mixed_version_schemachange.go:67,versionupgrade.go:200,mixed_version_schemachange.go:144,mixed_version_schemachange.go:36,test_runner.go:767: output in run_191808.931_n2_workload_run_schemachange: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2973815-1620846484-02-n4cpu4:2 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned: exit status 20
		(1) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2312
		  | main.(*cluster).Run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2232
		  | main.runSchemaChangeWorkloadStep.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:67
		  | main.(*versionUpgradeTest).run
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/versionupgrade.go:200
		  | main.runSchemaChangeMixedVersions
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:144
		  | main.registerSchemaChangeMixedVersions.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/mixed_version_schemachange.go:36
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:767
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1374
		Wraps: (2) output in run_191808.931_n2_workload_run_schemachange
		Wraps: (3) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2973815-1620846484-02-n4cpu4:2 -- ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  1 workload/cli/run.go:370  [-] 1722  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:17:49.767334 1 workload/cli/run.go:370  [-] 1723  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:17:51.562772 1 workload/cli/run.go:370  [-] 1724  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:17:53.350654 1 workload/cli/run.go:370  [-] 1725  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:17:55.703722 1 workload/cli/run.go:370  [-] 1726  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:17:57.716802 1 workload/cli/run.go:370  [-] 1727  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:17:59.559019 1 workload/cli/run.go:370  [-] 1728  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:18:01.527379 1 workload/cli/run.go:370  [-] 1729  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:18:03.806210 1 workload/cli/run.go:370  [-] 1730  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:18:05.933956 1 workload/cli/run.go:370  [-] 1731  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:18:08.183224 1 workload/cli/run.go:370  [-] 1732  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | W210512 20:18:10.292753 1 workload/cli/run.go:370  [-] 1733  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | E210512 20:18:10.374897 1 workload/cli/run.go:384  [-] 1734  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: type "enum1335" (437): referencing table 528 was dropped without dependency unlinking (SQLSTATE XX000)
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 2. Command with error:
		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 3bf34abe95cf99091c7ddea2eb767cdf0a297bd8:

		  | 35910.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35910.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35911.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35912.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35913.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35914.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35915.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35916.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35917.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35918.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35919.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35920.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35921.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35922.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35923.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35924.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35924.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35925.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35925.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35926.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35926.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35927.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35927.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35928.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35928.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35929.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35929.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35930.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35930.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ c6917a3058604cc42381915f108428533e1d47ff:

		  | 35775.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35775.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35776.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35776.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35776.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35776.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35777.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35777.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35777.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35777.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35778.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35778.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35778.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35778.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35779.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35779.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35779.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35779.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35780.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35780.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35780.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35780.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35781.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35781.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35781.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35781.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35782.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35782.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35782.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35782.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35783.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35783.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35783.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35783.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35784.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35784.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35784.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35784.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35785.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35785.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35785.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35785.0s       21            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ b600d9c2283e19886406e658369e70b5c61a612f:

		  |             0.0      0.0      0.0      0.0      0.0 opOk
		  | 35767.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35768.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35768.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35769.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35769.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35770.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35770.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35771.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35771.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35772.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35772.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35773.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35773.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35774.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35774.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35775.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35775.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35776.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35776.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35777.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35777.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35778.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35778.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35779.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35779.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35780.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35780.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35781.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35781.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35782.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35782.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35783.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35783.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35784.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35784.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35785.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35785.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35786.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35786.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35787.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35787.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35788.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35788.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:826: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 1e2234a8449f6fe886387128dc5103866c4d244e:

		  |             0.0      0.0      0.0      0.0      0.0 opOk
		  | 35943.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35944.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35944.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35945.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35945.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35946.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35946.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35947.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35947.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35948.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35948.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35949.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35949.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35950.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35950.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:826: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ d8f279692a7c23013dacb774df231d51d9c6cd35:

		  |     0.0 opOk
		  | 35828.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35828.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35828.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35829.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35829.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35829.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35829.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35830.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35830.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35830.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35830.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35831.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35831.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35831.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35831.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35832.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35832.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35832.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35832.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35833.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35833.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35833.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35833.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35834.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35834.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35834.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35834.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35835.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35835.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35835.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35835.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35836.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35836.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35836.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35836.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35837.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35837.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35837.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35837.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35838.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35838.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35838.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35838.0s       19            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:826: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@jlinder jlinder added the T-sql-schema-deprecated Use T-sql-foundations instead label Jun 16, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ c38781390f0d07cf3d39d1af9ed01b4e370aaffa:

		  |             0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35973.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35973.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35974.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35974.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35975.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35975.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35976.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35976.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:787: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ a59bb25a0c234e1752aa201fc8d6ccc4683c16b1:

		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35973.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35973.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ aa9506903f7a71f8117e89d1e344cd91a7d50cf4:

		  | 35479.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35479.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35480.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35480.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35480.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35480.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35481.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35481.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35481.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35481.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35482.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35482.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35482.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35482.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35483.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35483.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35483.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35483.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35484.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35484.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35484.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35484.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35485.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35485.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35485.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35485.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35486.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35486.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35486.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35486.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35487.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35487.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35487.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35487.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35488.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35488.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35488.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35488.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35489.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35489.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35489.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35489.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 82f3ddb1ec92af3538cc004522e6431bc9049503:

		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35872.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35873.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35874.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35875.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35876.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35877.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35878.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35878.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35879.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35879.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35880.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35880.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35881.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35881.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35882.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35882.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35883.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35883.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35884.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35884.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35885.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35885.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35886.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35886.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35887.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35887.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35888.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35888.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35889.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35889.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35890.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35890.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35891.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35891.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35892.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35892.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 28bb8e1ed0d5b9b88aefd51e5ec9a62d32514821:

		  |            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35810.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35811.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35811.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35811.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35812.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35812.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35812.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35813.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35813.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35813.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35814.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35814.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35814.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35820.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35820.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35820.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 3a28e789f6739906f0ebe333c822c22638fbeb3a:

		  |             0.0      0.0      0.0      0.0      0.0 opOk
		  | 35798.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35799.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35799.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35800.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35800.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35801.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35801.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35802.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35802.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35803.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35803.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35804.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35804.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35805.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35805.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35806.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35806.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35807.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35807.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35808.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35808.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35809.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35809.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35810.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35810.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35811.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35811.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35812.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35812.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35813.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35813.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35814.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35814.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 9bf0c8e769c56977c61579a875677ddd3aa2bcc0:

		  | ms)_pMax(ms)
		  | 35821.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35821.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35821.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35822.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35822.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35822.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35822.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35823.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35823.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35823.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35823.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35824.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35824.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35824.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35824.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35825.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35825.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35825.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35825.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35826.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35826.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35826.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35826.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35827.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35827.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35827.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35827.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35828.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35828.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35828.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35828.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35829.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35829.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35829.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35829.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35830.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35830.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35830.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35830.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35831.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35831.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35831.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35831.0s       16            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 2663b80c8bc3677ef509141bbe435de9e9781dbd:

		  |   0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 00ed8ae28914034bf12d00c17ec72205bf87ab01:

		  |             0.0      0.0      0.0      0.0      0.0 opOk
		  | 35683.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35684.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35684.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35685.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35685.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35686.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35686.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35687.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35687.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35688.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35688.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35689.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35689.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35690.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35690.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35691.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35691.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35692.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35692.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35693.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35693.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35694.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35694.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35695.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35695.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35696.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35696.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35697.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35697.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35698.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35698.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35699.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35699.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35700.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35700.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35701.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35701.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35702.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35702.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35703.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35703.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35704.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35704.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 7fe4c4d9727461bfac2356bbe195197a293f73f5:

		  |     0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35966.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35967.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35968.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35969.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35970.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35971.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35972.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 6988edf0505301f9ee52c50d2b88ef5e6999d214:

		  | 35842.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35842.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35843.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35843.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35843.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35843.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35844.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35844.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35844.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35844.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35845.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35845.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35845.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35845.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35846.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35846.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35846.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35846.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35847.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35847.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35847.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35847.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35848.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35848.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35848.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35848.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35849.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35849.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35849.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35849.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35850.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35850.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35850.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35850.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35851.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35851.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35851.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35851.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35852.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35852.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35852.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35852.0s       25            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 532d7b39beb287f98be3a674830bf718c709feab:

		  |     0.0 opOk
		  | 35884.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35884.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35884.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35885.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35885.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35885.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35885.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35886.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35886.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35886.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35886.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35887.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35887.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35887.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35887.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35888.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35888.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35888.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35888.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35889.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35889.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35889.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35889.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35890.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35890.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35890.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35890.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35891.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35891.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35891.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35891.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35892.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35892.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35892.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35892.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35893.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35893.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35893.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35893.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35894.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35894.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35894.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35894.0s       15            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 9c32f4116126c15fe5ac26ba62e4b4928b5ae371:

		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35815.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35816.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35817.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35818.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35819.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35820.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35820.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35825.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35825.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35826.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35826.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35827.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35827.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 58f6752aa6b1028448beaf8e891785ba38b620d0:

		  |            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35638.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35639.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35639.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35639.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35640.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35640.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35640.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35641.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35641.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35641.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35642.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35642.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35642.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35643.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35643.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35643.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35644.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35644.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35644.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35645.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35645.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35645.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35646.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35646.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35646.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35647.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35647.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35647.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35648.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35648.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35648.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35649.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35649.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35649.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35650.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35650.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35650.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35651.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35651.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35651.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35652.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35652.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35652.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 3bd14dde6b44ff6a0f3dc490a710f04ca33a773b:

		  | 35456.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35456.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35457.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35457.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35458.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35458.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35459.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35459.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35460.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35460.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35461.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35461.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35462.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35462.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35463.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35463.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35464.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35464.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35465.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35465.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35466.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35466.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35467.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35467.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35468.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35468.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35469.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35469.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35470.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35470.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35471.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35471.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35472.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35472.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35473.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35473.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35474.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35474.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35475.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35475.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35476.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35476.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 7b787af06d5e1794670506420e08f904539afe2f:

		  |             0.0      0.0      0.0      0.0      0.0 opOk
		  | 35681.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35682.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35682.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35683.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35683.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35684.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35684.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35685.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35685.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35686.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35686.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35687.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35687.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35688.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35688.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35689.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35689.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35690.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35690.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35691.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35691.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35692.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35692.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35693.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35693.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35694.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35694.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35695.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35695.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35696.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35696.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35697.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35697.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35698.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35698.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35699.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35699.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35700.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35700.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35701.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35701.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35702.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35702.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ d996a017141f114d8b0bb1a0fda7eec172edaff3:

		  |     0.0 opOk
		  | 35926.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35926.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35926.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35927.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35927.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35927.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35927.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35928.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35928.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35928.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35928.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35929.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35929.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35929.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35929.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35930.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35930.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35930.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35930.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35931.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35931.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35931.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35931.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35932.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35932.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35932.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35932.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35933.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35933.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35933.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35933.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35934.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35934.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35934.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35934.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35935.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35935.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35935.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35935.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35936.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35936.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35936.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35936.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 9e1e2235a05ab0b2a77f917ba7bce303f6b8fe4a:

		  |     0.0 opOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnCmtErr
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 858eb555aa3c52442d2695b80f3ea6d078b62cdb:

		  |   0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35821.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35822.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35823.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35824.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35825.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35825.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35826.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35826.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35827.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35827.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35828.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35829.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ 46315b601fdb9954ec7a2d7d4567ecae868e03ea:

		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35830.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35831.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35832.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35833.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35834.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35835.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35836.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35837.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35838.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35839.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35840.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35841.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35842.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35842.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35842.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35843.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35843.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35843.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) secondary error attachment
		  | signal: killed
		  | (1) signal: killed
		  | Error types: (1) *exec.ExitError
		Wraps: (5) context canceled
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-21.1 @ b537b9d8663060bc8befb81f98bb1479889641ea:

		  |            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35951.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35952.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35953.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35954.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35955.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35956.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35957.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35958.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35959.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35960.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35961.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35962.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35963.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35964.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 opOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnOk
		  | 35965.0s        1            0.0            0.0      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) exit status 10
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *main.withCommandDetails (4) *exec.ExitError

	test_runner.go:792: test timed out (10h0m0s)
Reproduce

To reproduce, try: 

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh schemachange/mixed-versions

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

This was referenced Mar 16, 2022
Closed
Closed
This was referenced Mar 25, 2022
Closed
Closed
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jlinder @postamar @JuanLeon1 @cockroach-teamcity