Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/mixed-versions failed #78304

Closed
cockroach-teamcity opened this issue Mar 23, 2022 · 10 comments
Closed

roachtest: schemachange/mixed-versions failed #78304

cockroach-teamcity opened this issue Mar 23, 2022 · 10 comments
Assignees
@postamar
Labels
branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 23, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ 26c05e15d3752f47a91592b5b870360c30c57dd5:

		  | runtime.gopark(0xc002020fb0, 0x2, 0x0, 0x0, 0xc002020fa4)
		  | 	GOROOT/src/runtime/proc.go:366 +0xd6 fp=0xc002020e28 sp=0xc002020e08 pc=0x462e76
		  | runtime.selectgo(0xc002020fb0, 0xc002020fa0, 0x0, 0x0, 0x0, 0x1)
		  | 	GOROOT/src/runtime/select.go:327 +0x772 fp=0xc002020f70 sp=0xc002020e28 pc=0x472c52
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:41 +0x72 fp=0xc002020fe0 sp=0xc002020f70 pc=0xd77712
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc002020fe8 sp=0xc002020fe0 pc=0x4943e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:40 +0xc5
		  |
		  | goroutine 319 [select]:
		  | runtime.gopark(0xc001e54fb0, 0x2, 0x0, 0x0, 0xc001e54fa4)
		  | 	GOROOT/src/runtime/proc.go:366 +0xd6 fp=0xc001e54e28 sp=0xc001e54e08 pc=0x462e76
		  | runtime.selectgo(0xc001e54fb0, 0xc001e54fa0, 0x0, 0x0, 0x0, 0x1)
		  | 	GOROOT/src/runtime/select.go:327 +0x772 fp=0xc001e54f70 sp=0xc001e54e28 pc=0x472c52
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:41 +0x72 fp=0xc001e54fe0 sp=0xc001e54f70 pc=0xd77712
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc001e54fe8 sp=0xc001e54fe0 pc=0x4943e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:40 +0xc5
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 1\"",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 3F000===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"public\"]] :true\nQUERY [\"\\n  WITH database_id AS (\\n                    SELECT id\\n                      FROM system.namespace\\n                     WHERE \\\"parentID\\\" = 0\\n                       AND \\\"parentSchemaID\\\" = 0\\n                       AND name = current_database()\\n                   ),\\n       schema_id AS (\\n                    SELECT nsp.id\\n                      FROM system.namespace AS nsp\\n                      JOIN database_id ON \\\"parentID\\\" = database_id.id\\n                                      AND \\\"parentSchemaID\\\" = 0\\n                                      AND name = $1\\n                 ),\\n       descriptor_ids AS (\\n                        SELECT nsp.id\\n                          FROM system.namespace AS nsp,\\n                               schema_id,\\n                               database_id\\n                         WHERE nsp.\\\"parentID\\\" = database_id.id\\n                           AND nsp.\\\"parentSchemaID\\\" = schema_id.id\\n                      ),\\n       descriptors AS (\\n                    SELECT crdb_internal.pb_to_json(\\n                            'cockroach.sql.sqlbase.Descriptor',\\n                            descriptor\\n                           ) AS descriptor\\n                      FROM system.descriptor AS descriptors\\n                      JOIN descriptor_ids ON descriptors.id\\n                                             = descriptor_ids.id\\n                   ),\\n       types AS (\\n                SELECT descriptor\\n                  FROM descriptors\\n                 WHERE (descriptor-\u003e'type') IS NOT NULL\\n             ),\\n       table_references AS (\\n                            SELECT json_array_elements(\\n                                    descriptor-\u003e'table'-\u003e'dependedOnBy'\\n                                   ) AS ref\\n                              FROM descriptors\\n                             WHERE (descriptor-\u003e'table') IS NOT NULL\\n                        ),\\n       dependent AS (\\n                    SELECT (ref-\u003e\u003e'id')::INT8 AS id FROM table_references\\n                 ),\\n       referenced_descriptors AS (\\n                                SELECT json_array_elements_text(\\n                                        descriptor-\u003e'type'-\u003e'referencingDescriptorIds'\\n                                       )::INT8 AS id\\n                                  FROM types\\n                              )\\nSELECT EXISTS(\\n        SELECT *\\n          FROM system.namespace\\n         WHERE id IN (SELECT id FROM referenced_descriptors)\\n           AND \\\"parentSchemaID\\\" NOT IN (SELECT id FROM schema_id)\\n           AND id NOT IN (SELECT id FROM dependent)\\n       );\" [\"public\"]] :false\n===========================Previous statements [ALTER DATABASE schemachange PRIMARY REGION \"us-east1\" DROP SCHEMA \"public\" CASCADE]: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "05:24:51.765884",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42602===========================Executed queries for generating errors: QUERY [SELECT region FROM [SHOW REGIONS FROM DATABASE]] : \nQUERY [SHOW DATABASE] :schemachange\n===========================Previous statements [ALTER DATABASE schemachange SURVIVE ZONE FAILURE]: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-14050
@cockroach-teamcity cockroach-teamcity added branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 23, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Mar 23, 2022
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 23, 2022
Closed
@stevendanna
Copy link
Collaborator

It looks like this might be resolved by backporting #78184

@ajwerner
Copy link
Contributor

@fqazi can you take this?

@fqazi fqazi mentioned this issue Mar 23, 2022
Merged
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ 9613c48ab9723f10ffb57f6c507b65bf819fbc95:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052946.348687330_n4_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | rs.go:79  [-] 3542 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +  FROM (
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +    SELECT name
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +      FROM (
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +           ) AS obj (name)
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +       )
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220324 06:29:45.420438 1 workload/pgx_helpers.go:79  [-] 3542 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220324 06:29:47.576050 1 workload/cli/run.go:423  [-] 3543  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table223" (238): secondary index "table223_col223_228_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table223" (238): secondary index "table223_col223_228_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220324 06:29:47.614919 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220324 06:29:47.778175 1 workload/cli/run.go:437  [-] 3545  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table223" (238): secondary index "table223_col223_228_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table223" (238): secondary index "table223_col223_228_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@fqazi fqazi mentioned this issue Mar 24, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ 9cf10ddee27563508b759df66ce8685bdca07bfb:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052749.304636765_n4_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | :46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +      FROM (
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +           ) AS obj (name)
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +       )
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220325 06:27:46.301369 1 workload/pgx_helpers.go:79  [-] 3542 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220325 06:27:48.456957 1 workload/cli/run.go:423  [-] 3543  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table88" (191): secondary index "table88_col88_98_col88_89_col88_91_col88_93_col88_94_col88_90_col88_96_col88_99_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table88" (191): secondary index "table88_col88_98_col88_89_col88_91_col88_93_col88_94_col88_90_col88_96_col88_99_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +  FROM (
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +    SELECT name
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +      FROM (
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +           ) AS obj (name)
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +       )
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220325 06:27:48.497238 1 workload/pgx_helpers.go:79  [-] 3544 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220325 06:27:50.545687 1 workload/cli/run.go:437  [-] 3545  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table88" (191): secondary index "table88_col88_98_col88_89_col88_91_col88_93_col88_94_col88_90_col88_96_col88_99_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table88" (191): secondary index "table88_col88_98_col88_89_col88_91_col88_93_col88_94_col88_90_col88_96_col88_99_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 4. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 25, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ 7fbd04334b4a5cd369d9df1afdcfa3f49707989d:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_053701.541629437_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | lpers.go:79  [-] 3520 +      FROM (
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +           ) AS obj (name)
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +       )
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220326 06:36:59.848190 1 workload/pgx_helpers.go:79  [-] 3520 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220326 06:37:01.740510 1 workload/cli/run.go:423  [-] 3521  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table268" (252): secondary index "table268_col268_273_col268_276_col268_271_col268_272_col268_270_col268_269_col268_274_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table268" (252): secondary index "table268_col268_273_col268_276_col268_271_col268_272_col268_270_col268_269_col268_274_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +  FROM (
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +    SELECT name
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +      FROM (
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +           ) AS obj (name)
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +       )
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220326 06:37:01.786849 1 workload/pgx_helpers.go:79  [-] 3522 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220326 06:37:02.928601 1 workload/cli/run.go:437  [-] 3523  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table268" (252): secondary index "table268_col268_273_col268_276_col268_271_col268_272_col268_270_col268_269_col268_274_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table268" (252): secondary index "table268_col268_273_col268_276_col268_271_col268_272_col268_270_col268_269_col268_274_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ 1226d6f5d25b6bd41264da64fe7e7ec0db1848f2:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052959.923763572_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | t(name, '[0-9]+$')::INT8)
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +  FROM (
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +    SELECT name
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +      FROM (
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +           ) AS obj (name)
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +       )
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220327 06:29:58.775250 1 workload/pgx_helpers.go:79  [-] 3528 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220327 06:30:00.749161 1 workload/cli/run.go:423  [-] 3529  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table350" (252): secondary index "table350_col350_354_col350_352_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table350" (252): secondary index "table350_col350_354_col350_352_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +  FROM (
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +    SELECT name
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +      FROM (
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +           ) AS obj (name)
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +       )
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220327 06:30:00.798438 1 workload/pgx_helpers.go:79  [-] 3530 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220327 06:30:01.275696 1 workload/cli/run.go:437  [-] 3531  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table350" (252): secondary index "table350_col350_354_col350_352_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table350" (252): secondary index "table350_col350_354_col350_352_idx" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052716.835584051_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | elpers.go:79  [-] 3540  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table239" (229): primary index column "col239_252" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +  FROM (
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +    SELECT name
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +      FROM (
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +           ) AS obj (name)
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +       )
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220328 06:27:14.201702 1 workload/pgx_helpers.go:79  [-] 3540 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220328 06:27:16.024075 1 workload/cli/run.go:423  [-] 3541  retrying after error while creating load: failed to initialize the load generator: ERROR: relation "table239" (229): primary index column "col239_252" cannot be virtual (SQLSTATE XXUUU)
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "table239" (229): primary index column "col239_252" cannot be virtual (SQLSTATE XXUUU) sql:
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +  FROM (
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +    SELECT name
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +      FROM (
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +           ) AS obj (name)
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +       )
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220328 06:27:16.073198 1 workload/pgx_helpers.go:79  [-] 3542 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220328 06:27:18.104935 1 workload/cli/run.go:437  [-] 3543  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: relation "table239" (229): primary index column "col239_252" cannot be virtual (SQLSTATE XXUUU)
		  | Error: failed to initialize the load generator: ERROR: relation "table239" (229): primary index column "col239_252" cannot be virtual (SQLSTATE XXUUU)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ fed021b0766c184d231b9a67e5c90d36e81f7c0a:

		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_052927.822648180_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | me
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +      FROM (
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +           ) AS obj (name)
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +       )
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220329 06:29:26.336070 1 workload/pgx_helpers.go:79  [-] 3530 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220329 06:29:28.251559 1 workload/cli/run.go:423  [-] 3531  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table452" (277): secondary index "table452_col452_466_expr_col452_459_col452_462_col452_458_col452_469_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table452" (277): secondary index "table452_col452_466_expr_col452_459_col452_462_col452_458_col452_469_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +  FROM (
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +    SELECT name
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +      FROM (
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +           ) AS obj (name)
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +       )
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220329 06:29:28.333824 1 workload/pgx_helpers.go:79  [-] 3532 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220329 06:29:29.106379 1 workload/cli/run.go:437  [-] 3533  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table452" (277): secondary index "table452_col452_466_expr_col452_459_col452_462_col452_458_col452_469_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table452" (277): secondary index "table452_col452_466_expr_col452_459_col452_462_col452_458_col452_469_expr1_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/mixed-versions failed with artifacts on release-22.1 @ ff4e23e53388d3639f4b1b23b66e87d436556a8f:

		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeMixedVersions
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:146
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeMixedVersions.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_schemachange.go:38
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:866
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_053201.653985179_n3_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4} returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 8 +      FROM (
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +           ) AS obj (name)
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +       )
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220330 06:31:58.972051 1 workload/pgx_helpers.go:79  [-] 3538 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | W220330 06:32:01.268146 1 workload/cli/run.go:423  [-] 3539  retrying after error while creating load: failed to initialize the load generator: ERROR: internal error: relation "table398" (265): secondary index "table398_col398_407_col398_405_col398_403_expr_col398_411_col398_406_col398_409_col398_401_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540  pgx logger [error]: Query logParams=map[args:[] err:ERROR: internal error: relation "table398" (265): secondary index "table398_col398_407_col398_405_col398_403_expr_col398_411_col398_406_col398_409_col398_401_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000) sql:
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +SELECT max(regexp_extract(name, '[0-9]+$')::INT8)
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +  FROM (
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +    SELECT name
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +      FROM (
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT table_name FROM [SHOW TABLES]) UNION
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT sequence_name FROM [SHOW SEQUENCES]) UNION
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT name FROM [SHOW ENUMS]) UNION
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +	           (SELECT schema_name FROM [SHOW SCHEMAS]) UNION
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT column_name FROM information_schema.columns) UNION
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +						 (SELECT index_name FROM information_schema.statistics)
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +           ) AS obj (name)
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +       )
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 + WHERE name ~ '^(table|view|seq|enum|schema)[0-9]+$'
		  | I220330 06:32:01.318211 1 workload/pgx_helpers.go:79  [-] 3540 +    OR name ~ '^(col|index)[0-9]+_[0-9]+$';]
		  | E220330 06:32:02.866270 1 workload/cli/run.go:437  [-] 3541  Attempt to create load generator failed. It's been more than 1h0m0s since we started trying to create the load generator so we're giving up. Last failure: failed to initialize the load generator: ERROR: internal error: relation "table398" (265): secondary index "table398_col398_407_col398_405_col398_403_expr_col398_411_col398_406_col398_409_col398_401_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  | Error: failed to initialize the load generator: ERROR: internal error: relation "table398" (265): secondary index "table398_col398_407_col398_405_col398_403_expr_col398_411_col398_406_col398_409_col398_401_key" has invalid version 4 which is for primary indexes (SQLSTATE XX000)
		  |
		  | stdout:
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 3. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=true --max-ops 100 --concurrency 5 {pgurl:1-4}
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@postamar
Copy link
Contributor

I expect this issue to have been resolved by #79000

@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@postamar postamar

Labels
branch-release-22.1 Used to mark GA and release blockers, technical advisories, and bugs for 22.1 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@stevendanna @fqazi @ajwerner @postamar @celiala @cockroach-teamcity