Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/random-load failed #70016

Closed
cockroach-teamcity opened this issue Sep 10, 2021 · 171 comments · Fixed by #78658
Closed

roachtest: schemachange/random-load failed #70016

cockroach-teamcity opened this issue Sep 10, 2021 · 171 comments · Fixed by #78658
Assignees
@fqazi
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 10, 2021 •
edited by cockroach-jira-scripts
Loading

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 99a4816fc272228a63df20dae3cc41d235e705f3:

		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:50:15.270454",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA IF NOT EXISTS schema30 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot add region \"us-west1\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:50:15.685587",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-east1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot add region \"us-east1\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:50:15.267228",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:50:15.266796",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: region \"us-west1\" already added to database (SQLSTATE 42710)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-9901
@cockroach-teamcity cockroach-teamcity added branch-release-21.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Sep 10, 2021
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Sep 10, 2021
@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 11, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 4811bb9c8c1d20b9f200d44bc7df388ad2d80898:

		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:40:37.375216",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE VIEW public.view87 AS SELECT public.table48.col48_53, public.table48.col48_52, public.table48.col48_49, public.table48.col48_50, public.table48.col48_51, public.table48.col48_52, public.table48.col48_53, public.table48.col48_50, public.table48.col48_49, public.table48.col48_52, public.table48.col48_51 FROM public.table48, public.table48, public.table48"
		  |  ],
		  |  "expectedExecErrors": "42701,42712",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: source name \"table48\" specified more than once (missing AS clause) (SQLSTATE 42712)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:40:37.860643",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE INDEX index70_82 ON public.table70 (col70_71 DESC) STORING (col70_74)",
		  |   "CREATE INDEX IF NOT EXISTS index48_90 ON public.table48 (col48_51, col48_50 DESC, col48_53 DESC, col48_52 ASC)"
		  |  ],
		  |  "expectedExecErrors": "0A000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: unimplemented: column col48_51 is of type geometry and thus is not indexable (SQLSTATE 0A000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:40:37.546193",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table48 ALTER COLUMN \"col48_52\" DROP DEFAULT",
		  |   "CREATE VIEW public.view91 AS SELECT public.table48.col48_51, public.table48.col48_49, public.table48.col48_50, public.table48.col48_53, public.table48.col48_52, public.table48.col48_49, public.table48.col48_51, public.table48.col48_50, public.table48.col48_53, public.table48.col48_50, public.table48.col48_51, public.table48.col48_52, public.table48.col48_49 FROM public.table48, public.table48, public.table48"
		  |  ],
		  |  "expectedExecErrors": "42701,42712",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: source name \"table48\" specified more than once (missing AS clause) (SQLSTATE 42712)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:40:37.401169",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table48 ALTER COLUMN \"col48_50\" SET NOT NULL",
		  |   "ALTER TABLE public.table48 ALTER COLUMN \"col48_52\" DROP NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Validation FAIL:\nid 68, db schemachange, schema public, name table48: mutation job 692355197303750657: job not found"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 8d491ced731e13da2377cda9961f577d7487d6a0:

		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER TABLE public.table93 ALTER COLUMN IrrelevantColumnName SET DEFAULT \"IrrelevantValue\""
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table93\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:31:01.703025",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=20ed4685 key=/Table/SystemConfigSpan/Start pri=0.00808979 epo=0 ts=1631532661.916913896,1 min=1631532661.702960463,0 seq=39} lock=true stat=PENDING rts=1631532661.702960463,0 wto=false gul=1631532662.202960463,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:31:01.736907",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema63 AUTHORIZATION root",
		  |   "CREATE TYPE public.enum94 AS ENUM ('adfd', 'daf')",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=2045f510 key=/Table/SystemConfigSpan/Start pri=0.00055577 epo=0 ts=1631532662.010066433,1 min=1631532661.736822886,0 seq=24} lock=true stat=PENDING rts=1631532661.736822886,0 wto=false gul=1631532662.236822886,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:31:01.653537",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq95",
		  |   "CREATE SCHEMA schema96 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "***FAIL; Failed to receive an execution error when errors were expected"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 14411b999aae710ca0f4a6376d58e302b197281b:

		  |  "workerId": 0,
		  |  "clientTimestamp": "12:31:38.549179",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE schema4.enum92 AS ENUM ('saa', 's')",
		  |   "ALTER DATABASE schemachange SURVIVE REGION FAILURE"
		  |  ],
		  |  "expectedExecErrors": "22023,42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:31:38.17109",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-east1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot add region \"us-east1\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:31:38.17901",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:31:38.188685",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE IF NOT EXISTS public.table42 (col42_43 NAME NOT NULL, col42_44 INT4 NULL, col42_45 REGROLE, col42_46 STRING NOT NULL, col42_47 GEOGRAPHY NOT NULL, col42_48 BYTES, col42_49 TIMETZ NULL, col42_50 DECIMAL NULL, col42_51 REGPROC NOT NULL, col42_52 TIMESTAMP[] NOT NULL, col42_53 DECIMAL NULL AS (col42_50 + (-3.422671353499638127E+30):::DECIMAL) VIRTUAL, FAMILY (col42_48), FAMILY (col42_49), FAMILY (col42_43, col42_51), FAMILY (col42_44, col42_46, col42_50, col42_45, col42_47, col42_52))",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation \"table42\" (71): invalid locality config: database schemachange is multi-region enabled, but table table42 has no locality set (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ d3fc366bcba04ab19f4cb59844212e908e2a9aa8:

		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot create \"schema59.seq60\" because the target database or schema does not exist (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:23:49.157987",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SEQUENCE public.seq66"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.seq66\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:23:48.939175",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER TABLE schema64.table65 ALTER COLUMN IrrelevantColumnName SET NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema64.table65\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:23:49.254526",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:23:48.961179",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "***FAIL; Failed to receive an execution error when errors were expected"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ ed81878bb4697c3a0a09e69b8a5053566297b142:

		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: index \"index59_193\" does not exist (SQLSTATE 42704)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:04.539385",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema179.table180 RENAME TO public.table187"
		  |  ],
		  |  "expectedExecErrors": "42602,42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema179.table180\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:04.605099",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA IF NOT EXISTS schema183 AUTHORIZATION root",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schemachange 52 2}], old versions still in use (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:04.380373",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table59 ALTER COLUMN \"col59_60\" DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42611",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col59_60\" is not a computed column (SQLSTATE 42611)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:03.866095",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table59 RENAME COLUMN \"col59_60\" TO \"col59_182\"",
		  |   "CREATE INDEX IF NOT EXISTS index59_199 ON public.table59 (col59_60)"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: column \"col59_60\" does not exist (SQLSTATE 42703)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 0babf97f52ed9e44036851b2a9868e17eeee95ed:

		  |   "CREATE SCHEMA IF NOT EXISTS schema63 AUTHORIZATION root",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=d2f2fd93 key=/Table/SystemConfigSpan/Start pri=0.01941626 epo=0 ts=1631881643.409447523,1 min=1631881642.896642076,0 seq=30} lock=true stat=PENDING rts=1631881642.896642076,0 wto=false gul=1631881643.396642076,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:27:22.898491",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=0a47e379 key=/Table/SystemConfigSpan/Start pri=0.01247013 epo=0 ts=1631881643.409447523,1 min=1631881642.898424145,0 seq=24} lock=true stat=PENDING rts=1631881642.898424145,0 wto=false gul=1631881643.398424145,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:27:22.897299",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "CREATE TYPE public.enum64 AS ENUM ('f')",
		  |   "CREATE TABLE public.table65 (col65_66 TIMESTAMP NOT NULL, col65_67 CHAR NOT NULL, col65_68 INT8 NOT NULL, PRIMARY KEY (col65_67 DESC, col65_68, col65_66 DESC), col65_69 STRING NOT NULL AS (lower(CAST(col65_66 AS STRING))) STORED, col65_70 STRING NOT NULL AS (lower(CAST(col65_66 AS STRING))) VIRTUAL, UNIQUE (col65_67, lower(CAST(col65_66 AS STRING)) DESC), INDEX (col65_70 ASC, col65_69 ASC), FAMILY (col65_68, col65_69), FAMILY (col65_66, col65_67))",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=82ec5025 key=/Table/SystemConfigSpan/Start pri=0.00227815 epo=0 ts=1631881643.409447523,2 min=1631881642.897230899,0 seq=41} lock=true stat=PENDING rts=1631881642.897230899,0 wto=false gul=1631881643.397230899,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:27:22.898298",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "CREATE TABLE public.table35 (col35_36 VARCHAR NOT NULL, col35_37 TIMESTAMPTZ NULL, col35_38 VARCHAR, col35_39 JSONB NOT NULL, col35_40 INT2 NULL, col35_41 REGROLE NOT NULL, PRIMARY KEY (col35_41 DESC))",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Validation FAIL:\nid 63, db schemachange, schema public, name table35: relation \"table35\" (63): invalid locality config: database schemachange is multi-region enabled, but table table35 has no locality set"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ a29fd0ad538d51050d82e92583a8206983622f6b:

		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.seq38\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:55:04.169059",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE REGION FAILURE"
		  |  ],
		  |  "expectedExecErrors": "22023,42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:55:04.174372",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange ADD REGION \"europe-west2\"",
		  |   "CREATE SCHEMA IF NOT EXISTS schema41 AUTHORIZATION root",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=d355e271 key=/Table/SystemConfigSpan/Start pri=0.00744406 epo=0 ts=1631966104.484505951,1 min=1631966104.174263983,0 seq=52} lock=true stat=PENDING rts=1631966104.174263983,0 wto=false gul=1631966104.674263983,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:55:04.173522",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA IF NOT EXISTS schema13 AUTHORIZATION root",
		  |   "CREATE TABLE public.table42 (col42_43 BOX2D NOT NULL, col42_44 BOOL NOT NULL, col42_45 INET NOT NULL, col42_46 INT4 NOT NULL, col42_47 DECIMAL NOT NULL, col42_48 REGCLASS NOT NULL, col42_49 FLOAT4 NOT NULL, PRIMARY KEY (col42_45 ASC, col42_46, col42_48 DESC, col42_43 DESC, col42_49 ASC, col42_47, col42_44 ASC), col42_50 STRING NOT NULL AS (lower(CAST(col42_48 AS STRING))) STORED, col42_51 DECIMAL NOT NULL AS (col42_47 + 1570938224636458.367:::DECIMAL) STORED, UNIQUE (col42_43 ASC, col42_50 ASC, col42_47 ASC) STORING (col42_51))",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq52",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation \"table42\" (73): invalid locality config: database schemachange is multi-region enabled, but table table42 has no locality set (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 5202c97fcfe7e51e244335c3cd68bd33647860ff:

		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:17.955325",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot add region \"us-west1\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:17.890137",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "DROP SEQUENCE IF EXISTS public.seq137",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:17.873511",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema1.seq133",
		  |   "ALTER TABLE public.table152 ALTER COLUMN \"IrrelevantColumnName\" DROP DEFAULT"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table152\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:17.376042",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema94 AUTHORIZATION root",
		  |   "CREATE TABLE schema1.table120 (col120_121 TIME NOT NULL, col120_122 NAME NOT NULL, col120_123 BYTES NOT NULL, col120_124 DECIMAL NULL, col120_125 VARCHAR NOT NULL, col120_126 REGPROC NULL, col120_127 TIME NOT NULL, col120_128 TIMESTAMP NOT NULL, col120_129 INT4 NOT NULL, col120_130 STRING, PRIMARY KEY (col120_128, col120_125, col120_127, col120_129 DESC, col120_123), col120_131 STRING NULL AS (lower(CAST(col120_126 AS STRING))) VIRTUAL, INDEX ((CASE WHEN col120_123 IS NULL THEN e'=\\x03aW2\u003eq ':::STRING ELSE 'X':::STRING END) DESC) STORING (col120_121, col120_124, col120_126), INDEX (col120_131, col120_122 DESC, col120_124 ASC, (col120_124 + (-1.234E+401):::DECIMAL) DESC, col120_125 ASC, col120_127 ASC, col120_126, lower(col120_122) ASC, col120_129 ASC, col120_128 DESC, col120_123 DESC) STORING (col120_130), INDEX (col120_127 ASC, col120_121 DESC, col120_126 ASC, col120_125 ASC, col120_123, col120_124 ASC), UNIQUE (col120_122 DESC, col120_121 DESC, col120_131, col120_127, col120_126), UNIQUE (col120_131 DESC, col120_122 ASC, col120_123, col120_125 ASC, col120_124 ASC, col120_130, lower(col120_122), col120_129 ASC, col120_121 ASC, col120_128 DESC, col120_126), INDEX (col120_128), INDEX (col120_124 ASC, col120_128 DESC, col120_123, col120_130 DESC, col120_122 DESC) STORING (col120_121), FAMILY (col120_127), FAMILY (col120_124, col120_130, col120_123, col120_128, col120_122), FAMILY (col120_125, col120_126, col120_129, col120_121))",
		  |   "CREATE INDEX index120_136 ON schema1.table120 (col120_125, col120_126 DESC, col120_124 ASC, crdb_internal_idx_expr_3 DESC, col120_121 ASC, col120_123, crdb_internal_idx_expr DESC, col120_130 DESC, col120_128 ASC, col120_129, crdb_internal_idx_expr_2 DESC, col120_122 DESC, col120_127) STORING (crdb_internal_idx_expr_1)"
		  |  ],
		  |  "expectedExecErrors": "XXUUU",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: column \"crdb_internal_idx_expr_3\" is inaccessible and cannot be referenced (SQLSTATE 42703)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ db7de996a04d0b7e7959decd06c2cd40d80e501f:

		  |  "workerId": 0,
		  |  "clientTimestamp": "12:07:23.448217",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "CREATE TABLE schema7.table203 (col203_204 BYTES NOT NULL, col203_205 CHAR NOT NULL, col203_206 BIT(45) NOT NULL, PRIMARY KEY (col203_204, col203_206 ASC, col203_205 ASC), col203_207 STRING NOT NULL AS (CASE WHEN col203_204 IS NULL THEN 'n]':::STRING ELSE '\"':::STRING END) VIRTUAL, col203_208 STRING NOT NULL AS (lower(col203_205)) VIRTUAL, INDEX (col203_204), UNIQUE (col203_206 ASC, col203_205, col203_204), UNIQUE (col203_206 DESC, col203_204 DESC, col203_205, col203_208 DESC), UNIQUE (col203_204 ASC, col203_208 ASC, col203_205))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=b3316542 key=/Table/SystemConfigSpan/Start pri=0.01983275 epo=0 ts=1632312444.058692195,1 min=1632312443.447963048,0 seq=32} lock=true stat=PENDING rts=1632312443.447963048,0 wto=false gul=1632312443.947963048,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:07:24.059254",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange ADD REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot add region \"europe-west2\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:07:23.349814",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema178 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:07:23.53197",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq174",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: region \"us-west1\" has not been added to the database (SQLSTATE 42602)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 184cac1592349a665c39a54e688de072388e7886:

		  |   "ALTER DATABASE schemachange SURVIVE REGION FAILURE"
		  |  ],
		  |  "expectedExecErrors": "22023,42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: at least 3 regions are required for surviving a region failure (SQLSTATE 22023)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:59:55.816312",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq101",
		  |   "DROP SEQUENCE public.seq52",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=a88b7b49 key=/Table/SystemConfigSpan/Start pri=0.00242121 epo=0 ts=1632398396.230254311,1 min=1632398395.816239182,0 seq=21} lock=true stat=PENDING rts=1632398395.816239182,0 wto=false gul=1632398396.316239182,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:59:55.805055",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SEQUENCE IF EXISTS public.seq52",
		  |   "CREATE SCHEMA schema119 AUTHORIZATION root",
		  |   "CREATE TYPE public.enum120 AS ENUM ('d', 'f', 'ddsa', 'fadads', 'dsfd')",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq121",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=6af9fb0a key=/Table/SystemConfigSpan/Start pri=0.13024011 epo=0 ts=1632398396.903724155,1 min=1632398395.804993512,0 seq=45} lock=true stat=PENDING rts=1632398395.804993512,0 wto=false gul=1632398396.304993512,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:59:55.848977",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE IF NOT EXISTS public.table103 (col103_104 BIT(22), col103_105 INET NOT NULL, col103_106 INET NULL, col103_107 \"char\", col103_108 REGCLASS NOT NULL, col103_109 FLOAT4 NOT NULL, col103_110 REGNAMESPACE NULL, col103_111 DECIMAL NOT NULL, col103_112 JSONB, col103_113 GEOMETRY NOT NULL, col103_114 BOOL NOT NULL, PRIMARY KEY (col103_105 ASC, col103_108 DESC, col103_111 DESC, col103_109 DESC, col103_114), col103_115 STRING AS (lower(CAST(col103_104 AS STRING))) STORED, col103_116 STRING AS (lower(CAST(col103_112 AS STRING))) STORED, INDEX (lower(CAST(col103_105 AS STRING)) DESC, col103_111, col103_106, col103_104), UNIQUE (col103_114 DESC, col103_104 DESC, lower(CAST(col103_114 AS STRING)), col103_110 ASC), INDEX (col103_116, col103_109 DESC) STORING (col103_106, col103_113), FAMILY (col103_116, col103_109), FAMILY (col103_112), FAMILY (col103_115), FAMILY (col103_114, col103_110), FAMILY (col103_105, col103_104, col103_106, col103_107), FAMILY (col103_113), FAMILY (col103_111), FAMILY (col103_108))",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation \"table103\" (98): invalid locality config: database schemachange is multi-region enabled, but table table103 has no locality set (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ c46f5a5a098577b936e56f03d20c97300b4cce61:

		  |  "workerId": 0,
		  |  "clientTimestamp": "11:47:51.228806",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE public.enum2 AS ENUM ('fddf')",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "CREATE INDEX \"IrrelevantName\" ON schema37.table38 (\"IrrelevantColumn\" ASC)"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema37.table38\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:47:51.229228",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE REGION FAILURE"
		  |  ],
		  |  "expectedExecErrors": "22023,42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:47:51.228263",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "CREATE TABLE schema39.table40 (col40_41 TIMESTAMP, col40_42 OID NULL, col40_43 GEOGRAPHY NULL, col40_44 OID, col40_45 GEOGRAPHY, col40_46 OID NOT NULL, col40_47 JSONB NOT NULL, col40_48 CHAR NULL, col40_49 FLOAT8 NOT NULL, col40_50 GEOMETRY NOT NULL, col40_51 REGCLASS NULL, col40_52 REGPROCEDURE NOT NULL, col40_53 INT2 NOT NULL, col40_54 STRING NULL AS (lower(CAST(col40_51 AS STRING))) VIRTUAL, col40_55 STRING NULL AS (lower(CAST(col40_51 AS STRING))) VIRTUAL, col40_56 STRING AS (lower(CAST(col40_41 AS STRING))) STORED, col40_57 STRING NOT NULL AS (lower(CAST(col40_50 AS STRING))) STORED, col40_58 FLOAT8 NOT NULL AS (col40_49 + 'NaN':::FLOAT8) STORED, INVERTED INDEX (col40_47 ASC), FAMILY (col40_41, col40_49), FAMILY (col40_44, col40_48), FAMILY (col40_45), FAMILY (col40_56, col40_46), FAMILY (col40_50, col40_52), FAMILY (col40_43, col40_53), FAMILY (col40_58, col40_42), FAMILY (col40_51), FAMILY (col40_57), FAMILY (col40_47))"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot create \"schema39.table40\" because the target database or schema does not exist (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:47:51.228764",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema1 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
		  |   "ALTER DATABASE schemachange ADD REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "***FAIL; Failed to receive an execution error when errors were expected"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 24021ba163e4ac438b169d575cf1527a4aae394d:

		  |   | ``````
		  |   | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  |   | ``````
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | :37:47.601296",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SEQUENCE public.seq173",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "CREATE TABLE IF NOT EXISTS public.table346 (col346_347 GEOGRAPHY NOT NULL, col346_348 REGNAMESPACE NOT NULL, col346_349 GEOMETRY, col346_350 INT8, col346_351 INT4, col346_352 BOX2D, col346_353 INT8 NULL, col346_354 UUID NOT NULL, col346_355 DATE NOT NULL, FAMILY (col346_351, col346_350), FAMILY (col346_352, col346_354, col346_355), FAMILY (col346_347), FAMILY (col346_349, col346_348, col346_353))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=b66c34e9 key=/Table/SystemConfigSpan/Start pri=0.01001218 epo=0 ts=1632573467.837100539,1 min=1632573467.601209870,0 seq=20} lock=true stat=PENDING rts=1632573467.601209870,0 wto=false gul=1632573468.101209870,0 (SQLSTATE 40001)"
		  | }
		  |    10.0s        0            7.0           12.8     65.0    251.7    251.7    251.7 opOk
		  |    10.0s        0            1.0            1.5   2684.4   2684.4   2684.4   2684.4 txnCmtErr
		  |    10.0s        0            0.0            1.2      0.0      0.0      0.0      0.0 txnOk
		  |    10.0s        0            4.0           11.9      0.9   2013.3   2013.3   2013.3 txnRbk
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:37:47.505387",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE public.enum228 AS ENUM ('df', 'sddsd', 'da', 'fsfsdd', 'saffds', 'daf')"
		  |  ],
		  |  "expectedExecErrors": "42710",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: type \"schemachange.public.enum228\" already exists (SQLSTATE 42710)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:37:47.698774",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE IF NOT EXISTS public.table307 (col307_308 UUID NOT NULL, col307_309 GEOGRAPHY, col307_310 INT2 NOT NULL, col307_311 REGTYPE, col307_312 DECIMAL, col307_313 OID, col307_314 OID, col307_315 \"char\" NOT NULL, col307_316 INT2 NULL, col307_317 REGPROCEDURE NOT NULL, col307_318 REGROLE, col307_319 GEOMETRY, col307_320 OID NOT NULL, col307_321 TIMETZ, col307_322 OID NOT NULL, col307_323 INT2 NULL AS (col307_316 + 22778:::INT8) STORED, col307_324 STRING AS (lower(CAST(col307_321 AS STRING))) STORED, col307_325 STRING AS (lower(CAST(col307_311 AS STRING))) VIRTUAL, col307_326 DECIMAL AS (col307_312 + '-Infinity':::DECIMAL) STORED, UNIQUE (col307_322, col307_312, col307_323, col307_318, col307_311 ASC, col307_324, col307_316 DESC, col307_317 DESC, col307_320 ASC, col307_313, col307_326, col307_314, col307_325 DESC, col307_321 DESC) STORING (col307_309, col307_310, col307_315, col307_319), UNIQUE (col307_313, col307_323, col307_322 DESC, col307_317, col307_326 DESC) STORING (col307_308, col307_311, col307_312, col307_316, col307_321, col307_324), UNIQUE (lower(CAST(col307_308 AS STRING)) DESC, col307_313 ASC, col307_323 ASC, col307_320 DESC) STORING (col307_308, col307_311, col307_312, col307_317, col307_318, col307_321, col307_322, col307_324), FAMILY (col307_315), FAMILY (col307_322, col307_311, col307_309, col307_326), FAMILY (col307_308), FAMILY (col307_317), FAMILY (col307_324, col307_314), FAMILY (col307_310), FAMILY (col307_319, col307_316, col307_312, col307_321, col307_313, col307_323), FAMILY (col307_318, col307_320))",
		  |   "ALTER TABLE public.table307 ADD COLUMN col307_345 OID NULL",
		  |   "CREATE INDEX IF NOT EXISTS index307_356 ON public.table307 (col307_309, col307_323 DESC, col307_319 DESC, col307_326 ASC, col307_314 ASC, col307_312 DESC, col307_316 DESC, col307_318, col307_311, col307_313 ASC, col307_308, crdb_internal_idx_expr, col307_322, col307_317 ASC) STORING (col307_324, col307_325, col307_310, col307_320)"
		  |  ],
		  |  "expectedExecErrors": "0A000,XXUUU",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: column \"crdb_internal_idx_expr\" is inaccessible and cannot be referenced (SQLSTATE 42703)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 947d24dc88c05bf698d8011e1b5cf3b2017f26a3:

		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE public.table86 AS SELECT \"IrrelevantColumnName\", public.table43.col43_44, public.table43.col43_49, public.table43.col43_54, public.table43.col43_48, public.table43.col43_46, public.table43.col43_45, public.table43.col43_56, public.table43.col43_52, public.table43.crdb_internal_idx_expr_1, public.table43.crdb_internal_idx_expr_3, public.table43.col43_51, public.table43.crdb_internal_idx_expr_4, public.table43.col43_58, public.table43.col43_50, public.table43.col43_47, public.table43.crdb_internal_idx_expr_2, public.table43.col43_53, public.table43.crdb_internal_idx_expr, public.table43.col43_57, public.table43.col43_55 FROM public.table81, public.table43"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table81\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:06:23.284517",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema82.table83 RENAME COLUMN \"IrrelevantColumnName\" TO \"OtherIrrelevantName\""
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema82.table83\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:06:23.332893",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema1.seq87",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=331f476b key=/Table/SystemConfigSpan/Start pri=0.00679004 epo=0 ts=1632830783.395552058,1 min=1632830783.332797968,0 seq=8} lock=true stat=PENDING rts=1632830783.332797968,0 wto=false gul=1632830783.832797968,0 (SQLSTATE 40001)"
		  | }
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |     1.0s        0           18.4           20.0     12.1    109.1    184.5    184.5 opOk
		  |     1.0s        0            1.8            2.0    285.2    402.7    402.7    402.7 txnCmtErr
		  |     1.0s        0            0.9            1.0    671.1    671.1    671.1    671.1 txnOk
		  |     1.0s        0           30.4           33.0     12.6    201.3    268.4    268.4 txnRbk
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:06:23.395683",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE INDEX index43_88 ON public.table43 (crdb_internal_idx_expr_3 DESC, col43_48, col43_49, col43_51, crdb_internal_idx_expr_4 ASC, col43_52, col43_54 DESC, col43_45, crdb_internal_idx_expr_1 ASC, col43_50 ASC, crdb_internal_idx_expr ASC, col43_53 ASC) STORING (col43_46, crdb_internal_idx_expr_2, col43_47, col43_58, col43_44, col43_56, col43_55, col43_57)"
		  |  ],
		  |  "expectedExecErrors": "0A000,XXUUU",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: column \"crdb_internal_idx_expr_3\" is inaccessible and cannot be referenced (SQLSTATE 42703)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 8d1d2b688ceedb0e5577d1da789da74d060c92ad:

		  |   "CREATE TABLE public.table85 (col85_86 REGTYPE, col85_87 REGCLASS NULL, col85_88 TIME NULL, col85_89 JSONB NULL, col85_90 REGNAMESPACE NULL, col85_91 REGNAMESPACE NOT NULL, col85_92 JSONB, col85_93 GEOGRAPHY NULL, col85_94 STRING NULL AS (lower(CAST(col85_87 AS STRING))) VIRTUAL, col85_95 STRING AS (lower(CAST(col85_92 AS STRING))) STORED, col85_96 STRING NULL AS (lower(CAST(col85_87 AS STRING))) STORED, col85_97 STRING NULL AS (lower(CAST(col85_93 AS STRING))) STORED, col85_98 STRING NULL AS (lower(CAST(col85_90 AS STRING))) VIRTUAL, INVERTED INDEX (col85_89), FAMILY (col85_91, col85_96, col85_88, col85_97), FAMILY (col85_92, col85_90, col85_87), FAMILY (col85_95, col85_93), FAMILY (col85_89), FAMILY (col85_86))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=120cb015 key=/Table/SystemConfigSpan/Start pri=0.03352310 epo=0 ts=1632916331.921246505,1 min=1632916331.669613803,0 seq=7} lock=true stat=PENDING rts=1632916331.669613803,0 wto=false gul=1632916332.169613803,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:52:11.748089",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq101",
		  |   "DROP TABLE schema1.table111"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema1.table111\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:52:11.919429",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema1.seq105",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  |     2.0s        0            6.0           14.5    939.5   1208.0   1208.0   1208.0 opOk
		  |     2.0s        0            2.0            1.5    939.5   1140.9   1140.9   1140.9 txnCmtErr
		  |     2.0s        0            0.5            0.5   1208.0   1208.0   1208.0   1208.0 txnOk
		  |     2.0s        0            3.0           17.5      0.8      0.8      0.8      0.8 txnRbk
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:52:11.511721",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: region \"europe-west2\" has not been added to the database (SQLSTATE 42602)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 88d8c1ce72a3988332e502e103421996d3908ab3:

		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table273\" does not exist (SQLSTATE 42P01)"
		  | }
		  |     3.0s        0           28.0           30.3     37.7    402.7    436.2    436.2 opOk
		  |     3.0s        0            4.0            4.3    570.4    939.5    939.5    939.5 txnCmtErr
		  |     3.0s        0            4.0            2.7    604.0   2684.4   2684.4   2684.4 txnOk
		  |     3.0s        0           26.0           25.7      5.8    260.0    352.3    352.3 txnRbk
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:57:29.316934",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange ADD REGION \"europe-west2\"",
		  |   "ALTER DATABASE schemachange ADD REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "42710",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: region \"europe-west2\" already added to database (SQLSTATE 42710)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:57:29.183019",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema242 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=34551422 key=/Table/SystemConfigSpan/Start pri=0.00889963 epo=0 ts=1633003049.844152512,1 min=1633003049.182927886,0 seq=31} lock=true stat=PENDING rts=1633003049.182927886,0 wto=false gul=1633003049.682927886,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:57:29.452395",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE IF NOT EXISTS public.table243 (col243_244 TIMESTAMP NULL, col243_245 NAME, col243_246 DECIMAL NULL, col243_247 REGTYPE NOT NULL, col243_248 DATE, INDEX (col243_244 ASC, col243_245 ASC, col243_247, col243_246, col243_248 ASC), UNIQUE (col243_244 DESC, col243_245 DESC, col243_247 ASC), UNIQUE (col243_246, col243_245 DESC), INDEX (col243_245, col243_246, lower(col243_245) ASC, col243_247, col243_248 DESC), INDEX (col243_247, col243_244 DESC, col243_245, col243_246 DESC, col243_248 DESC), UNIQUE (col243_244, col243_247 DESC) STORING (col243_245, col243_246, col243_248))",
		  |   "ALTER TABLE public.table243 RENAME COLUMN \"col243_247\" TO \"col243_276\"",
		  |   "DROP TABLE public.table243 RESTRICT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: ERROR: relation \"public.table243\" does not exist (SQLSTATE 42P01)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 57bb9046cd2a3090dc5f17b79912f34db8dde33e:

		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | SC, col90_95, col90_109 DESC, col90_105 ASC, col90_100 ASC, col90_103 ASC, col90_107 DESC, col90_93 ASC), UNIQUE (col90_95, col90_106, col90_102 DESC, col90_103, col90_93, col90_91 ASC, col90_104, col90_108 ASC, col90_100, col90_98 DESC, col90_94 ASC, col90_109), INDEX (col90_108 DESC, col90_93 ASC, lower(CAST(col90_91 AS STRING)), col90_107) STORING (col90_91, col90_92, col90_95, col90_100), INDEX (col90_105, col90_103 DESC) STORING (col90_91, col90_92, col90_93, col90_95, col90_100, col90_102, col90_108))",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schemachange 52 1}], old versions still in use (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:12.889604",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SCHEMA \"schema76\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: unknown schema \"schema76\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:12.368821",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA IF NOT EXISTS schema69 AUTHORIZATION root",
		  |   "CREATE TABLE schema1.table113 (col113_114 INTERVAL, col113_115 INT2 NULL, col113_116 REGTYPE, col113_117 DECIMAL NULL, col113_118 BIT(48) NOT NULL, col113_119 OID NULL, col113_120 TIMETZ, col113_121 OID NULL, col113_122 INT8 NULL, col113_123 INT8 NOT NULL, col113_124 TIME, col113_125 INT2 NULL AS (col113_115 + 2021:::INT8) STORED, col113_126 INT8 NULL AS (col113_122 + NULL) VIRTUAL, col113_127 INT8 NULL AS (col113_122 + 6742261453571753526:::INT8) VIRTUAL, col113_128 INT2 NULL AS (col113_115 + col113_123) VIRTUAL, UNIQUE (col113_115 DESC, col113_116 ASC, col113_126 DESC, col113_124 ASC, col113_121, ((col113_122 + col113_115) + col113_123) ASC, col113_125, lower(CAST(col113_120 AS STRING)), col113_118), UNIQUE (col113_127 ASC, col113_120, (col113_115 + (-18802):::INT8)), UNIQUE (col113_114, col113_115 ASC, col113_122 ASC, col113_119, col113_126 DESC, col113_124 DESC, col113_121 ASC, col113_118 DESC, col113_123 DESC, col113_125, col113_120 DESC, col113_116 ASC) STORING (col113_117), INDEX (col113_126, col113_114 DESC, col113_128 ASC, col113_117, col113_116) STORING (col113_115, col113_118, col113_119, col113_121, col113_123), UNIQUE (col113_122 ASC, col113_124 DESC, col113_115 ASC, col113_125 ASC, col113_123 DESC, col113_116, col113_126 DESC, col113_121 ASC), INDEX (col113_121 ASC, col113_123 DESC, col113_128 DESC, col113_118 ASC, col113_122 ASC, col113_124, col113_117 ASC, col113_119 DESC, col113_126 DESC, col113_116 ASC) STORING (col113_115), INDEX (col113_122 DESC, col113_117 ASC, col113_125, col113_118 ASC, col113_124 ASC, col113_115, col113_127 ASC, col113_114 ASC, col113_120 DESC, col113_119 ASC, col113_116 DESC, col113_126 ASC, col113_121, col113_123, col113_128 ASC), FAMILY (col113_124), FAMILY (col113_123, col113_117, col113_115), FAMILY (col113_119, col113_125), FAMILY (col113_116), FAMILY (col113_121, col113_118, col113_114), FAMILY (col113_120), FAMILY (col113_122))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=6745ce3c key=/Table/SystemConfigSpan/Start pri=0.01190756 epo=0 ts=1633089073.042178737,1 min=1633089072.368770643,0 seq=22} lock=true stat=PENDING rts=1633089072.368770643,0 wto=false gul=1633089072.868770643,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:51:12.720434",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "***FAIL; Failed to receive an execution error when errors were expected"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 47b426a9d692a9faebb3749f25fcbfef4f64af34:

		  |   "CREATE TYPE public.enum198 AS ENUM ('daa', 'ass', 'fd', 'ffa')",
		  |   "CREATE TYPE public.enum214 AS ENUM ('faafd', 'sda')",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER TABLE public.table218 ALTER COLUMN IrrelevantColumnName DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table218\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:35.164361",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema197 AUTHORIZATION root",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:35.174714",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE public.table199 (col199_200 INET NOT NULL, col199_201 INT4 NOT NULL, PRIMARY KEY (col199_201 DESC, col199_200 ASC))",
		  |   "DROP TABLE IF EXISTS public.table199 RESTRICT",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=b1214350 key=/Table/SystemConfigSpan/Start pri=0.00275676 epo=0 ts=1633176335.791423360,1 min=1633176335.174654868,0 seq=35} lock=true stat=PENDING rts=1633176335.174654868,0 wto=false gul=1633176335.674654868,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:05:34.705487",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE IF NOT EXISTS public.table185 (col185_186 TIMESTAMPTZ NOT NULL, col185_187 BOOL, col185_188 INT4 NOT NULL, col185_189 BYTES NOT NULL, col185_190 BOOL NOT NULL, col185_191 INTERVAL NOT NULL, col185_192 CHAR[] NOT NULL, PRIMARY KEY (col185_189 DESC, col185_191 ASC, col185_190 DESC), FAMILY (col185_186), FAMILY (col185_187, col185_189, col185_190, col185_191), FAMILY (col185_192, col185_188))",
		  |   "CREATE TABLE public.table216 AS SELECT public.table185.col185_186, public.table185.col185_192, public.table185.col185_187, public.table185.col185_188 FROM public.table185",
		  |   "ALTER TABLE public.table185 RENAME TO public.table219"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: scanBool: \"\\n\\t\\tSELECT json_array_length(\\n        crdb_internal.pb_to_json(\\n            'cockroach.sql.sqlbase.Descriptor',\\n            descriptor\\n        )-\u003e'table'-\u003e'mutations'\\n       )\\n       \u003e 0\\n\\t\\tFROM system.descriptor\\n\\t  WHERE id = $1::REGCLASS\\n\\t\\t\" [\"public.table185\"]: ERROR: error in argument for $1: relation \"public.table185\" does not exist (SQLSTATE 42P01)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ d1231cff60125b397ccce6c79c9aeea771cdcca4:

		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=fa327ffa key=/Table/SystemConfigSpan/Start pri=0.00875900 epo=0 ts=1633262270.756410651,1 min=1633262270.157623079,0 seq=24} lock=true stat=PENDING rts=1633262270.157623079,0 wto=false gul=1633262270.657623079,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:57:50.172783",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=2f2ff736 key=/Table/SystemConfigSpan/Start pri=0.01732316 epo=0 ts=1633262270.759287899,1 min=1633262270.172700288,0 seq=24} lock=true stat=PENDING rts=1633262270.172700288,0 wto=false gul=1633262270.672700288,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:57:50.125448",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA IF NOT EXISTS schema60 AUTHORIZATION root",
		  |   "CREATE SCHEMA IF NOT EXISTS schema152 AUTHORIZATION root",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=fc1d1195 key=/Table/SystemConfigSpan/Start pri=0.00509443 epo=0 ts=1633262270.759287899,1 min=1633262270.125367507,0 seq=15} lock=true stat=PENDING rts=1633262270.125367507,0 wto=false gul=1633262270.625367507,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:57:50.095822",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "CREATE SCHEMA IF NOT EXISTS schema207 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "***FAIL; Failed to receive an execution error when errors were expected"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ bd00c351494b44b31178350e79c72244c93ea224:

		  |   "BEGIN",
		  |   "CREATE TYPE public.enum58 AS ENUM ('fsaada', 'fa', 'dfaaa', 'fda', 'affd', 'fsf')",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:39:56.380378",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER TABLE public.table60 ALTER COLUMN \"IrrelevantColumnName\" DROP NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table60\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:39:56.381109",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-west1\"",
		  |   "ALTER TABLE public.table61 ALTER COLUMN IrrelevantColumnName SET NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table61\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:39:56.391862",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq20",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq62"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Validation FAIL:\nid 56, db schemachange, schema public, name seq20: relation \"seq20\" (56): invalid locality config: database schemachange is multi-region enabled, but table seq20 has no locality set"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 95c3da44a6c38aa1ae22fe9de5549632b7527aa5:

		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE public.enum133 AS ENUM ('asddaf', 'sasf', 'sfad')",
		  |   "ALTER TABLE public.table157 ALTER COLUMN IrrelevantColumnName DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table157\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:19:18.739307",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema25.seq156",
		  |   "ALTER TABLE schema158.table159 SET LOCALITY REGIONAL BY ROW"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema158.table159\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:19:18.741512",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE schema25.enum136 AS ENUM ('saf')",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=057b5b67 key=/Table/SystemConfigSpan/Start pri=0.00749929 epo=0 ts=1633436358.783863014,1 min=1633436358.741463884,0 seq=9} lock=true stat=PENDING rts=1633436358.741463884,0 wto=false gul=1633436359.241463884,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:19:18.59958",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA IF NOT EXISTS schema86 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\"",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "***FAIL; Failed to receive an execution error when errors were expected"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ f542fd2731127dab80599943b6041364fcc920e5:

		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE public.enum67 AS ENUM ('fd', 'dffd', 'dfs')",
		  |   "ALTER TABLE public.table38 ALTER COLUMN \"col38_53\" DROP STORED",
		  |   "DROP SEQUENCE public.seq54",
		  |   "CREATE INDEX \"IrrelevantName\" ON public.table104 (\"IrrelevantColumn\" ASC)"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table104\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:30:01.159904",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:30:01.10405",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP INDEX public.table90@\"IrrelevantIndexName\""
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table90\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:30:00.406373",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table38 ALTER COLUMN \"col38_41\" DROP DEFAULT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=1d7a505a key=/Table/SystemConfigSpan/Start pri=0.00387994 epo=0 ts=1633523401.618042944,1 min=1633523400.406317713,0 seq=1} lock=true stat=PENDING rts=1633523400.406317713,0 wto=false gul=1633523400.906317713,0 (SQLSTATE 40001)"
		  | }
		  |     2.0s        0           18.2           24.5      7.1    570.4    671.1    671.1 opOk
		  |     2.0s        0            8.1            4.5   1275.1   1543.5   1543.5   1543.5 txnCmtErr
		  |     2.0s        0            0.0            1.0      0.0      0.0      0.0      0.0 txnOk
		  |     2.0s        0           10.1           19.0     23.1    285.2    285.2    285.2 txnRbk
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 82e8782453b1cf14460d93b6bf8328a7b2964575:

		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=aa6024d4 key=/Table/SystemConfigSpan/Start pri=0.03057793 epo=0 ts=1633607190.949621491,2 min=1633607190.411337023,0 seq=32} lock=true stat=PENDING rts=1633607190.411337023,0 wto=false gul=1633607190.911337023,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:46:30.336363",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=c0000a3d key=/Table/SystemConfigSpan/Start pri=0.01499862 epo=0 ts=1633607191.267678248,1 min=1633607190.336312593,0 seq=8} lock=true stat=PENDING rts=1633607190.336312593,0 wto=false gul=1633607190.836312593,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:46:30.651781",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE VIEW public.view219 AS SELECT public.table180.col180_181, public.table180.col180_187, public.table180.col180_193, public.table180.col180_186, public.table180.col180_195, public.table180.col180_190, public.table180.col180_189, public.table180.col180_191, public.table180.col180_183, public.table180.col180_193, public.table180.col180_194, public.table180.col180_192, public.table180.col180_189, public.table197.col197_201, public.table197.col197_205, public.table197.col197_206, public.table197.col197_204, public.table197.col197_203, public.table197.crdb_internal_idx_expr_1, public.table197.col197_199 FROM public.table180, public.table180, public.table197"
		  |  ],
		  |  "expectedExecErrors": "42701,42712",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: source name \"table180\" specified more than once (missing AS clause) (SQLSTATE 42712)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:46:30.599537",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table180 ALTER COLUMN \"col180_194\" DROP NOT NULL",
		  |   "ALTER TABLE public.table180 ALTER COLUMN \"col180_217\" DROP DEFAULT"
		  |  ],
		  |  "expectedExecErrors": "42703",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col180_217\" does not exist (SQLSTATE 42703)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:46:30.738991",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table180 ALTER COLUMN col180_194 SET DEFAULT 219002036:::INT8"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: scanBool: \"\\n\\t\\tSELECT json_array_length(\\n        crdb_internal.pb_to_json(\\n            'cockroach.sql.sqlbase.Descriptor',\\n            descriptor\\n        )-\u003e'table'-\u003e'mutations'\\n       )\\n       \u003e 0\\n\\t\\tFROM system.descriptor\\n\\t  WHERE id = $1::REGCLASS\\n\\t\\t\" [\"public.table180\"]: can't scan into dest[0]: cannot assign NULL to *bool"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 4c94716dae27348c5e9092a621b45e76a5c8f4b0:

		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema3.table70 RENAME TO schema3.table78",
		  |   "DROP TABLE IF EXISTS schema3.table78 RESTRICT",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schemachange 52 1} {table70 72 0} {seq77 73 0}], old versions still in use (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:37.388917",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema3.table70 SET LOCALITY REGIONAL BY ROW"
		  |  ],
		  |  "expectedExecErrors": "42P16",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot alter a table's LOCALITY if its database is not multi-region enabled (SQLSTATE 42P16)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:35.518496",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP TABLE schema3.table70",
		  |   "DROP SCHEMA \"schema3\" CASCADE",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: cannot publish new versions for descriptors: [{schemachange 52 1} {schema3 53 0} {table70 72 0} {seq77 73 0}], old versions still in use (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:37.367412",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema3.table70 ALTER COLUMN col70_76 SET DEFAULT e'\\r7@':::STRING"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: relation \"table70\" (72): computed column \"col70_76\" cannot also have a DEFAULT expression (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 31f167ca5bbe404abcb215f80524770ddc8c0163:

		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:04:54.452295",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema28.seq212 OWNED BY schema28.table218.\"IrrelevantColumnName\""
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema28.table218\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:04:52.296811",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER SEQUENCE schema28.seq61 RENAME TO schema28.seq91",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:04:54.285787",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER VIEW schema28.view195 RENAME TO schema28.view196"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema28.view195\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:04:54.572212",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE): \"sql txn\" meta={id=a6cd2e43 key=/Table/SystemConfigSpan/Start pri=0.05551744 epo=0 ts=1652526295.000898150,1 min=1652526294.572134976,0 seq=19} lock=true stat=PENDING rts=1652526294.763345064,1 wto=false gul=1652526295.072134976,0 (SQLSTATE 40001)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ b4acf0d7aab35f5d3e1693c018a96756ef0bc7d6:

		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:19:13.974412",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table505 ALTER COLUMN IrrelevantColumnName DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table505\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:19:13.829897",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:19:14.10913",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema221.table517 DROP CONSTRAINT IrrelevantConstraintName"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema221.table517\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:19:14.198189",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SEQUENCE IF EXISTS public.seq451",
		  |   "CREATE TABLE schema221.table518 (col518_519 FLOAT4, col518_520 GEOMETRY NULL, col518_521 REGPROC NULL, col518_522 BYTES NULL, col518_523 TIME NULL, col518_524 GEOMETRY NULL, col518_525 TIMETZ, col518_526 NAME, col518_527 FLOAT4 NOT NULL, col518_528 STRING NULL AS (lower(CAST(col518_520 AS STRING))) STORED, col518_529 STRING NULL AS (CASE WHEN col518_522 IS NULL THEN e'$\\x1cx':::STRING ELSE '':::STRING END) VIRTUAL, INVERTED INDEX (col518_524 ASC), INDEX (col518_528, col518_521 DESC, (CASE WHEN col518_521 IS NULL THEN e'L\\x159\\x13%o':::STRING ELSE e',\\\\f4\\x07([ZJ':::STRING END), col518_526 ASC), INVERTED INDEX (col518_521, col518_525 ASC, col518_526 ASC, col518_528, col518_522 DESC, col518_527 DESC, col518_529, col518_520 ASC), INDEX (col518_525 DESC, col518_523 ASC, col518_526 ASC, col518_528 DESC) STORING (col518_519, col518_522, col518_524, col518_527), FAMILY (col518_528, col518_524, col518_520), FAMILY (col518_523), FAMILY (col518_521), FAMILY (col518_526), FAMILY (col518_525, col518_522), FAMILY (col518_527, col518_519))",
		  |   "ALTER TABLE schema221.table518 ALTER COLUMN \"col518_527\" SET NOT NULL",
		  |   "ALTER TABLE schema221.table518 ALTER COLUMN col518_528 SET DEFAULT e'Q\\f':::STRING"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: relation \"table518\" (182): computed column \"col518_528\" cannot also have a DEFAULT expression (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@postamar
Copy link
Contributor

@fqazi shouldn't this workload be disabled in release-21.2 ?

@fqazi
Copy link
Collaborator

fqazi commented May 17, 2022

Yes should be disabled via: e4a7903. Let me investigate

@fqazi
Copy link
Collaborator

fqazi commented May 17, 2022 •
edited
Loading

Okay, this wasn't properly disabled, only the bench variant was disabled 🤦

fqazi added a commit to fqazi/cockroach that referenced this issue May 17, 2022
@fqazi
workload/schemachange: disable schemachange workload
cd39f92 
Fixes: cockroachdb#70016

Previously, the schemachanger workload was enabled on 21.2
but failed due to a number of test issues. Our previous attempt
to disable this test got rid of the wrong configuration. To address,
this patch will disabled the schemachange/random-load test configuration.

Release note: None

Release note (<category, see below>): <what> <show> <why>
@fqazi fqazi mentioned this issue May 17, 2022
Merged
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 965686c845970621b3e21620774d31a84f5926ff:

		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:13:32.85657",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-east1\"",
		  |   "CREATE TABLE IF NOT EXISTS public.table179 (col179_180 BOOL, col179_181 BOOL NOT NULL, col179_182 VARCHAR NOT NULL, col179_183 BOOL, col179_184 STRING, col179_185 REGPROC NOT NULL, PRIMARY KEY (col179_182), col179_186 STRING AS (lower(CAST(col179_183 AS STRING))) STORED, INDEX (col179_184 ASC, col179_186 ASC, col179_183 ASC, col179_180 DESC, lower(col179_184) ASC) STORING (col179_185), UNIQUE (col179_183 DESC, col179_184 ASC, col179_180, col179_186 ASC), UNIQUE (col179_186, lower(col179_182) DESC), UNIQUE (col179_181 DESC), INDEX (col179_182 DESC, col179_186 DESC, col179_183 DESC, col179_181 DESC), INDEX (col179_185 DESC, col179_184 ASC, col179_183 ASC) STORING (col179_180, col179_181, col179_186), FAMILY (col179_185, col179_183, col179_180), FAMILY (col179_182, col179_181, col179_184), FAMILY (col179_186))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=65e280c6 key=/Table/SystemConfigSpan/Start pri=0.03037797 epo=0 ts=1652872414.034931812,1 min=1652872412.856475540,0 seq=33} lock=true stat=PENDING rts=1652872412.856475540,0 wto=false gul=1652872413.356475540,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:13:34.075054",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange ADD REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "42P12",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot add region \"us-west1\" to database schemachange (SQLSTATE 42P12)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:13:33.369559",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table91 RENAME TO public.table151",
		  |   "ALTER TABLE public.table151 ALTER COLUMN \"col91_99\" DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42611",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col91_99\" is not a stored computed column (SQLSTATE 42611)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:13:32.94781",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table91 ADD COLUMN col91_170 DATE NULL"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Validation FAIL:\nid 86, db schemachange, schema public, name table91: mutation job 762845639100366849: job not found"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ ed2a24d64a01fa0af27433dd57b2433e2ff407a1:

		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schemachange.public.table38\" already exists (SQLSTATE 42P07)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:21:45.893755",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE VIEW public.view70 AS SELECT \"IrrelevantColumnName\", public.table38.col38_40, public.table38.col38_39, \"IrrelevantColumnName\" FROM public.table45, public.table38, public.table59"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table45\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:21:45.902374",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE public.table71 AS SELECT public.table38.col38_40, public.table38.col38_39 FROM public.table38, public.table38"
		  |  ],
		  |  "expectedExecErrors": "42712",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: source name \"table38\" specified more than once (missing AS clause) (SQLSTATE 42712)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:21:46.147597",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table38 ALTER COLUMN \"col38_61\" DROP NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "42703",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col38_61\" does not exist (SQLSTATE 42703)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:21:46.235885",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq53",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh): \"sql txn\" meta={id=6cda13a0 key=/Table/SystemConfigSpan/Start pri=0.01166188 epo=0 ts=1652959306.315943446,2 min=1652959306.235812385,0 seq=8} lock=true stat=PENDING rts=1652959306.235812385,0 wto=false gul=1652959306.735812385,0 (SQLSTATE 40001)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ bdde4c6820454c38c2bd31dc27c1381246c4d7fe:

		  |  "clientTimestamp": "11:01:01.77777",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER INDEX public.table66@\"index66_82\" RENAME TO \"index66_83\""
		  |  ],
		  |  "expectedExecErrors": "42704",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: index \"index66_82\" does not exist (SQLSTATE 42704)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:01:01.919127",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP TABLE public.table84 CASCADE"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table84\" does not exist (SQLSTATE 42P01)"
		  | }
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |     1.0s        0           32.2           35.0      8.9    260.0    352.3    352.3 opOk
		  |     1.0s        0            0.9            1.0    268.4    268.4    268.4    268.4 txnCmtErr
		  |     1.0s        0            7.4            8.0    151.0    604.0    604.0    604.0 txnOk
		  |     1.0s        0           22.1           24.0      4.7     50.3    402.7    402.7 txnRbk
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:01:01.899826",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq61",
		  |   "ALTER TABLE public.table66 ALTER COLUMN \"col66_75\" DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42611",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col66_75\" is not a computed column (SQLSTATE 42611)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:01:01.762372",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: scanBool: \"\\n\\t\\tSELECT json_array_length(\\n        crdb_internal.pb_to_json(\\n            'cockroach.sql.sqlbase.Descriptor',\\n            descriptor\\n        )-\u003e'table'-\u003e'mutations'\\n       )\\n       \u003e 0\\n\\t\\tFROM system.descriptor\\n\\t  WHERE id = $1::REGCLASS\\n\\t\\t\" [\"public.table66\"]: can't scan into dest[0]: cannot assign NULL to *bool"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 2bd5952ed0db3dc898c0d75eb50bc65c26feef04:

		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | _233 GEOMETRY, col226_234 BOX2D, col226_235 DECIMAL, col226_236 FLOAT4 NULL, col226_237 OID NOT NULL, col226_238 REGCLASS, col226_239 REGPROC, col226_240 NAME NULL, col226_241 REGTYPE, col226_242 DATE, col226_243 INT2, col226_244 UUID NULL, col226_245 FLOAT4 NULL AS (col226_232 + col226_236) VIRTUAL, UNIQUE (col226_237 ASC, col226_242 DESC, col226_230 ASC, col226_239, col226_245), UNIQUE (col226_228, col226_240) STORING (col226_229, col226_232, col226_233, col226_234, col226_235, col226_236, col226_241, col226_242, col226_244))",
		  |   "CREATE TYPE public.enum247 AS ENUM ('fdds')",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:12:52.446192",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=3e3991d6 key=/Table/SystemConfigSpan/Start pri=0.02425951 epo=0 ts=1653131573.203129718,1 min=1653131572.446107357,0 seq=8} lock=true stat=PENDING rts=1653131572.446107357,0 wto=false gul=1653131572.946107357,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:12:52.891485",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE IF NOT EXISTS public.table197 (col197_198 TIME, col197_199 REGPROCEDURE NOT NULL, col197_200 BIT(29) NOT NULL, col197_201 TIME NOT NULL, col197_202 REGNAMESPACE NULL, col197_203 STRING NULL, col197_204 UUID NOT NULL, col197_205 DECIMAL NOT NULL, col197_206 REGNAMESPACE NOT NULL, col197_207 TIMETZ NULL, col197_208 DATE NOT NULL, col197_209 INT4 NOT NULL, col197_210 REGPROCEDURE NOT NULL, col197_211 TIMESTAMP NOT NULL, col197_212 STRING, col197_213 DATE, col197_214 NAME NOT NULL, col197_215 \"char\" NULL, PRIMARY KEY (col197_214), UNIQUE (col197_205 ASC, col197_214 ASC, col197_209, col197_204, (CASE WHEN col197_199 IS NULL THEN e'\\x02!\\x0e\u003c0\\x0b':::STRING ELSE e'6?^F@\\x123\\x7fd':::STRING END) DESC, col197_203 DESC, col197_210 ASC, col197_213 DESC, lower(CAST(col197_207 AS STRING)), col197_201 DESC, col197_202, col197_199, col197_211 ASC, col197_208 DESC, col197_200 ASC), INDEX (col197_198 DESC, col197_203 ASC, col197_199 DESC, (CASE WHEN col197_213 IS NULL THEN '':::STRING ELSE e'8]\\x03b\\x07E3\\f':::STRING END) DESC, col197_209 ASC, (CASE WHEN col197_199 IS NULL THEN 'o!3':::STRING ELSE e'\\x18X\\x0b6\\x16FB':::STRING END) DESC, col197_215 ASC), UNIQUE (col197_200 ASC, col197_204 ASC, col197_215 ASC), INDEX (col197_198 DESC, col197_203, col197_208 DESC, col197_209, (CASE WHEN col197_208 IS NULL THEN 'KG':::STRING ELSE ';a1x':::STRING END), (true) ASC, col197_200 DESC, lower(col197_214), col197_210 DESC, col197_211 DESC, col197_212 ASC))",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): \"sql txn\" meta={id=879652e8 key=/Table/SystemConfigSpan/Start pri=0.00599991 epo=0 ts=1653131572.984463743,1 min=1653131572.891413489,0 seq=15} lock=true stat=ABORTED rts=1653131572.891413489,0 wto=false gul=1653131573.391413489,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:12:52.949085",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema219.seq246",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  |     3.0s        0           25.0           33.0      8.1    671.1    906.0    906.0 opOk
		  |     3.0s        0            3.0            3.7    419.4    872.4    872.4    872.4 txnCmtErr
		  |     3.0s        0            3.0            2.3    369.1   1946.2   1946.2   1946.2 txnOk
		  |     3.0s        0           29.0           27.3      9.4    838.9   1040.2   1040.2 txnRbk
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ ef393bba2596c5b680a8fd07ba6fa34cca47ce4e:

		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:06:31.494778",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq65",
		  |   "ALTER TABLE public.table5 RENAME COLUMN \"col5_8\" TO \"col5_87\"",
		  |   "CREATE SEQUENCE IF NOT EXISTS public.seq108",
		  |   "CREATE INDEX \"IrrelevantName\" ON schema109.table110 (\"IrrelevantColumn\" ASC)"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"schema109.table110\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:06:32.051086",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE public.table73 AS SELECT public.table5.col5_11, public.table5.col5_6 FROM public.table5",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:06:32.471867",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table75 ALTER COLUMN IrrelevantColumnName DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42P01",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: relation \"public.table75\" does not exist (SQLSTATE 42P01)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:06:32.290996",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table5 ADD COLUMN col5_83 public.enum58 NULL"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Validation FAIL:\nid 54, db schemachange, schema public, name table5: mutation job 763976721305632769: job not found"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 4e40ba1736e13125dabd8d78211df7e3ad9b610e:

		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:57.543182",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema351 AUTHORIZATION root",
		  |   "ALTER DATABASE schemachange SURVIVE ZONE FAILURE"
		  |  ],
		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:57.546998",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE schema3.enum352 AS ENUM ('sfd', 's', 'da')",
		  |   "ALTER TABLE schema3.table325 RENAME COLUMN \"col325_340\" TO \"col325_375\"",
		  |   "ALTER TABLE schema3.table325 SET LOCALITY REGIONAL BY ROW"
		  |  ],
		  |  "expectedExecErrors": "42P16",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot alter a table's LOCALITY if its database is not multi-region enabled (SQLSTATE 42P16)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:57.833409",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SCHEMA \"public\" CASCADE"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:58.312905",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SCHEMA schema394 AUTHORIZATION root",
		  |   "ALTER TABLE schema3.table325 ALTER COLUMN col325_344 SET DEFAULT e'y\\x02\\x14\\x0f.{(X':::STRING"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: relation \"table325\" (152): computed column \"col325_344\" cannot also have a DEFAULT expression (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ aaaa2f5eb8df283ad9e3f4e60391614b0d9046ee:

		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=a46cdb4a key=/Table/SystemConfigSpan/Start pri=0.00726967 epo=0 ts=1653391067.612112978,2 min=1653391067.305825225,0 seq=1} lock=true stat=PENDING rts=1653391067.305825225,0 wto=false gul=1653391067.805825225,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:47.879143",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": ""
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:47.292776",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table72 ALTER COLUMN \"col72_79\" DROP NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=e16f8697 key=/Table/SystemConfigSpan/Start pri=0.01958800 epo=0 ts=1653391068.259353487,1 min=1653391067.292685810,0 seq=1} lock=true stat=PENDING rts=1653391067.292685810,0 wto=false gul=1653391067.792685810,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:47.370119",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table72 ALTER COLUMN \"col72_74\" SET NOT NULL"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=3eb4469c key=/Table/SystemConfigSpan/Start pri=0.02332609 epo=0 ts=1653391068.263008541,1 min=1653391067.370030245,0 seq=1} lock=true stat=PENDING rts=1653391067.370030245,0 wto=false gul=1653391067.870030245,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:17:47.940099",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table36 ALTER COLUMN col36_55 SET DEFAULT e'\\x7f\\n(\\\\\",eyA':::STRING"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error: ERROR: relation \"table36\" (70): computed column \"col36_55\" cannot also have a DEFAULT expression (SQLSTATE 42P16)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ fba8f834d787f09906f4e5e7d10eb0ead47dd1a4:

		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:15:43.034759",
		  |  "ops": [
		  |   "BEGIN",
		  |   "DROP SCHEMA \"schema1\" CASCADE",
		  |   "ALTER TABLE public.table48 DROP CONSTRAINT \"primary\"",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "0A000",
		  |  "message": "COMMIT; Successfully got expected commit error"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:15:44.973188",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange SURVIVE REGION FAILURE"
		  |  ],
		  |  "expectedExecErrors": "22023,42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:15:42.968817",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER SEQUENCE public.seq154 RENAME TO public.seq256",
		  |   "ALTER TABLE public.table48 ADD COLUMN col48_261 FLOAT8",
		  |   "ALTER TABLE schema45.table135 SET LOCALITY REGIONAL BY ROW"
		  |  ],
		  |  "expectedExecErrors": "42P16",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: cannot alter a table's LOCALITY if its database is not multi-region enabled (SQLSTATE 42P16)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:15:41.044554",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE schema45.table135 ADD COLUMN col135_236 schema1.enum47"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Validation FAIL:\nid 94, db schemachange, schema schema45, name table135: mutation job 764827870469750785: job not found"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on release-21.2 @ 96a2f03ef664468cf22ebe11624a5562ed2d3913:

		  |  "expectedExecErrors": "42602",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: database must have associated regions before a survival goal can be set (SQLSTATE 42602)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:53.955876",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table296 ALTER COLUMN \"col296_311\" DROP STORED"
		  |  ],
		  |  "expectedExecErrors": "42611",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error: ERROR: column \"col296_311\" is not a computed column (SQLSTATE 42611)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:53.656933",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table296 RENAME TO public.table566"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=ed722c52 key=/Table/SystemConfigSpan/Start pri=0.03466196 epo=0 ts=1653565014.316233798,1 min=1653565013.656879715,0 seq=1} lock=true stat=PENDING rts=1653565013.656879715,0 wto=false gul=1653565014.156879715,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:53.451429",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table329 ALTER COLUMN \"col329_534\" DROP DEFAULT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=2ccd04f4 key=/Table/SystemConfigSpan/Start pri=0.04957858 epo=0 ts=1653565014.319082161,1 min=1653565013.451314343,0 seq=1} lock=true stat=PENDING rts=1653565013.451314343,0 wto=false gul=1653565013.951314343,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "11:36:53.776839",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER TABLE public.table296 ADD COLUMN col296_567 REGCLASS"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=cff0b8bc key=/Table/SystemConfigSpan/Start pri=0.03198208 epo=0 ts=1653565014.316233798,1 min=1653565013.776760034,0 seq=1} lock=true stat=PENDING rts=1653565013.776760034,0 wto=false gul=1653565014.276760034,0 (SQLSTATE 40001)"
		  | }
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError
Reproduce

See: roachtest README

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

@ajwerner
Copy link
Contributor

@fqazi should this still be running?

@fqazi
Copy link
Collaborator

fqazi commented May 26, 2022

PR wasn't merged will get it in today

fqazi added a commit to fqazi/cockroach that referenced this issue May 26, 2022
@fqazi
workload/schemachange: disable schemachange workload
15751eb 
Fixes: cockroachdb#70016

Previously, the schemachanger workload was enabled on 21.2
but failed due to a number of test issues. Our previous attempt
to disable this test got rid of the wrong configuration. To address,
this patch will disabled the schemachange/random-load test configuration.

Release note: None
craig bot pushed a commit that referenced this issue May 26, 2022
@fqazi
Merge #81400
9662f9a 
81400: workload/schemachange: disable schemachange workload r=fqazi a=fqazi

Fixes: #70016

Previously, the schemachanger workload was enabled on 21.2
but failed due to a number of test issues. Our previous attempt
to disable this test got rid of the wrong configuration. To address,
this patch will disabled the schemachange/random-load test configuration.

Release note: None

Release note (<category, see below>): <what> <show> <why>

Co-authored-by: Faizan Qazi <[email protected]>
@jlinder jlinder removed the sync-me-3 label May 27, 2022
@cockroach-teamcity cockroach-teamcity mentioned this issue May 31, 2022
Closed
@fqazi fqazi closed this as completed Jun 1, 2022
@healthy-pod healthy-pod added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workload/schemachange: disable schema change mixed versions workload fqazi/cockroach
6 participants
@jlinder @fqazi @ajwerner @postamar @cockroach-teamcity @healthy-pod