roachtest: schemachange/random-load failed #80699

cockroach-teamcity · 2022-04-28T12:17:16Z

roachtest.schemachange/random-load failed with artifacts on master @ 72e485705aa859ef850ebc67568bc29fe9b7cfd6:

		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"us-west1\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=d9726548 key=/Table/3/1/107/2/1 pri=0.02280027 epo=0 ts=1651148189.802201007,2 min=1651148189.270156485,0 seq=9} lock=true stat=PENDING rts=1651148189.270156485,0 wto=false gul=1651148189.770156485,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:16:29.219789",
		  |  "ops": [
		  |   "BEGIN",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_WRITE_TOO_OLD - WriteTooOld flag converted to WriteTooOldError): \"sql txn\" meta={id=279c91e2 key=/Table/3/1/107/2/1 pri=0.01537437 epo=0 ts=1651148189.938497018,1 min=1651148189.219725199,0 seq=9} lock=true stat=PENDING rts=1651148189.219725199,0 wto=false gul=1651148189.719725199,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:16:29.157272",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE public.table64 (col64_67 JSONB NOT NULL, col64_68 STRING, col64_69 GEOMETRY, col64_70 STRING NOT NULL, col64_71 VARCHAR NULL, col64_72 TIMESTAMP NOT NULL, col64_73 FLOAT8 NOT NULL, col64_74 DECIMAL NULL, col64_75 DECIMAL NULL, col64_76 BIT(25), col64_77 STRING NOT NULL AS (lower(col64_70)) VIRTUAL, col64_78 STRING AS (lower(col64_68)) VIRTUAL, col64_79 STRING AS (lower(CAST(col64_76 AS STRING))) VIRTUAL, col64_80 DECIMAL NULL AS (col64_75 + col64_74) STORED, col64_81 DECIMAL NULL AS (col64_74 + col64_75) VIRTUAL, col64_82 STRING AS (lower(CAST(col64_69 AS STRING))) STORED, UNIQUE (col64_74, col64_68 DESC, col64_80, col64_73 ASC, col64_79 ASC, col64_77 ASC, col64_70 DESC, col64_76 ASC, col64_72 ASC, col64_71 ASC, col64_82 DESC, col64_75) STORING (col64_69), UNIQUE (col64_76, col64_81 DESC, col64_75 ASC, col64_82, col64_73 DESC), INDEX (col64_71, col64_70 ASC) STORING (col64_67, col64_68, col64_80), INDEX (col64_82), FAMILY (col64_71, col64_82), FAMILY (col64_73, col64_72, col64_76, col64_74), FAMILY (col64_75), FAMILY (col64_67), FAMILY (col64_68, col64_69), FAMILY (col64_80), FAMILY (col64_70))",
		  |   "ALTER SEQUENCE public.seq6 RENAME TO public.seq90",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/120/2/1): \"sql txn\" meta={id=e4386e39 key=/NamespaceTable/30/1/104/105/\"table64\"/4/1 pri=0.02364945 epo=0 ts=1651148189.373625030,1 min=1651148189.157176251,0 seq=22} lock=true stat=PENDING rts=1651148189.157176251,0 wto=false gul=1651148189.657176251,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "12:16:29.491936",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [SELECT region FROM [SHOW REGIONS FROM CLUSTER]] : us-east1,us-west1,europe-west2,\nQUERY [SELECT region FROM [SHOW REGIONS FROM DATABASE]] : \nQUERY [SHOW DATABASE] :schemachange\n===========================Previous statements [ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"]: ERROR: cannot convert database schemachange to a multi-region database (SQLSTATE 55000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: schemachange/random-load failed #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/random-load failed #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

/cc @cockroachdb/sql-schema _{This test on roachdash | Improve this report!

Jira issue: CRDB-15545}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2022-04-30T12:31:01Z

roachtest.schemachange/random-load failed with artifacts on master @ a2e1910f51593bd2ef72e1d7c615e08f95791186:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:163,schemachange_random_load.go:58,test_runner.go:876: output in run_123015.473613413_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1962
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:160
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:876
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_123015.473613413_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | relevantColumnName SET DEFAULT "IrrelevantValue"]
		  | I220430 12:30:20.836146 238 workload/pgx_helpers.go:79  [-] 98  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot drop schema "public" (SQLSTATE 3F000) pid:1438949 sql:DROP SCHEMA "public" CASCADE]
		  | I220430 12:30:20.873410 236 workload/pgx_helpers.go:79  [-] 99  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot convert database schemachange to a multi-region database (SQLSTATE 55000) pid:1312293 sql:ALTER DATABASE schemachange PRIMARY REGION "europe-west2"]
		  | I220430 12:30:20.875760 230 workload/pgx_helpers.go:79  [-] 100  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "us-west1" to database schemachange (SQLSTATE 42P12) pid:1454306 sql:ALTER DATABASE schemachange ADD REGION "us-west1"]
		  | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
		  | Expected errors: ===========================Executed queries for generating errors: QUERY [SELECT region FROM [SHOW REGIONS FROM CLUSTER]] : us-west1,europe-west2,us-east1,
		  | QUERY [SELECT region FROM [SHOW REGIONS FROM DATABASE]] : 
		  | QUERY [SHOW DATABASE] :schemachange
		  | ===========================Previous statements [CREATE TABLE schema124.table357 (col357_358 TIMESTAMPTZ NOT NULL, col357_359 REGTYPE NOT NULL, col357_360 TIMESTAMP NOT NULL, col357_361 DATE NOT NULL, col357_362 TIMETZ NOT NULL, col357_363 DATE NOT NULL, col357_364 DECIMAL NOT NULL, col357_365 NAME NOT NULL, col357_366 REGROLE NOT NULL, col357_367 TIMESTAMPTZ NULL, PRIMARY KEY (col357_358 ASC, col357_364 DESC, col357_360 ASC, col357_365 ASC, col357_361 ASC, col357_359 DESC, col357_362 DESC), INDEX (col357_359 ASC) STORING (col357_366), INDEX (col357_364, col357_366 ASC, col357_363 ASC, col357_360 DESC, col357_358 ASC, col357_362 ASC, col357_365) PARTITION BY LIST (col357_364, col357_366, col357_363, col357_360, col357_358) (PARTITION table357_part_0 VALUES IN ((4.936713840530872015E+37:::DECIMAL, 2311029775:::OID, '1997-04-15':::DATE, '1974-01-04 01:19:00.000344':::TIMESTAMP, '2003-02-01 18:51:18.000288+00:00':::TIMESTAMPTZ), (6.454777933923291516E+33:::DECIMAL, 143014662:::OID, '1971-05-05':::DATE, '3000-01-01 00:00:00':::TIMESTAMP, '1982-02-13 05:39:21.000138+00:00':::TIMESTAMPTZ)), PARTITION table357_part_1 VALUES IN (((-1.932724788278843973E+27):::DECIMAL, 1936914651:::OID, '1992-01-21':::DATE, '2012-02-29 07:49:29.000666':::TIMESTAMP, '-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ), ((-4.018893389376167857E+34):::DECIMAL, 95560620:::OID, '1990-08-01':::DATE, '2027-05-31 08:25:45.00076':::TIMESTAMP, '1971-12-28 12:41:54.000413+00:00':::TIMESTAMPTZ)), PARTITION table357_part_2 VALUES IN ((364526120709.3253223:::DECIMAL, 2679756020:::OID, '1970-01-01':::DATE, '2001-09-18 23:36:59.000265':::TIMESTAMP, '2011-01-11 11:54:04.000283+00:00':::TIMESTAMPTZ), (2.438527359832560807E+26:::DECIMAL, 2161887562:::OID, '1981-09-04':::DATE, '2032-07-24 17:46:28.000341':::TIMESTAMP, '1998-04-01 09:47:55.000523+00:00':::TIMESTAMPTZ)), PARTITION table357_part_3 VALUES IN ((4.143155374805245801E+25:::DECIMAL, 3413744868:::OID, '5874897-12-31':::DATE, '1980-01-11 15:17:41.000315':::TIMESTAMP, '2024-10-14 22:27:01.000651+00:00':::TIMESTAMPTZ), ((-7.473888515062706787E+24):::DECIMAL, 0:::OID, '1984-09-19':::DATE, '2021-02-27 19:06:08.000167':::TIMESTAMP, '2014-03-04 07:56:32.00023+00:00':::TIMESTAMPTZ)), PARTITION "DEFAULT" VALUES IN ((DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT))), INDEX (col357_360 ASC, col357_359, col357_365 DESC, col357_364 DESC, col357_367, col357_366 ASC, col357_362, col357_358 ASC, col357_361), UNIQUE (col357_366, col357_358 DESC, col357_360 DESC, col357_365 DESC, col357_364 DESC, col357_363 DESC, col357_359 ASC, col357_367 ASC, col357_362), INDEX (col357_360 ASC, col357_365, col357_358 DESC, col357_361 DESC, col357_364, col357_366 ASC, col357_362, col357_363), INDEX (col357_366, col357_361 DESC, col357_363 ASC) STORING (col357_367)) ALTER DATABASE schemachange PRIMARY REGION "europe-west2"]: ERROR: cannot convert database schemachange to a multi-region database (SQLSTATE 55000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | '2021-02-27 19:06:08.000167':::TIMESTAMP, '2014-03-04 07:56:32.00023+00:00':::TIMESTAMPTZ)), PARTITION \"DEFAULT\" VALUES IN ((DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT))), INDEX (col357_360 ASC, col357_359, col357_365 DESC, col357_364 DESC, col357_367, col357_366 ASC, col357_362, col357_358 ASC, col357_361), UNIQUE (col357_366, col357_358 DESC, col357_360 DESC, col357_365 DESC, col357_364 DESC, col357_363 DESC, col357_359 ASC, col357_367 ASC, col357_362), INDEX (col357_360 ASC, col357_365, col357_358 DESC, col357_361 DESC, col357_364, col357_366 ASC, col357_362, col357_363), INDEX (col357_366, col357_361 DESC, col357_363 ASC) STORING (col357_367))",
		  |   "ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\""
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: ===========================Executed queries for generating errors: QUERY [SELECT region FROM [SHOW REGIONS FROM CLUSTER]] : us-west1,europe-west2,us-east1,\nQUERY [SELECT region FROM [SHOW REGIONS FROM DATABASE]] : \nQUERY [SHOW DATABASE] :schemachange\n===========================Previous statements [CREATE TABLE schema124.table357 (col357_358 TIMESTAMPTZ NOT NULL, col357_359 REGTYPE NOT NULL, col357_360 TIMESTAMP NOT NULL, col357_361 DATE NOT NULL, col357_362 TIMETZ NOT NULL, col357_363 DATE NOT NULL, col357_364 DECIMAL NOT NULL, col357_365 NAME NOT NULL, col357_366 REGROLE NOT NULL, col357_367 TIMESTAMPTZ NULL, PRIMARY KEY (col357_358 ASC, col357_364 DESC, col357_360 ASC, col357_365 ASC, col357_361 ASC, col357_359 DESC, col357_362 DESC), INDEX (col357_359 ASC) STORING (col357_366), INDEX (col357_364, col357_366 ASC, col357_363 ASC, col357_360 DESC, col357_358 ASC, col357_362 ASC, col357_365) PARTITION BY LIST (col357_364, col357_366, col357_363, col357_360, col357_358) (PARTITION table357_part_0 VALUES IN ((4.936713840530872015E+37:::DECIMAL, 2311029775:::OID, '1997-04-15':::DATE, '1974-01-04 01:19:00.000344':::TIMESTAMP, '2003-02-01 18:51:18.000288+00:00':::TIMESTAMPTZ), (6.454777933923291516E+33:::DECIMAL, 143014662:::OID, '1971-05-05':::DATE, '3000-01-01 00:00:00':::TIMESTAMP, '1982-02-13 05:39:21.000138+00:00':::TIMESTAMPTZ)), PARTITION table357_part_1 VALUES IN (((-1.932724788278843973E+27):::DECIMAL, 1936914651:::OID, '1992-01-21':::DATE, '2012-02-29 07:49:29.000666':::TIMESTAMP, '-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ), ((-4.018893389376167857E+34):::DECIMAL, 95560620:::OID, '1990-08-01':::DATE, '2027-05-31 08:25:45.00076':::TIMESTAMP, '1971-12-28 12:41:54.000413+00:00':::TIMESTAMPTZ)), PARTITION table357_part_2 VALUES IN ((364526120709.3253223:::DECIMAL, 2679756020:::OID, '1970-01-01':::DATE, '2001-09-18 23:36:59.000265':::TIMESTAMP, '2011-01-11 11:54:04.000283+00:00':::TIMESTAMPTZ), (2.438527359832560807E+26:::DECIMAL, 2161887562:::OID, '1981-09-04':::DATE, '2032-07-24 17:46:28.000341':::TIMESTAMP, '1998-04-01 09:47:55.000523+00:00':::TIMESTAMPTZ)), PARTITION table357_part_3 VALUES IN ((4.143155374805245801E+25:::DECIMAL, 3413744868:::OID, '5874897-12-31':::DATE, '1980-01-11 15:17:41.000315':::TIMESTAMP, '2024-10-14 22:27:01.000651+00:00':::TIMESTAMPTZ), ((-7.473888515062706787E+24):::DECIMAL, 0:::OID, '1984-09-19':::DATE, '2021-02-27 19:06:08.000167':::TIMESTAMP, '2014-03-04 07:56:32.00023+00:00':::TIMESTAMPTZ)), PARTITION \"DEFAULT\" VALUES IN ((DEFAULT, DEFAULT, DEFAULT, DEFAULT, DEFAULT))), INDEX (col357_360 ASC, col357_359, col357_365 DESC, col357_364 DESC, col357_367, col357_366 ASC, col357_362, col357_358 ASC, col357_361), UNIQUE (col357_366, col357_358 DESC, col357_360 DESC, col357_365 DESC, col357_364 DESC, col357_363 DESC, col357_359 ASC, col357_367 ASC, col357_362), INDEX (col357_360 ASC, col357_365, col357_358 DESC, col357_361 DESC, col357_364, col357_366 ASC, col357_362, col357_363), INDEX (col357_366, col357_361 DESC, col357_363 ASC) STORING (col357_367)) ALTER DATABASE schemachange PRIMARY REGION \"europe-west2\"]: ERROR: cannot convert database schemachange to a multi-region database (SQLSTATE 55000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

roachtest: schemachange/random-load failed #70016 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.2]
roachtest: schemachange/random-load failed #61698 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-release-21.1]

_{This test on roachdash | Improve this report!}

Fixes: cockroachdb#80699 Previously, the schema changer would attempt to convert a database to multi-region without confirm if any of the tables were already partitioned. This could lead to "cannot convert database .. to multiregion" errors. To address this, this patch add supports for detecting when the conversion will fail. Release note: None

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Apr 28, 2022

blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label Apr 28, 2022

craig bot closed this as completed in 322c2b3 May 3, 2022

jlinder added the sync-me label May 20, 2022

healthy-pod added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: schemachange/random-load failed #80699

roachtest: schemachange/random-load failed #80699

cockroach-teamcity commented Apr 28, 2022 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Apr 30, 2022

roachtest: schemachange/random-load failed #80699

roachtest: schemachange/random-load failed #80699

Comments

cockroach-teamcity commented Apr 28, 2022 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Apr 30, 2022

cockroach-teamcity commented Apr 28, 2022 •

edited by cockroach-jira-scripts

Loading