Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: schemachange/random-load failed #82133

Closed
cockroach-teamcity opened this issue May 31, 2022 · 65 comments · Fixed by #88537
Closed

roachtest: schemachange/random-load failed #82133

cockroach-teamcity opened this issue May 31, 2022 · 65 comments · Fixed by #88537
Assignees
@fqazi
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. sync-me-8 T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 31, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.schemachange/random-load failed with artifacts on master @ 662cc5c3070e6d64d155a9cc9f33253ee5d99ee9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_124321.258348568_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1947
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_124321.258348568_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 09FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);" []] :false
		  | QUERY ["SELECT (((col8964_8969, col8964_8968))::STRING) IS NULL AS c FROM ( VALUES('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);" []] :false
		  | QUERY ["SELECT (((col8964_8969))::STRING) IS NULL AS c FROM ( VALUES(NULL,'0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY)) AS t(col8964_8969,col8964_8968);" []] :true
		  | QUERY ["\n    WITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema4785.table8964"]] :[["(col8964_8969)"] ["(CASE WHEN col8964_8969 IS NULL THEN e'\\'':::STRING ELSE e'\\x1e!#':::STRING END)"] ["(col8964_8969)"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema4785.table8964'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema4785'\n\t\t           AND table_name = 'table8964'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
		  | ===========================Previous statements [INSERT INTO schema4785.table8964 (col8964_8968,col8964_8969) VALUES ('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)]: ERROR: duplicate key value violates unique constraint "table8964_expr_key" (SQLSTATE 23505)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 2C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)) AS t(col8964_8968,col8964_8969);\" []] :false\nQUERY [\"SELECT (((col8964_8969))::STRING) IS NULL AS c FROM ( VALUES(NULL,'0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY)) AS t(col8964_8969,col8964_8968);\" []] :true\nQUERY [\"\\n    WITH tab_json AS (\\n                    SELECT crdb_internal.pb_to_json(\\n                            'desc',\\n                            descriptor\\n                           )-\u003e'table' AS t\\n                      FROM system.descriptor\\n                     WHERE id = $1::REGCLASS\\n                  ),\\n         columns_json AS (\\n                        SELECT json_array_elements(t-\u003e'columns') AS c FROM tab_json\\n                      ),\\n         columns AS (\\n                    SELECT (c-\u003e\u003e'id')::INT8 AS col_id,\\n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema4785.table8964\"]] :[[\"(col8964_8969)\"] [\"(CASE WHEN col8964_8969 IS NULL THEN e'\\\\'':::STRING ELSE e'\\\\x1e!#':::STRING END)\"] [\"(col8964_8969)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema4785.table8964'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema4785'\\n\\t\\t           AND table_name = 'table8964'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema4785.table8964 (col8964_8968,col8964_8969) VALUES ('0101000060E61000008C88F909FFDE52C00C6834256F555440200303D07CBF0142':::GEOGRAPHY,NULL)]: ERROR: duplicate key value violates unique constraint \"table8964_expr_key\" (SQLSTATE 23505)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-16222
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 31, 2022
@blathers-crl blathers-crl bot added the T-sql-schema-deprecated Use T-sql-foundations instead label May 31, 2022
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 2c08debd6d8c89019f26a3e58cf3b5d3d97b6495:

		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_131707.590160241_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | ema5510.seq5511" because the target database or schema does not exist (SQLSTATE 3F000) pid:1700748 sql:CREATE SEQUENCE IF NOT EXISTS schema5510.seq5511 time:474.303µs]
		  | I220606 13:34:50.243786 129 workload/pgx_helpers.go:79  [-] 1344  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-west1" already added to database (SQLSTATE 42710) pid:1291274 sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:10.037632ms]
		  | I220606 13:34:50.274010 212 workload/pgx_helpers.go:79  [-] 1345  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema5513.table5547" does not exist (SQLSTATE 42P01) pid:1393661 sql:ALTER TABLE schema5513.table5547 DROP CONSTRAINT IrrelevantConstraintName time:88.671403ms]
		  | I220606 13:34:50.285571 211 workload/pgx_helpers.go:79  [-] 1346  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema1129.table5506" does not exist (SQLSTATE 42P01) pid:1781910 sql:CREATE TABLE schema4074.table5549 AS SELECT "IrrelevantColumnName" FROM schema1129.table5506 time:2.147877ms]
		  | I220606 13:34:50.778757 221 workload/pgx_helpers.go:79  [-] 1347  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot create "schema5555.table5556" because the target database or schema does not exist (SQLSTATE 3F000) pid:1694075 sql:CREATE TABLE schema5555.table5556 AS SELECT public.table907.col907_911, public.table907.col907_916, public.table907.col907_918, public.table907.col907_917, public.table907.col907_910, public.table907.col907_908, public.table907.col907_915, public.table907.col907_909, public.table907.col907_919, public.table907.col907_914, schema3441.table4899.col907_910 FROM public.table907, schema3441.table4899 time:826.696µs]
		  | I220606 13:34:51.287192 228 workload/pgx_helpers.go:79  [-] 1348  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schemachange.schema4074.view5371" already exists (SQLSTATE 42P07) pid:1091750 sql:CREATE VIEW schema4074.view5371 AS SELECT schema3224.table5225.col5225_5235, schema3224.table5225.col5225_5229, schema3224.table5225.col5225_5231, schema3224.table5225.col5225_5238, schema3224.table5225.col5225_5228, schema3224.table5225.col5225_5239, schema3224.table5225.col5225_5243 FROM schema3224.table5225 time:3.018861ms]
		  | I220606 13:34:51.492491 225 workload/pgx_helpers.go:79  [-] 1349  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot change schema of table with RENAME (SQLSTATE 42602) pid:1191175 sql:ALTER SEQUENCE schema5048.seq5471 RENAME TO public.seq5561 time:60.185157ms]
		  | I220606 13:34:52.023311 217 workload/pgx_helpers.go:79  [-] 1350  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: region "us-east1" already added to database (SQLSTATE 42710) pid:2075137 sql:ALTER DATABASE schemachange ADD REGION "us-east1" time:2.901426ms]
		  | I220606 13:34:52.308661 210 workload/pgx_helpers.go:79  [-] 1351  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema4812.table5559" does not exist (SQLSTATE 42P01) pid:1700748 sql:CREATE INDEX "IrrelevantName" ON schema4812.table5559 ("IrrelevantColumn" ASC) time:41.18307ms]
		  | I220606 13:34:52.330158 211 workload/pgx_helpers.go:79  [-] 1352  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema4812.table5558" does not exist (SQLSTATE 42P01) pid:1781910 sql:ALTER TABLE schema4812.table5558 ALTER COLUMN IrrelevantColumnName SET DEFAULT "IrrelevantValue" time:39.693446ms]
		  | I220606 13:34:52.362434 224 workload/pgx_helpers.go:79  [-] 1353  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3224.seq5560" does not exist (SQLSTATE 42P01) pid:2090407 sql:DROP SEQUENCE schema3224.seq5560 time:7.329099ms]
		  | I220606 13:34:52.618296 214 workload/pgx_helpers.go:79  [-] 1354  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "seq5548" (1453): expected matching namespace entry, found none (SQLSTATE XXUUU) pid:1792622 sql:commit time:221.080649ms]
		  | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation "seq5548" (1453): expected matching namespace entry, found none (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | ntTimestamp": "13:34:46.174947",
		  |  "ops": [
		  |   "BEGIN",
		  |   "ALTER SEQUENCE schema5048.seq5471 RENAME TO schema5048.seq5548",
		  |   "INSERT INTO schema3224.table5225 (col5225_5227,col5225_5228,col5225_5229,col5225_5230,col5225_5231,col5225_5232,col5225_5233,col5225_5234,col5225_5235,col5225_5236,col5225_5237,col5225_5238,col5225_5239,col5225_5240) VALUES ('01070000A0E610000000000000':::GEOGRAPHY,'16:37:35.406972-04:01:00':::TIMETZ,1005:::OID,'BOX(-10 -10,10 10)':::BOX2D,2202:::OID,'1978-06-17':::DATE,B'11101111100101000010',0.4085620270341669:::FLOAT8,3781804427339826273:::INT8,true,'01030000000100000006000000B83122E5F06AD0415F6CB2E1115BF2C1563D18105319F2412008A153A25DD2C1229210CA8811E6C1DAA3881587FF0042F8802F502B6BDDC1706BA8A0C068F041CD61513922B3FFC1887B15B196EBE941B83122E5F06AD0415F6CB2E1115BF2C1':::GEOMETRY,'\"':::STRING,'d8ff:e1d1:15f7:566a:64cd:4f43:484f:a837/107':::INET,4096:::OID),(NULL,'09:42:03.309393-03:14:00':::TIMETZ,1700:::OID,'BOX(-0.7915415944938847 0.42891940566383313,0.5517028864132273 2.0529120145413016)':::BOX2D,4089:::OID,NULL,B'01000000111100001011',3.4028234663852886e+38:::FLOAT8,NULL,false,'01030000000100000005000000A416EF1108A0FD4112E10F5E0426F1C1CA6DF4B8D0840042B4E3B211F9F90042AA9865CA1476FAC1206559BC313C0042DEE78754221C02C2443CCBA59169FB41A416EF1108A0FD4112E10F5E0426F1C1':::GEOMETRY,e'\\x16\\x04:\\x0f\\x1a,':::STRING:::NAME,'5509:6236:8491:e2a6:3b56:7362:aaf1:e17e/35':::INET,1183:::OID),('0103000060E61000000100000008000000C843ED93E4DF3DC00A4A04E50D643EC088C40695BAADE7C1D8A78A5C056A37C064CE405B158330C05C186F68B67DE24140213C8F87FE2B403012A55257AC124014CB71E49A0CFB41C63506C7A8F66440404F7D6985FDF8BF925EB7F62215FBC13879BB9F68AC46408A85758C8770514050F8D66CAE6DDAC1BA115C1B0E2E62C000C63528B35D3B401050CE3188A0CEC164989AB35A1762C08CC92C1630DE34406CDFF3D82EA5E5C1C843ED93E4DF3DC00A4A04E50D643EC088C40695BAADE7C1':::GEOGRAPHY,'00:32:01.204457+11:24:00':::TIMETZ,21:::OID,'BOX(-1.77477918270027 -1.6240761077574726,0.020742573623563132 1.2915659074199173)':::BOX2D,1560:::OID,'1991-03-16':::DATE,B'00100111100001110101','NaN':::FLOAT8,3686258930713013282:::INT8,true,'0107000080030000000106000080020000000103000080010000000B000000A67037E88E6BE1C1B0026A812DDFD0C1C1438BDA445BFCC1D0EFEEF16FF5EA4120A5E761368EF6C10E9F2B2AF9C1EBC116C39756D9E3F1419E2D7332A1A401C2C4BC39B2D981ED418075DE55DE40FF4174CEAE9AB69CFFC160B0075379D4DB41504E41C55B5BFB417C6A86C89B6EF0C14CF62559FC14EAC1F02B15BAB87AFC418CFD52730575E8411E7E7DD0D8F5F8C100DDFA57B6E2E2414AC9391FB29F004294EF04784F0DDEC180D648A5E9D1B4C1BCAA0305F8B7EC4148666C9EA8C0F641F89CD8C88C4AD2C1DC536BFE7A75E64198B1794ADB58FFC14E827E947693FFC15ED1103A3087F3413927F83089DBF0C1A67037E88E6BE1C1B0026A812DDFD0C1C1438BDA445BFCC10103000080010000000A0000004EABCDDBADE2F8C168AC3CB055E6E6C1A00F5A7DDECB00422009E62A3012F84142C6F34D2FC3FBC17A438C041F20024202A0F6F8647EF941E282934B0F44F8C138596CD28954F24148BDC46E9F30EB4148E17AC531D0E9C17C3090E65A0BF141BC6977AFEBC4004200A6128CAB289AC1B8AF61B92BE1F0C112B233B638C4EEC1F4A3AF694442FF41D432A404FEDDF8413EAABD1184AEF7C11499C827C849F841D8F8524937A2FC41E91254ABB40701C23AE1FA8296C3F8410A1AAA50A911F8C105A8406A45D700C2607F0E9D55DBB44154C59BBC63D2DFC14EABCDDBADE2F8C168AC3CB055E6E6C1A00F5A7DDECB004201050000800300000001020000800200000098DD3FE649AEF14187877C11A94101C2A2521F665BCFE0C1E6FB831F99FAF7410060B7732296B9C1247BFBF71F8EF3C1010200008002000000B00E2BBD425DEA41984F43B5F6C9D541EC9203B5B1C7FC41B01910D1F083FD41D41923CD4AB9E3414CB017A99699F6410102000080020000008046481EA5A9D4C168E6DFB0B30EE0C120C01D56D72A01C20089EDED1E359041A0C150215D43CCC1AC65D520D8B4F2C10101000080C4B3BE80F571FAC1B2E4669A809400C26CF11E9A678EF741':::GEOMETRY,e'\\'\\x15B !\\f\u0026^[':::STRING:::NAME,'55.253.119.32/16':::INET,1560:::OID)",
		  |   "ALTER VIEW schema4074.view5371 RENAME TO schema4074.view5563",
		  |   "CREATE TYPE schema4074.enum5564 AS ENUM ('asda', 'f', 'dsdf')",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: relation \"seq5548\" (1453): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 175eebd03e5f2674a2d1f6bfea8bd72f131a6532:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_130727.839700873_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1948
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_130727.839700873_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | S (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema3146.table5807"]] :[["(CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END)"]]
		  | QUERY ["SELECT EXISTS ( SELECT * FROM schema3146.table5807 WHERE (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END)= ( SELECT  (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END) FROM (VALUES( ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810) ) )" []] :false
		  | QUERY ["SELECT array[((CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\x0e\\\\\\bG':::STRING END))::STRING] FROM (VALUES(ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810)" []] :[["@\x0e\\\bG"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema3146.table5807'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema3146'\n\t\t           AND table_name = 'table5807'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
		  | ===========================Previous statements [INSERT INTO schema3146.table5807 (col5807_5810) VALUES (ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL])]: ERROR: duplicate key value violates unique constraint "table5807_expr_key" (SQLSTATE 23505)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |    IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema3146.table5807\"]] :[[\"(CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END)\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema3146.table5807 WHERE (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END)= ( SELECT  (CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END) FROM (VALUES( ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810) ) )\" []] :false\nQUERY [\"SELECT array[((CASE WHEN col5807_5810 IS NULL THEN ',kJ6':::STRING ELSE e'@\\\\x0e\\\\\\\\\\\\bG':::STRING END))::STRING] FROM (VALUES(ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL]) ) AS T(col5807_5810)\" []] :[[\"@\\x0e\\\\\\bG\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema3146.table5807'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema3146'\\n\\t\\t           AND table_name = 'table5807'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema3146.table5807 (col5807_5810) VALUES (ARRAY['-56 years -11 mons -333 days -14:34:42.532035':::INTERVAL,'70 years 11 mons 668 days 15:57:25.330249':::INTERVAL,'-32 years -257 days -12:56:54.368779':::INTERVAL])]: ERROR: duplicate key value violates unique constraint \"table5807_expr_key\" (SQLSTATE 23505)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ c070f2c70e0c92573a99dcb80b8a0cb0532bd06f:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:884: output in run_133618.008142896_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1949
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:884
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_133618.008142896_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | ["(5\x11k\x16,06:43:38.98882-00:11:00,5\x11k\x16)"]]
		  | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6557, col6553_6559, col6553_6556)= ( SELECT  (col6553_6557, col6553_6559, col6553_6556) FROM (VALUES( '06:43:38.98882-00:11:00':::TIMETZ,e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556) ) )" []] :false
		  | QUERY ["SELECT array[((col6553_6557, col6553_6559, col6553_6556))::STRING] FROM (VALUES('06:43:38.98882-00:11:00':::TIMETZ,e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556)" []] :[["(06:43:38.98882-00:11:00,5\x11k\x16,5\x11k\x16)"]]
		  | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (lower(CAST(col6553_6558 AS STRING)), col6553_6556)= ( SELECT  (lower(CAST(col6553_6558 AS STRING)), col6553_6556) FROM (VALUES( B'11101111011001100101101000010100',e'5\\x11k\\x16':::STRING) ) AS T(col6553_6558,col6553_6556) ) )" []] :false
		  | QUERY ["SELECT array[((lower(CAST(col6553_6558 AS STRING)), col6553_6556))::STRING] FROM (VALUES(B'11101111011001100101101000010100',e'5\\x11k\\x16':::STRING) ) AS T(col6553_6558,col6553_6556)" []] :[["(11101111011001100101101000010100,5\x11k\x16)"]]
		  | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559)= ( SELECT  (col6553_6559) FROM (VALUES( e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559) ) )" []] :false
		  | QUERY ["SELECT array[((col6553_6559))::STRING] FROM (VALUES(e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559)" []] :[["5\x11k\x16"]]
		  | QUERY ["SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559, col6553_6556)= ( SELECT  (col6553_6559, col6553_6556) FROM (VALUES( e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559,col6553_6556) ) )" []] :false
		  | QUERY ["SELECT array[((col6553_6559, col6553_6556))::STRING] FROM (VALUES(e'5\\x11k\\x16':::STRING,e'5\\x11k\\x16':::STRING) ) AS T(col6553_6559,col6553_6556)" []] :[["(5\x11k\x16,5\x11k\x16)"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema6147.table6553'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema6147' AND\n\t\t\t\t\t\t\t\t    table_name='table6553' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema6147.table6553', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema6147'\n\t\t           AND table_name = 'table6553'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["schema6147" "table6553" "col6553_6559" "col6553_6556"]]
		  | QUERY ["\n\t    SELECT count(*) = 0 from schema6147.table6553\n\t    WHERE col6553_6559 = e'5\\x11k\\x16':::STRING\n\t" []] :true
		  | ===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema6147.table6553 (col6553_6556,col6553_6557,col6553_6558) VALUES (e'5\x11k\x16':::STRING,'06:43:38.98882-00:11:00':::TIMETZ,B'11101111011001100101101000010100')]
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | :43:38.98882-00:11:00':::TIMETZ,e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556) ) )\" []] :false\nQUERY [\"SELECT array[((col6553_6557, col6553_6559, col6553_6556))::STRING] FROM (VALUES('06:43:38.98882-00:11:00':::TIMETZ,e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6557,col6553_6559,col6553_6556)\" []] :[[\"(06:43:38.98882-00:11:00,5\\x11k\\x16,5\\x11k\\x16)\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (lower(CAST(col6553_6558 AS STRING)), col6553_6556)= ( SELECT  (lower(CAST(col6553_6558 AS STRING)), col6553_6556) FROM (VALUES( B'11101111011001100101101000010100',e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6558,col6553_6556) ) )\" []] :false\nQUERY [\"SELECT array[((lower(CAST(col6553_6558 AS STRING)), col6553_6556))::STRING] FROM (VALUES(B'11101111011001100101101000010100',e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6558,col6553_6556)\" []] :[[\"(11101111011001100101101000010100,5\\x11k\\x16)\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559)= ( SELECT  (col6553_6559) FROM (VALUES( e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559) ) )\" []] :false\nQUERY [\"SELECT array[((col6553_6559))::STRING] FROM (VALUES(e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559)\" []] :[[\"5\\x11k\\x16\"]]\nQUERY [\"SELECT EXISTS ( SELECT * FROM schema6147.table6553 WHERE (col6553_6559, col6553_6556)= ( SELECT  (col6553_6559, col6553_6556) FROM (VALUES( e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559,col6553_6556) ) )\" []] :false\nQUERY [\"SELECT array[((col6553_6559, col6553_6556))::STRING] FROM (VALUES(e'5\\\\x11k\\\\x16':::STRING,e'5\\\\x11k\\\\x16':::STRING) ) AS T(col6553_6559,col6553_6556)\" []] :[[\"(5\\x11k\\x16,5\\x11k\\x16)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema6147.table6553'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema6147' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table6553' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema6147.table6553', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema6147'\\n\\t\\t           AND table_name = 'table6553'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"schema6147\" \"table6553\" \"col6553_6559\" \"col6553_6556\"]]\nQUERY [\"\\n\\t    SELECT count(*) = 0 from schema6147.table6553\\n\\t    WHERE col6553_6559 = e'5\\\\x11k\\\\x16':::STRING\\n\\t\" []] :true\n===========================Previous statements [SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema6147.table6553 (col6553_6556,col6553_6557,col6553_6558) VALUES (e'5\\x11k\\x16':::STRING,'06:43:38.98882-00:11:00':::TIMETZ,B'11101111011001100101101000010100')]"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@meridional meridional mentioned this issue Jun 9, 2022
Merged
craig bot pushed a commit that referenced this issue Jun 9, 2022
roachtest: skip flaky schemachange/random-load
563e33d 
Github issue: #82133

Release note: none
craig bot pushed a commit that referenced this issue Jun 9, 2022
Merge #82662
e8aeb0e 
82662: roachtest: skip flaky schemachange/random-load r=meridional a=meridional

Github issue: #82133

Release note: none

Co-authored-by: Ye Ji <[email protected]>
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

		  | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
		  | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT sequence_name\n    FROM information_schema.sequences\n   WHERE sequence_schema = $1\n     AND sequence_name = $2\n   )" ["schema3710" "seq4194"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema3710"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT sequence_name\n    FROM information_schema.sequences\n   WHERE sequence_schema = $1\n     AND sequence_name = $2\n   )" ["schema3710" "seq4498"]] :false
		  | ===========================Previous statements [CREATE TYPE schema3710.enum4454 AS ENUM ('a', 'ssadda', 'fa') INSERT INTO schema3074.table3873 (col3873_3874,col3873_3875,col3873_3876,col3873_3877,col3873_3878,col3873_3879,col3873_3880) VALUES (e'\x05\x1cM\x0b$':::STRING:::NAME,'t':::STRING,1266:::OID,'9c9bada6-c05c-430a-b6b8-a0d51db4dc99':::UUID,0.08652160551068466:::FLOAT8,26:::OID,(-0.2958253258539698):::FLOAT8) ALTER SEQUENCE schema3710.seq4194 RENAME TO schema3710.seq4498]: ERROR: relation "schema3710.seq4194" does not exist (SQLSTATE 42P01)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | , FAMILY (col4479_4485, col4479_4496), FAMILY (col4479_4486), FAMILY (col4479_4491), FAMILY (col4479_4493, col4479_4482, col4479_4494, col4479_4481, col4479_4488))",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: intent on key /Table/3/1/104/2/1): \"sql txn\" meta={id=fd808be9 key=/Table/1001/1/0.669857324225041/\"0\\x03Y\\t\\x1cx$SB\"/18/0 pri=0.05514920 epo=0 ts=1655560956.503322420,2 min=1655560953.301654010,0 seq=29} lock=true stat=PENDING rts=1655560953.301654010,0 wto=false gul=1655560953.801654010,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "14:02:35.489648",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE SEQUENCE IF NOT EXISTS schema4462.seq4463"
		  |  ],
		  |  "expectedExecErrors": "3F000",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 3F000Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema4462\"]] :false\nQUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema4462\" \"seq4463\"]] :false\n===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema4462.seq4463]: ERROR: cannot create \"schema4462.seq4463\" because the target database or schema does not exist (SQLSTATE 3F000)"
		  | }
		  |   733.0s        0            7.0            3.8     96.5   4563.4   4563.4   4563.4 opOk
		  |   733.0s        0            2.0            0.8   6710.9   6710.9   6710.9   6710.9 txnCmtErr
		  |   733.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 txnOk
		  |   733.0s        0            1.0            2.2     21.0     21.0     21.0     21.0 txnRbk
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "14:02:29.392667",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TYPE schema3710.enum4454 AS ENUM ('a', 'ssadda', 'fa')",
		  |   "INSERT INTO schema3074.table3873 (col3873_3874,col3873_3875,col3873_3876,col3873_3877,col3873_3878,col3873_3879,col3873_3880) VALUES (e'\\x05\\x1cM\\x0b$':::STRING:::NAME,'t':::STRING,1266:::OID,'9c9bada6-c05c-430a-b6b8-a0d51db4dc99':::UUID,0.08652160551068466:::FLOAT8,26:::OID,(-0.2958253258539698):::FLOAT8)",
		  |   "ALTER SEQUENCE schema3710.seq4194 RENAME TO schema3710.seq4498"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema3710\" \"seq4194\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema3710\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT sequence_name\\n    FROM information_schema.sequences\\n   WHERE sequence_schema = $1\\n     AND sequence_name = $2\\n   )\" [\"schema3710\" \"seq4498\"]] :false\n===========================Previous statements [CREATE TYPE schema3710.enum4454 AS ENUM ('a', 'ssadda', 'fa') INSERT INTO schema3074.table3873 (col3873_3874,col3873_3875,col3873_3876,col3873_3877,col3873_3878,col3873_3879,col3873_3880) VALUES (e'\\x05\\x1cM\\x0b$':::STRING:::NAME,'t':::STRING,1266:::OID,'9c9bada6-c05c-430a-b6b8-a0d51db4dc99':::UUID,0.08652160551068466:::FLOAT8,26:::OID,(-0.2958253258539698):::FLOAT8) ALTER SEQUENCE schema3710.seq4194 RENAME TO schema3710.seq4498]: ERROR: relation \"schema3710.seq4194\" does not exist (SQLSTATE 42P01)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 8d34ef1ea15850ee1c70470610b6652df4c317de:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:897: output in run_130204.189452170_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_130204.189452170_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | :00,11100101111010100001011100101111101)"]]
		  | QUERY ["SELECT num_nulls(col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501) > 0 FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)" []] :false
		  | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table3500 WHERE (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501)= ( SELECT  (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501) FROM (VALUES( B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507) ) )" []] :false
		  | QUERY ["SELECT array[((col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501))::STRING] FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)" []] :[["(25,20:06:57.841197,\"2001-10-16 07:31:34.000275\",5117979915624448840,12:07:52.582766-06:40:00,10000100000011001001110010111010100)"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table3500'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table3500' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table3500', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table3500'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
		  | ===========================Previous statements [CREATE TABLE public.table4712 AS SELECT public.table3500.col3500_3501 FROM public.table3500 INSERT INTO public.table3500 (col3500_3501,col3500_3502,col3500_3503,col3500_3504,col3500_3505,col3500_3506,col3500_3507) VALUES (B'10101000111100111101101010101000101','03:39:40.426812':::TIME,(-7841707273834830937):::INT8,'1994-09-15 09:14:57.00021':::TIMESTAMP,NULL,90004:::OID,'04:22:35.192972+14:48:00':::TIMETZ),(B'11100101111010100001011100101111101','07:28:52.590851':::TIME,32767:::INT8,'2006-09-01 16:29:13.000445':::TIMESTAMP,(-2147483648):::INT8,1700:::OID,'12:19:14.113199-13:37:00':::TIMETZ),(B'10000100000011001001110010111010100','20:06:57.841197':::TIME,1292809320963222986:::INT8,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,25:::OID,'12:07:52.582766-06:40:00':::TIMETZ)]: ERROR: relation "table4712" (1203): expected matching namespace entry, found none (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 00_3501) \u003e 0 FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table3500 WHERE (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501)= ( SELECT  (col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501) FROM (VALUES( B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507) ) )\" []] :false\nQUERY [\"SELECT array[((col3500_3506, col3500_3502, col3500_3504, col3500_3505, col3500_3507, col3500_3501))::STRING] FROM (VALUES(B'10000100000011001001110010111010100',25:::OID,'20:06:57.841197':::TIME,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,'12:07:52.582766-06:40:00':::TIMETZ) ) AS T(col3500_3501,col3500_3506,col3500_3502,col3500_3504,col3500_3505,col3500_3507)\" []] :[[\"(25,20:06:57.841197,\\\"2001-10-16 07:31:34.000275\\\",5117979915624448840,12:07:52.582766-06:40:00,10000100000011001001110010111010100)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table3500'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table3500' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table3500', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table3500'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [CREATE TABLE public.table4712 AS SELECT public.table3500.col3500_3501 FROM public.table3500 INSERT INTO public.table3500 (col3500_3501,col3500_3502,col3500_3503,col3500_3504,col3500_3505,col3500_3506,col3500_3507) VALUES (B'10101000111100111101101010101000101','03:39:40.426812':::TIME,(-7841707273834830937):::INT8,'1994-09-15 09:14:57.00021':::TIMESTAMP,NULL,90004:::OID,'04:22:35.192972+14:48:00':::TIMETZ),(B'11100101111010100001011100101111101','07:28:52.590851':::TIME,32767:::INT8,'2006-09-01 16:29:13.000445':::TIMESTAMP,(-2147483648):::INT8,1700:::OID,'12:19:14.113199-13:37:00':::TIMETZ),(B'10000100000011001001110010111010100','20:06:57.841197':::TIME,1292809320963222986:::INT8,'2001-10-16 07:31:34.000275':::TIMESTAMP,5117979915624448840:::INT8,25:::OID,'12:07:52.582766-06:40:00':::TIMETZ)]: ERROR: relation \"table4712\" (1203): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 377b113bd7e7b679c55a0f93e0d8ab5dbb6f2e97:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:897: output in run_132814.766426103_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1957
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_132814.766426103_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | death:
		  | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema1970"]] :false
		  | ===========================Previous statements [CREATE TABLE IF NOT EXISTS schema877.table1931 (col1931_1933 REGPROCEDURE NULL, col1931_1934 TIMETZ, col1931_1935 REGROLE NOT NULL, col1931_1936 TIMESTAMP NOT NULL, col1931_1937 REGROLE, col1931_1938 INT8 NOT NULL, col1931_1939 TIMESTAMP NULL, col1931_1940 "char", col1931_1941 STRING AS (CASE WHEN col1931_1937 IS NULL THEN e' \x0e4HQ/':::STRING ELSE e'k\x1b@\f':::STRING END) STORED, INDEX (col1931_1934, (col1931_1938 + (-7086260798774548313):::INT8), col1931_1933 ASC, col1931_1938 ASC, lower(CAST(col1931_1934 AS STRING)), col1931_1937), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1937, col1931_1933 ASC), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1938, col1931_1941 DESC, col1931_1936, col1931_1937 DESC), INDEX (col1931_1941 DESC, col1931_1937 DESC, col1931_1935 ASC, col1931_1938 ASC, col1931_1936, col1931_1933 ASC, col1931_1934, col1931_1939 DESC), INDEX (col1931_1936 ASC, col1931_1941, col1931_1937 DESC, col1931_1935, col1931_1939, col1931_1934 ASC, col1931_1940, col1931_1933), UNIQUE (col1931_1939, col1931_1934, col1931_1940 ASC) STORING (col1931_1933, col1931_1935, col1931_1938, col1931_1941), INDEX (col1931_1937 ASC, col1931_1934 ASC, col1931_1936 ASC, col1931_1935 ASC), INDEX (col1931_1938 DESC, col1931_1937 ASC, col1931_1933 DESC, col1931_1935 ASC, col1931_1939 DESC, col1931_1936 DESC, col1931_1941), FAMILY (col1931_1936, col1931_1939, col1931_1935, col1931_1941, col1931_1933), FAMILY (col1931_1937), FAMILY (col1931_1940, col1931_1934, col1931_1938)) INSERT INTO schema877.table1258 (col1258_1259,col1258_1260,col1258_1261,col1258_1262,col1258_1263,col1258_1264,col1258_1265,col1258_1266,col1258_1267,col1258_1268,col1258_1269,col1258_1270,col1258_1271,col1258_1272,col1258_1273,col1258_1274,col1258_1275) VALUES ('-56 years -103 days -08:24:29.822279':::INTERVAL,e'%MEc\x10A':::STRING:::NAME,'f4a0:bb31:5dbe:5224:45f:63bf:cbb2:eb71/87':::INET,1266:::OID,1266:::OID,10.228008859700175:::DECIMAL,'[[[["a", [], {}, []], true], []], {"a": false, "c": {"bar": []}, "foobar": {}}, []]':::JSONB,ARRAY['1989-03-06':::DATE,'1984-04-24':::DATE,'1984-07-20':::DATE,'1990-06-09':::DATE,'1979-08-25':::DATE],90002:::OID,'*':::STRING,e'{"?=\\\\<oIaR3h{": {"&|f&n+D(~<S": {"a": [], "baz": [false]}}, "bG(.-": []}':::JSONB,1560:::OID,1.7976931348623157e+308:::FLOAT8,1186:::OID,4089:::OID,1114:::OID,3802:::OID),('-30 years -3 mons -536 days -01:13:14.812631':::INTERVAL,'':::STRING:::NAME,'40.131.66.109/15':::INET,26:::OID,2202:::OID,(-2.667423998350947061E+37):::DECIMAL,e'{"8E9O:I#h4": {"r8jsH]``": {}}, ">L*me[$\\"A%a_": [{}, []], "EOG": [[{}, {}], "b:$*\\"=W,55q", "-Z9t9-1%n5"], "F.X~pIxo$": [], "``ng(|\\"}9zh#Y": {}, "a": true, "c": [[{}], []]}':::JSONB,ARRAY['1988-01-07':::DATE,'1970-05-04':::DATE],18:::OID,'3':::STRING,e'{">TFhOr": [{"Ud42HZe": 4.367809801176824}, []], "H$#OD": {}, "a": {}, "b": {" \\"|{+O;!41F": {"bar": []}, "c": [null], "foobar": []}, "bar": "baz"}':::JSONB,17:::OID,(-1.2201161276955275):::FLOAT8,90004:::OID,90000:::OID,17:::OID,869:::OID),('41 years 1 mon 429 days 17:18:59.405336':::INTERVAL,e'PXt\r\x16]':::STRING:::NAME,'7162:1d9c:2cd:9e6e:63a4:208d:a0eb:c4d9/15':::INET,700:::OID,21:::OID,(-7.531607206776994903E+23):::DECIMAL,e'[{"F(o~F/x": [{"c": {}}], "bar": {"Y)3(\\"q\\\\^Yg(": {}}, "foo": [null, [false], null]}, true, null]':::JSONB,ARRAY['1974-02-18':::DATE],90004:::OID,e'\x13':::STRING,e'{"& MtAo3-{x": {"&:&GZNCs12i": {}}, "a\'S@;;)wci8": [[[{"#gD^__E\\"": true, "Dlj": "bj!/nJ"}, [null], []]], "c", []], "wMi08n": {"b": {}}}':::JSONB,17:::OID,(-2.5003141290217457):::FLOAT8,1266:::OID,2205:::OID,21:::OID,701:::OID) CREATE SCHEMA schema1970 AUTHORIZATION root]: ERROR: relation "table1931" (572): expected matching namespace entry, found none (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema1970\"]] :false\n===========================Previous statements [CREATE TABLE IF NOT EXISTS schema877.table1931 (col1931_1933 REGPROCEDURE NULL, col1931_1934 TIMETZ, col1931_1935 REGROLE NOT NULL, col1931_1936 TIMESTAMP NOT NULL, col1931_1937 REGROLE, col1931_1938 INT8 NOT NULL, col1931_1939 TIMESTAMP NULL, col1931_1940 \"char\", col1931_1941 STRING AS (CASE WHEN col1931_1937 IS NULL THEN e' \\x0e4HQ/':::STRING ELSE e'k\\x1b@\\f':::STRING END) STORED, INDEX (col1931_1934, (col1931_1938 + (-7086260798774548313):::INT8), col1931_1933 ASC, col1931_1938 ASC, lower(CAST(col1931_1934 AS STRING)), col1931_1937), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1937, col1931_1933 ASC), UNIQUE (col1931_1940 ASC, col1931_1934 DESC, col1931_1938, col1931_1941 DESC, col1931_1936, col1931_1937 DESC), INDEX (col1931_1941 DESC, col1931_1937 DESC, col1931_1935 ASC, col1931_1938 ASC, col1931_1936, col1931_1933 ASC, col1931_1934, col1931_1939 DESC), INDEX (col1931_1936 ASC, col1931_1941, col1931_1937 DESC, col1931_1935, col1931_1939, col1931_1934 ASC, col1931_1940, col1931_1933), UNIQUE (col1931_1939, col1931_1934, col1931_1940 ASC) STORING (col1931_1933, col1931_1935, col1931_1938, col1931_1941), INDEX (col1931_1937 ASC, col1931_1934 ASC, col1931_1936 ASC, col1931_1935 ASC), INDEX (col1931_1938 DESC, col1931_1937 ASC, col1931_1933 DESC, col1931_1935 ASC, col1931_1939 DESC, col1931_1936 DESC, col1931_1941), FAMILY (col1931_1936, col1931_1939, col1931_1935, col1931_1941, col1931_1933), FAMILY (col1931_1937), FAMILY (col1931_1940, col1931_1934, col1931_1938)) INSERT INTO schema877.table1258 (col1258_1259,col1258_1260,col1258_1261,col1258_1262,col1258_1263,col1258_1264,col1258_1265,col1258_1266,col1258_1267,col1258_1268,col1258_1269,col1258_1270,col1258_1271,col1258_1272,col1258_1273,col1258_1274,col1258_1275) VALUES ('-56 years -103 days -08:24:29.822279':::INTERVAL,e'%MEc\\x10A':::STRING:::NAME,'f4a0:bb31:5dbe:5224:45f:63bf:cbb2:eb71/87':::INET,1266:::OID,1266:::OID,10.228008859700175:::DECIMAL,'[[[[\"a\", [], {}, []], true], []], {\"a\": false, \"c\": {\"bar\": []}, \"foobar\": {}}, []]':::JSONB,ARRAY['1989-03-06':::DATE,'1984-04-24':::DATE,'1984-07-20':::DATE,'1990-06-09':::DATE,'1979-08-25':::DATE],90002:::OID,'*':::STRING,e'{\"?=\\\\\\\\\u003coIaR3h{\": {\"\u0026|f\u0026n+D(~\u003cS\": {\"a\": [], \"baz\": [false]}}, \"bG(.-\": []}':::JSONB,1560:::OID,1.7976931348623157e+308:::FLOAT8,1186:::OID,4089:::OID,1114:::OID,3802:::OID),('-30 years -3 mons -536 days -01:13:14.812631':::INTERVAL,'':::STRING:::NAME,'40.131.66.109/15':::INET,26:::OID,2202:::OID,(-2.667423998350947061E+37):::DECIMAL,e'{\"8E9O:I#h4\": {\"r8jsH]``\": {}}, \"\u003eL*me[$\\\\\"A%a_\": [{}, []], \"EOG\": [[{}, {}], \"b:$*\\\\\"=W,55q\", \"-Z9t9-1%n5\"], \"F.X~pIxo$\": [], \"``ng(|\\\\\"}9zh#Y\": {}, \"a\": true, \"c\": [[{}], []]}':::JSONB,ARRAY['1988-01-07':::DATE,'1970-05-04':::DATE],18:::OID,'3':::STRING,e'{\"\u003eTFhOr\": [{\"Ud42HZe\": 4.367809801176824}, []], \"H$#OD\": {}, \"a\": {}, \"b\": {\" \\\\\"|{+O;!41F\": {\"bar\": []}, \"c\": [null], \"foobar\": []}, \"bar\": \"baz\"}':::JSONB,17:::OID,(-1.2201161276955275):::FLOAT8,90004:::OID,90000:::OID,17:::OID,869:::OID),('41 years 1 mon 429 days 17:18:59.405336':::INTERVAL,e'PXt\\r\\x16]':::STRING:::NAME,'7162:1d9c:2cd:9e6e:63a4:208d:a0eb:c4d9/15':::INET,700:::OID,21:::OID,(-7.531607206776994903E+23):::DECIMAL,e'[{\"F(o~F/x\": [{\"c\": {}}], \"bar\": {\"Y)3(\\\\\"q\\\\\\\\^Yg(\": {}}, \"foo\": [null, [false], null]}, true, null]':::JSONB,ARRAY['1974-02-18':::DATE],90004:::OID,e'\\x13':::STRING,e'{\"\u0026 MtAo3-{x\": {\"\u0026:\u0026GZNCs12i\": {}}, \"a\\'S@;;)wci8\": [[[{\"#gD^__E\\\\\"\": true, \"Dlj\": \"bj!/nJ\"}, [null], []]], \"c\", []], \"wMi08n\": {\"b\": {}}}':::JSONB,17:::OID,(-2.5003141290217457):::FLOAT8,1266:::OID,2205:::OID,21:::OID,701:::OID) CREATE SCHEMA schema1970 AUTHORIZATION root]: ERROR: relation \"table1931\" (572): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 7af6c0d3e2085cafda90d5f0b15d06f6a360f154:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_131241.151871516_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1964
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_131241.151871516_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n\t\t" ["schema2091.table3716"]] :[]
		  | QUERY ["\nWITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n\t\t" ["schema2091.table3716"]] :[]
		  | QUERY ["\n    WITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema2091.table3716"]] :[]
		  | Stmts: 
		  | []
		  | : ERROR: unknown schema "[1188]" (SQLSTATE 3F000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 01000080A29617CD28C8FDC1627898F1282EF241B841D34E6BE4FA4101010000809839950618CDED4160857E101C24BBC1FC7E53303852E5C10101000080303FD618B326024280075B9A0C46D7C1C08355394470CE4101010000809C21F0974AD1F74108310A4D79DDE54178925ADF3880DB4101010000803E0E14320B6C00C2E04934AC4145D7415EA68AA1E964F9C101030000800100000004000000C821B775C34ED2C180C56B09953FBA41F63A7C9FF886F3C18E273EE93B2FEDC1D0893EE2103DD6C1A6217CDC84EEF1412A84143B585A0142E0E2D363B506E241F45720A5B45AD9C1C821B775C34ED2C180C56B09953FBA41F63A7C9FF886F3C10104000080070000000101000080FE4E13446283F7418A9038588589E9C1FC82A2C6797DE5410101000080F464A09C909AE941D3C2F91EF60AF6C10C73262E987FF2410101000080885C7962E6D1DBC17092A066127CDA41C020A318E3F3F241010100008080D1D19EDF2AA5419C5595385F94F4416CBF5D2944CBE9C1010100008002B09C88DEAEF1C130588FEC278ACBC1988AEDEC9D4AF9410101000080D4E3DD372AC7E041EAC2910E956FF241468A35CA1301F9C10101000080F898C41EA824FB41C871C433ABFAFB41AEBDB3DDEF17F8C10104000080070000000101000080C052FECD37D2A941E848DA4603DCF841C830E144A160E24101010000809EC8EF6CE253EEC1D05DE650C651E1C17C8B67DEA06FF6410101000080947083658299E9419A6F6EC000B8EEC13C553E43F3F801C20101000080B0438FBC09DCD4C12C56EEFB60A2F44107C6C6A500BFF6C10101000080805C36645612A24168F4CC65ADC8D4C1C08AB16535ABC54101010000802B64FF4EFA4402C2D8A7732A9D63E14100F9705EB8AF88410101000080806F5C1EDDEFBDC1C4E18C04102BFB4181EBAFC5617402C2010200008003000000D872FEAF5365DEC1A083B9B84852C54158DD6247D464FAC1B0B47E0F865ED8417072065A0CD9C5411F0826C63686F2C184E66AAE95B5E1C10E51DDAE51C6014284517B19BEAAE341':::GEOMETRY,1083:::OID,19:::OID,0.6792910099029541:::FLOAT8,NULL,1082:::OID,(-661791291):::INT8){
		  |  "workerId": 0,
		  |  "clientTimestamp": "13:32:13.391106",
		  |  "ops": [
		  |   "BEGIN",
		  |   "CREATE TABLE schema2091.table3716 AS SELECT schema2091.table3716.col2719_2728, schema2091.table3716.col2719_2728, schema5028.table6347.col6347_6348, schema5028.table6347.col6347_6358, schema5028.table6347.col6347_6357, schema5028.table6347.col6347_6360, schema5028.table6347.col6347_6350, schema5028.table6347.col6347_6355, schema5028.table6347.col6347_6353, schema5028.table6347.col6347_6351, schema5028.table6347.col6347_6354 FROM schema2091.table3716, schema2091.table3716, schema5028.table6347"
		  |  ],
		  |  "expectedExecErrors": "42701,42712,42P07",
		  |  "expectedCommitErrors": "",
		  |  "message": "ROLLBACK; Successfully got expected execution error. Dumping state before death:\nExpected errors: 42701,42712,42P07Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2091\" \"table3716\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2091\" \"table3716\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema5028\" \"table6347\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema2091\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema2091\" \"table3716\"]] :true\n===========================Previous statements [CREATE TABLE schema2091.table3716 AS SELECT schema2091.table3716.col2719_2728, schema2091.table3716.col2719_2728, schema5028.table6347.col6347_6348, schema5028.table6347.col6347_6358, schema5028.table6347.col6347_6357, schema5028.table6347.col6347_6360, schema5028.table6347.col6347_6350, schema5028.table6347.col6347_6355, schema5028.table6347.col6347_6353, schema5028.table6347.col6347_6351, schema5028.table6347.col6347_6354 FROM schema2091.table3716, schema2091.table3716, schema5028.table6347]: ERROR: source name \"table3716\" specified more than once (missing AS clause) (SQLSTATE 42712)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ e1e99da678e29db4ef21813423f012273c99add2:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	test_runner.go:1027,test_runner.go:926: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 4e8134033890c956d39020f28a0e99b21b32649f:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133240.358591906_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_133240.358591906_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 2291)" []] :false
		  | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM schema895.table2289 WHERE (col2289_2291)= ( SELECT  (col2289_2291) FROM (VALUES( B'0111111111111111111111111111111111111') ) AS T(col2289_2291) ) )" []] :true
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema895.table2289'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema895' AND\n\t\t\t\t\t\t\t\t    table_name='table2289' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema895.table2289', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema895'\n\t\t           AND table_name = 'table2289'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["schema895" "table2289" "col2289_2291" "col2289_2291"]]
		  | ===========================Previous statements [INSERT INTO schema3085.table3840 (col3840_3843,col3840_3844,col3840_3845,col3840_3846,col3840_3847,col3840_3848,col3840_3849,col3840_3850,col3840_3851,col3840_3852,col3840_3853,col3840_3854,col3840_3855,col3840_3856,col3840_3857,col3840_3858,col3840_3859,col3840_3860,col3840_3861) VALUES (3803780901110908529:::INT8,3802:::OID,true,'14:12:36.797101-12:24:00':::TIMETZ,1:::INT8,ARRAY[B'11100111101101000001011010111000110010101111',B'00010001001100111000001010111101100110010001',B'10011011001011101111010110111011101111101111',B'00101000001011010011100110111111000001010101'],1083:::OID,1560:::OID,24:::OID,26:::OID,'z'::CHAR,'-59 years -811 days -02:04:13.458369':::INTERVAL,'\x4280454255a8c6':::BYTES,'122.23.1.169/1':::INET,'46 years 10 mons 356 days 12:26:51.38384':::INTERVAL,'oz0?2$k'::NAME:::NAME,e'\x03rFm'::VARCHAR,'o>,nK'::STRING,'3000-01-01 00:00:00':::TIMESTAMP),((-6087578964869502355):::INT8,2205:::OID,true,'04:01:08.380617+04:14:00':::TIMETZ,282654241:::INT8,ARRAY[B'01100100101110010101100001100100000000001101',B'01100011111100110010011010010110011100101011'],20:::OID,19:::OID,90002:::OID,19:::OID,e'\x1c'::CHAR,'36 years 8 mons 297 days 15:12:15.082235':::INTERVAL,'\xb305bb41ba07b8':::BYTES,'a953:3d0f:ee2f:6d2f:508a:93a0:bc9b:79f4/59':::INET,'-60 years -1 mons -164 days -18:38:14.39587':::INTERVAL,e'LE\x14'::NAME:::NAME,e'\x167C%\x17v f'::VARCHAR,e'\''::STRING,'1984-12-15 20:21:10.00023':::TIMESTAMP),(4027233331045315694:::INT8,90000:::OID,false,'09:45:48.944205+12:40:00':::TIMETZ,(-507780718):::INT8,ARRAY[B'00110011010001000101011111110111001000011100',B'01111111111111111111111111111111111111111111'],17:::OID,26:::OID,1114:::OID,2206:::OID,e'\f'::CHAR,'-23 years -3 mons -705 days -14:36:11.562614':::INTERVAL,'\x9c4581':::BYTES,'152.51.89.37/2':::INET,'-4 years -2 mons -217 days -14:28:10.220544':::INTERVAL,e'\x1e@M'::NAME:::NAME,e'\x11F\x14\x0fnD%'::VARCHAR,NULL,'1984-08-30 01:27:04.000602':::TIMESTAMP) INSERT INTO schema895.table2289 (col2289_2290,col2289_2291) VALUES (1560:::OID,B'0111111111111111111111111111111111111')]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  B'0111111111111111111111111111111111111') ) AS T(col2289_2291) ) )\" []] :true\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema895.table2289'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema895' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table2289' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema895.table2289', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema895'\\n\\t\\t           AND table_name = 'table2289'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"schema895\" \"table2289\" \"col2289_2291\" \"col2289_2291\"]]\n===========================Previous statements [INSERT INTO schema3085.table3840 (col3840_3843,col3840_3844,col3840_3845,col3840_3846,col3840_3847,col3840_3848,col3840_3849,col3840_3850,col3840_3851,col3840_3852,col3840_3853,col3840_3854,col3840_3855,col3840_3856,col3840_3857,col3840_3858,col3840_3859,col3840_3860,col3840_3861) VALUES (3803780901110908529:::INT8,3802:::OID,true,'14:12:36.797101-12:24:00':::TIMETZ,1:::INT8,ARRAY[B'11100111101101000001011010111000110010101111',B'00010001001100111000001010111101100110010001',B'10011011001011101111010110111011101111101111',B'00101000001011010011100110111111000001010101'],1083:::OID,1560:::OID,24:::OID,26:::OID,'z'::CHAR,'-59 years -811 days -02:04:13.458369':::INTERVAL,'\\x4280454255a8c6':::BYTES,'122.23.1.169/1':::INET,'46 years 10 mons 356 days 12:26:51.38384':::INTERVAL,'oz0?2$k'::NAME:::NAME,e'\\x03rFm'::VARCHAR,'o\u003e,nK'::STRING,'3000-01-01 00:00:00':::TIMESTAMP),((-6087578964869502355):::INT8,2205:::OID,true,'04:01:08.380617+04:14:00':::TIMETZ,282654241:::INT8,ARRAY[B'01100100101110010101100001100100000000001101',B'01100011111100110010011010010110011100101011'],20:::OID,19:::OID,90002:::OID,19:::OID,e'\\x1c'::CHAR,'36 years 8 mons 297 days 15:12:15.082235':::INTERVAL,'\\xb305bb41ba07b8':::BYTES,'a953:3d0f:ee2f:6d2f:508a:93a0:bc9b:79f4/59':::INET,'-60 years -1 mons -164 days -18:38:14.39587':::INTERVAL,e'LE\\x14'::NAME:::NAME,e'\\x167C%\\x17v f'::VARCHAR,e'\\''::STRING,'1984-12-15 20:21:10.00023':::TIMESTAMP),(4027233331045315694:::INT8,90000:::OID,false,'09:45:48.944205+12:40:00':::TIMETZ,(-507780718):::INT8,ARRAY[B'00110011010001000101011111110111001000011100',B'01111111111111111111111111111111111111111111'],17:::OID,26:::OID,1114:::OID,2206:::OID,e'\\f'::CHAR,'-23 years -3 mons -705 days -14:36:11.562614':::INTERVAL,'\\x9c4581':::BYTES,'152.51.89.37/2':::INET,'-4 years -2 mons -217 days -14:28:10.220544':::INTERVAL,e'\\x1e@M'::NAME:::NAME,e'\\x11F\\x14\\x0fnD%'::VARCHAR,NULL,'1984-08-30 01:27:04.000602':::TIMESTAMP) INSERT INTO schema895.table2289 (col2289_2290,col2289_2291) VALUES (1560:::OID,B'0111111111111111111111111111111111111')]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 3c9b17113488d2ee6929936aa6ec48396f3ed71c:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_134252.570148638_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_134252.570148638_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  (CASE WHEN col54_60 IS NULL THEN '+':::STRING ELSE e'R\\x0e':::STRING END, col54_55, col54_60, col54_65, lower(CAST(col54_63 AS STRING)), col54_58, col54_63, col54_66, col54_56, col54_59) FROM (VALUES( e'p\\x7f|H1\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55) ) )" []] :false
		  | QUERY ["SELECT array[((CASE WHEN col54_60 IS NULL THEN '+':::STRING ELSE e'R\\x0e':::STRING END, col54_55, col54_60, col54_65, lower(CAST(col54_63 AS STRING)), col54_58, col54_63, col54_66, col54_56, col54_59))::STRING] FROM (VALUES(e'p\\x7f|H1\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55)" []] :[["(R\x0e,*,\"\\\\x98f4c5d53a22\",\"p\u007f|H1\vyn\",15:03:13.818412,\"-2000-01-01 00:00:00\",15:03:13.818412,u;RF,24,19)"]]
		  | QUERY ["SELECT num_nulls(col54_59) > 0 FROM (VALUES(19:::OID) ) AS T(col54_59)" []] :false
		  | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table54 WHERE (col54_59)= ( SELECT  (col54_59) FROM (VALUES( 19:::OID) ) AS T(col54_59) ) )" []] :false
		  | QUERY ["SELECT array[((col54_59))::STRING] FROM (VALUES(19:::OID) ) AS T(col54_59)" []] :[["19"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table54'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table54' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table54', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table54'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["public" "table54" "col54_59" "col54_59"]]
		  | ===========================Previous statements [DROP SCHEMA "schema607" CASCADE INSERT INTO public.table54 (col54_55,col54_56,col54_57,col54_58,col54_59,col54_60,col54_61,col54_62,col54_63) VALUES (e'}>\'e$\'\x04'::NAME:::NAME,1114:::OID,'+'::"char",'1987-05-24 02:15:25.000317':::TIMESTAMP,1700:::OID,'\xb3ea8d':::BYTES,(-0.705607090841737):::FLOAT8,'4d5b9439-605d-4b4c-bfc8-41c9d454f0c3':::UUID,'11:53:14.69639':::TIME),(NULL,1186:::OID,'o'::"char",'1999-11-15 07:53:18.000567':::TIMESTAMP,1186:::OID,'\xda565e79c4b8':::BYTES,0.28914216093517053:::FLOAT8,'8771a065-8648-4901-9747-c082ee793943':::UUID,'23:32:57.012975':::TIME),('*'::NAME:::NAME,24:::OID,'{'::"char",'-2000-01-01 00:00:00':::TIMESTAMP,19:::OID,'\x98f4c5d53a22':::BYTES,(-1.0715612012272748):::FLOAT8,'c6dbc4a2-43e5-497e-a368-10696211875e':::UUID,'15:03:13.818412':::TIME)]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 775898849214332929 could not be loaded: job with ID 775898849214332929 does not exist (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | \\\\x7f|H1\\\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55) ) )\" []] :false\nQUERY [\"SELECT array[((CASE WHEN col54_60 IS NULL THEN '+':::STRING ELSE e'R\\\\x0e':::STRING END, col54_55, col54_60, col54_65, lower(CAST(col54_63 AS STRING)), col54_58, col54_63, col54_66, col54_56, col54_59))::STRING] FROM (VALUES(e'p\\\\x7f|H1\\\\x0byn':::STRING,'15:03:13.818412':::TIME,'-2000-01-01 00:00:00':::TIMESTAMP,'u;RF':::STRING,24:::OID,19:::OID,'\\\\x98f4c5d53a22':::BYTES,'*'::NAME:::NAME) ) AS T(col54_65,col54_63,col54_58,col54_66,col54_56,col54_59,col54_60,col54_55)\" []] :[[\"(R\\x0e,*,\\\"\\\\\\\\x98f4c5d53a22\\\",\\\"p\\u007f|H1\\vyn\\\",15:03:13.818412,\\\"-2000-01-01 00:00:00\\\",15:03:13.818412,u;RF,24,19)\"]]\nQUERY [\"SELECT num_nulls(col54_59) \u003e 0 FROM (VALUES(19:::OID) ) AS T(col54_59)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table54 WHERE (col54_59)= ( SELECT  (col54_59) FROM (VALUES( 19:::OID) ) AS T(col54_59) ) )\" []] :false\nQUERY [\"SELECT array[((col54_59))::STRING] FROM (VALUES(19:::OID) ) AS T(col54_59)\" []] :[[\"19\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table54'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table54' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table54', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table54'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"public\" \"table54\" \"col54_59\" \"col54_59\"]]\n===========================Previous statements [DROP SCHEMA \"schema607\" CASCADE INSERT INTO public.table54 (col54_55,col54_56,col54_57,col54_58,col54_59,col54_60,col54_61,col54_62,col54_63) VALUES (e'}\u003e\\'e$\\'\\x04'::NAME:::NAME,1114:::OID,'+'::\"char\",'1987-05-24 02:15:25.000317':::TIMESTAMP,1700:::OID,'\\xb3ea8d':::BYTES,(-0.705607090841737):::FLOAT8,'4d5b9439-605d-4b4c-bfc8-41c9d454f0c3':::UUID,'11:53:14.69639':::TIME),(NULL,1186:::OID,'o'::\"char\",'1999-11-15 07:53:18.000567':::TIMESTAMP,1186:::OID,'\\xda565e79c4b8':::BYTES,0.28914216093517053:::FLOAT8,'8771a065-8648-4901-9747-c082ee793943':::UUID,'23:32:57.012975':::TIME),('*'::NAME:::NAME,24:::OID,'{'::\"char\",'-2000-01-01 00:00:00':::TIMESTAMP,19:::OID,'\\x98f4c5d53a22':::BYTES,(-1.0715612012272748):::FLOAT8,'c6dbc4a2-43e5-497e-a368-10696211875e':::UUID,'15:03:13.818412':::TIME)]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 775898849214332929 could not be loaded: job with ID 775898849214332929 does not exist (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 33d70998719051ee058bc9e516afa238ea7b7451:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_133321.381602086_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_133321.381602086_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | col1437_1442);" []] :[["[{\"7qanfbm *87\": {\"/n5#\": []}, \"d+ub0ui\": {\"93yb!q*%l2\": 1.1152646744807915, \"baz\": \"ir8m!_<k1\"}, \"v%viii3\": [{\"baz\": {\"foo\": []}, \"fpfbqqyjf(f\": {}}], \"a\": {}, \"foo\": {\"b\": {\"k&8ormm>al\": true, \"vruw{o;\": \"m~w:\\\\&l\"}}}, {}]"]]
		  | QUERY ["SELECT array[(CASE WHEN col1437_1444 IS NULL THEN e'``W\\x0fk':::STRING ELSE 'gH':::STRING END)::STRING::STRING] AS c FROM ( VALUES('2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_<K1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K&8ORMM>al\": true, \"VRUw{O;\": \"m~W:\\\\\\\\&L\"}}}, {}]':::JSONB,701:::OID)) AS t(col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439);" []] :[["gH"]]
		  | QUERY ["SELECT array[(CASE WHEN col1437_1444 IS NULL THEN e'?=\\f\\x1eEx':::STRING ELSE e'<\\x03bbX\\x19a':::STRING END)::STRING::STRING] AS c FROM ( VALUES('1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_<K1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K&8ORMM>al\": true, \"VRUw{O;\": \"m~W:\\\\\\\\&L\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP)) AS t(col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439,col1437_1440);" []] :[["<\x03bbX\x19a"]]
		  | QUERY ["SELECT (((col1437_1443, col1437_1439, col1437_1447, col1437_1442 + (-2780600270711972420):::INT8, col1437_1445, col1437_1440, col1437_1441, col1437_1446))::STRING) IS NULL AS c FROM ( VALUES(e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_<K1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K&8ORMM>al\": true, \"VRUw{O;\": \"m~W:\\\\\\\\&L\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\"7qanfbm *87\": {\"/n5#\": []}, \"d+ub0ui\": {\"93yb!q*%l2\": 1.1152646744807915, \"baz\": \"ir8m!_<k1\"}, \"v%viii3\": [{\"baz\": {\"foo\": []}, \"fpfbqqyjf(f\": {}}], \"a\": {}, \"foo\": {\"b\": {\"k&8ormm>al\": true, \"vruw{o;\": \"m~w:\\\\\\\\&l\"}}}, {}]':::STRING,'gH':::STRING,e'<\\x03bbX\\x19a':::STRING)) AS t(col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1445,col1437_1446,col1437_1447);" []] :false
		  | ===========================Previous statements [CREATE TYPE schema3287.enum3465 AS ENUM ('da') INSERT INTO public.table1437 (col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444) VALUES ('[[{}], [{"5%ZxBJy4&2o": true, "JnF6": null}, [], {"9jCt-4%Q": null}, 1.9554469664025482, "~/sDY>", {}], [{"m``d9k] ^P": 1.9124982004432292}, {}, {}], false]':::JSONB,1005:::OID,'1978-11-11 07:01:55.000027':::TIMESTAMP,NULL,(-3046394468882016911):::INT8,']xi'::STRING,'1973-10-16':::DATE),('[[[null, {"foobar": {}}], {"T>~q:": null}, {"D8(C;Laz<": {}, "foo": {}, "foobar": []}, true], {"}uEUn": true}]':::JSONB,2202:::OID,'-4713-11-24 00:00:00':::TIMESTAMP,'2009-04-27 21:35:11.00064+00:00':::TIMESTAMPTZ,(-5174918294687126107):::INT8,e'$n\x07d\x12\x1dr_+'::STRING,'1985-12-10':::DATE),(e'[{"7QANfbm *87": {"/N5#": []}, "D+uB0Ui": {"93yB!q*%l2": 1.1152646744807915, "baz": "IR8m!_<K1"}, "V%viIi3": [{"baz": {"foo": []}, "fPfBqqYjf(F": {}}], "a": {}, "foo": {"b": {"K&8ORMM>al": true, "VRUw{O;": "m~W:\\\\&L"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | Z,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\\\"7QANfbm *87\\\": {\\\"/N5#\\\": []}, \\\"D+uB0Ui\\\": {\\\"93yB!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"IR8m!_\u003cK1\\\"}, \\\"V%viIi3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fPfBqqYjf(F\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"K\u00268ORMM\u003eal\\\": true, \\\"VRUw{O;\\\": \\\"m~W:\\\\\\\\\\\\\\\\\u0026L\\\"}}}, {}]':::JSONB,701:::OID)) AS t(col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439);\" []] :[[\"gH\"]]\nQUERY [\"SELECT array[(CASE WHEN col1437_1444 IS NULL THEN e'?=\\\\f\\\\x1eEx':::STRING ELSE e'\u003c\\\\x03bbX\\\\x19a':::STRING END)::STRING::STRING] AS c FROM ( VALUES('1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\\\"7QANfbm *87\\\": {\\\"/N5#\\\": []}, \\\"D+uB0Ui\\\": {\\\"93yB!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"IR8m!_\u003cK1\\\"}, \\\"V%viIi3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fPfBqqYjf(F\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"K\u00268ORMM\u003eal\\\": true, \\\"VRUw{O;\\\": \\\"m~W:\\\\\\\\\\\\\\\\\u0026L\\\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP)) AS t(col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1438,col1437_1439,col1437_1440);\" []] :[[\"\u003c\\x03bbX\\x19a\"]]\nQUERY [\"SELECT (((col1437_1443, col1437_1439, col1437_1447, col1437_1442 + (-2780600270711972420):::INT8, col1437_1445, col1437_1440, col1437_1441, col1437_1446))::STRING) IS NULL AS c FROM ( VALUES(e'[{\\\"7QANfbm *87\\\": {\\\"/N5#\\\": []}, \\\"D+uB0Ui\\\": {\\\"93yB!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"IR8m!_\u003cK1\\\"}, \\\"V%viIi3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fPfBqqYjf(F\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"K\u00268ORMM\u003eal\\\": true, \\\"VRUw{O;\\\": \\\"m~W:\\\\\\\\\\\\\\\\\u0026L\\\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE,e'[{\\\"7qanfbm *87\\\": {\\\"/n5#\\\": []}, \\\"d+ub0ui\\\": {\\\"93yb!q*%l2\\\": 1.1152646744807915, \\\"baz\\\": \\\"ir8m!_\u003ck1\\\"}, \\\"v%viii3\\\": [{\\\"baz\\\": {\\\"foo\\\": []}, \\\"fpfbqqyjf(f\\\": {}}], \\\"a\\\": {}, \\\"foo\\\": {\\\"b\\\": {\\\"k\u00268ormm\u003eal\\\": true, \\\"vruw{o;\\\": \\\"m~w:\\\\\\\\\\\\\\\\\u0026l\\\"}}}, {}]':::STRING,'gH':::STRING,e'\u003c\\\\x03bbX\\\\x19a':::STRING)) AS t(col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444,col1437_1445,col1437_1446,col1437_1447);\" []] :false\n===========================Previous statements [CREATE TYPE schema3287.enum3465 AS ENUM ('da') INSERT INTO public.table1437 (col1437_1438,col1437_1439,col1437_1440,col1437_1441,col1437_1442,col1437_1443,col1437_1444) VALUES ('[[{}], [{\"5%ZxBJy4\u00262o\": true, \"JnF6\": null}, [], {\"9jCt-4%Q\": null}, 1.9554469664025482, \"~/sDY\u003e\", {}], [{\"m``d9k] ^P\": 1.9124982004432292}, {}, {}], false]':::JSONB,1005:::OID,'1978-11-11 07:01:55.000027':::TIMESTAMP,NULL,(-3046394468882016911):::INT8,']xi'::STRING,'1973-10-16':::DATE),('[[[null, {\"foobar\": {}}], {\"T\u003e~q:\": null}, {\"D8(C;Laz\u003c\": {}, \"foo\": {}, \"foobar\": []}, true], {\"}uEUn\": true}]':::JSONB,2202:::OID,'-4713-11-24 00:00:00':::TIMESTAMP,'2009-04-27 21:35:11.00064+00:00':::TIMESTAMPTZ,(-5174918294687126107):::INT8,e'$n\\x07d\\x12\\x1dr_+'::STRING,'1985-12-10':::DATE),(e'[{\"7QANfbm *87\": {\"/N5#\": []}, \"D+uB0Ui\": {\"93yB!q*%l2\": 1.1152646744807915, \"baz\": \"IR8m!_\u003cK1\"}, \"V%viIi3\": [{\"baz\": {\"foo\": []}, \"fPfBqqYjf(F\": {}}], \"a\": {}, \"foo\": {\"b\": {\"K\u00268ORMM\u003eal\": true, \"VRUw{O;\": \"m~W:\\\\\\\\\u0026L\"}}}, {}]':::JSONB,701:::OID,'2019-04-01 16:12:27.000103':::TIMESTAMP,'1972-12-29 00:20:48.000353+00:00':::TIMESTAMPTZ,(-2847739122589181656):::INT8,'sB'::STRING,'1981-08-10':::DATE)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 03d5260a467b8640de41ee08054c1ce05f91966c:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_131334.550474315_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_131334.550474315_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | TABLE schema1323.table3498 AS SELECT "IrrelevantColumnName", schema1487.table3124.col180_185, schema1487.table3124.col180_184, schema1487.table3124.col180_183, schema1487.table3124.col1201_1213, schema1487.table3124.col1201_1208, schema1487.table3124.col180_181, schema1487.table3124.col180_182, schema1487.table3124.col1201_1203, schema1487.table3124.col1201_1205, schema1487.table3124.col1201_1212, schema1487.table3124.col1201_1206, schema1487.table3124.col1201_1204, schema1487.table3124.col1201_1202, schema1487.table3124.col180_186, schema1487.table3124.col1201_1209, schema1487.table3124.col1201_1207, public.table72.col72_75 FROM schema3021.table3497, schema1487.table3124, public.table72 time:1.405343ms]
		  | I220707 13:21:50.609828 213 workload/pgx_helpers.go:79  [-] 980  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema524.table3502" does not exist (SQLSTATE 42P01) pid:1693724 sql:INSERT INTO schema524.table3502 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:175.276332ms]
		  | I220707 13:21:52.384812 219 workload/pgx_helpers.go:79  [-] 981  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3249.table3495" does not exist (SQLSTATE 42P01) pid:1448082 sql:ALTER TABLE schema3249.table3495 RENAME COLUMN "IrrelevantColumnName" TO "OtherIrrelevantName" time:2.199876ms]
		  | I220707 13:21:53.111826 221 workload/pgx_helpers.go:79  [-] 982  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "seq3501" (886): expected matching namespace entry, found none (SQLSTATE XXUUU) pid:2086717 sql:commit time:263.093072ms]
		  | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:
		  | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema1809.seq3501 INSERT INTO schema1487.table3124 (col180_185,col180_184,col180_186,col180_182,col180_181,col180_183,col1201_1208,col1201_1203,col1201_1213,col1201_1209,col1201_1204,col1201_1202,col1201_1205,col1201_1207,col1201_1206,col1201_1212) VALUES (NULL,17:::OID,'7/W^8'::STRING,B'11110101101010101001000110111111001011',(-1.6509901285171509):::FLOAT8,'\x00':::BYTES,(-1.582002878189087):::FLOAT8,NULL,''::STRING,NULL,5.632314407969060020E+23:::DECIMAL,'01050000E0E61000000200000001020000C008000000D8C755C4DAD853401C246E55AE0633C0AB04032F7680F0C18037C72FF101D741AE82D011E19065409C84A160D1B239C0943CE6628327F9418E82F872DA500242843F62C66F7E6140CC2F0879766B4F40C447261AAC57FE4104917AC3470BF9C1E8F87A21AF973D40141A06396AB94340160F6E1354070042624A878C3FA30042801F4699B0C9FB3F66876B3BEC4440405C7CB9ECB874FC41CC3A6D6D7058F641D092588EB51E21C0E06A2DE039433540D1846FA08699F2C100A0D710F955EA419277E50BA56154C0AC41D993DEF73A40B448F0F8EF90EC4185BBE5FBED90F6C1156B3305049B5CC0B43E2051971F3B406049B55702C5C4C180443E9F35C0B7C101020000C002000000DD1457968EBE50C08B3CBF07482E56C0A82148463003D9C12336CD57187DFEC1800D11CC2ADB2D40CF41D5A7C9A950C090EED075DDB0CBC1E8B4062CB3560242':::GEOGRAPHY,'\x9c9b2d48':::BYTES,26:::OID,(-0.8428827144913749):::FLOAT8,e',\' '::STRING),('18:49:08.988719':::TIME,1186:::OID,e'u\x1a\b'::STRING,B'11111100110101100101010110110010111000',NULL,'\x435923fc88ac13':::BYTES,0.6958065032958984:::FLOAT8,''::NAME:::NAME,e'g\x05pEG\x1d,\x03\x19'::STRING,ARRAY['EZZ':::STRING:::NAME,e'\x11qO\x19#p':::STRING:::NAME],8.924687560078392295E+20:::DECIMAL,'01040000E0E61000000400000001010000C040E2BA2BB3B53140A8F644901D11504074581180C5EF01422A8C02C4A686F04101010000C04727831D17E054C090B8184DB93B504010A2F7F5419AFBC1F65792F8EDD8F1C101010000C056E73B71EF466140D0FF1F760B264F40748672390D02F9414A5BC295C670E7C101010000C0A02EF3A4F3F2204040217795C5304D4020768B48A887F741E60A7EE68383F0C1':::GEOGRAPHY,'\x200953d461a9cb96':::BYTES,90004:::OID,(-1.005883226858427):::FLOAT8,e'%+!;6H\x0f\x10'::STRING) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()]: ERROR: relation "seq3501" (886): expected matching namespace entry, found none (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 327F9418E82F872DA500242843F62C66F7E6140CC2F0879766B4F40C447261AAC57FE4104917AC3470BF9C1E8F87A21AF973D40141A06396AB94340160F6E1354070042624A878C3FA30042801F4699B0C9FB3F66876B3BEC4440405C7CB9ECB874FC41CC3A6D6D7058F641D092588EB51E21C0E06A2DE039433540D1846FA08699F2C100A0D710F955EA419277E50BA56154C0AC41D993DEF73A40B448F0F8EF90EC4185BBE5FBED90F6C1156B3305049B5CC0B43E2051971F3B406049B55702C5C4C180443E9F35C0B7C101020000C002000000DD1457968EBE50C08B3CBF07482E56C0A82148463003D9C12336CD57187DFEC1800D11CC2ADB2D40CF41D5A7C9A950C090EED075DDB0CBC1E8B4062CB3560242':::GEOGRAPHY,'\\x9c9b2d48':::BYTES,26:::OID,(-0.8428827144913749):::FLOAT8,e',\\' '::STRING),('18:49:08.988719':::TIME,1186:::OID,e'u\\x1a\\b'::STRING,B'11111100110101100101010110110010111000',NULL,'\\x435923fc88ac13':::BYTES,0.6958065032958984:::FLOAT8,''::NAME:::NAME,e'g\\x05pEG\\x1d,\\x03\\x19'::STRING,ARRAY['EZZ':::STRING:::NAME,e'\\x11qO\\x19#p':::STRING:::NAME],8.924687560078392295E+20:::DECIMAL,'01040000E0E61000000400000001010000C040E2BA2BB3B53140A8F644901D11504074581180C5EF01422A8C02C4A686F04101010000C04727831D17E054C090B8184DB93B504010A2F7F5419AFBC1F65792F8EDD8F1C101010000C056E73B71EF466140D0FF1F760B264F40748672390D02F9414A5BC295C670E7C101010000C0A02EF3A4F3F2204040217795C5304D4020768B48A887F741E60A7EE68383F0C1':::GEOGRAPHY,'\\x200953d461a9cb96':::BYTES,90004:::OID,(-1.005883226858427):::FLOAT8,e'%+!;6H\\x0f\\x10'::STRING)",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [CREATE SEQUENCE IF NOT EXISTS schema1809.seq3501 INSERT INTO schema1487.table3124 (col180_185,col180_184,col180_186,col180_182,col180_181,col180_183,col1201_1208,col1201_1203,col1201_1213,col1201_1209,col1201_1204,col1201_1202,col1201_1205,col1201_1207,col1201_1206,col1201_1212) VALUES (NULL,17:::OID,'7/W^8'::STRING,B'11110101101010101001000110111111001011',(-1.6509901285171509):::FLOAT8,'\\x00':::BYTES,(-1.582002878189087):::FLOAT8,NULL,''::STRING,NULL,5.632314407969060020E+23:::DECIMAL,'01050000E0E61000000200000001020000C008000000D8C755C4DAD853401C246E55AE0633C0AB04032F7680F0C18037C72FF101D741AE82D011E19065409C84A160D1B239C0943CE6628327F9418E82F872DA500242843F62C66F7E6140CC2F0879766B4F40C447261AAC57FE4104917AC3470BF9C1E8F87A21AF973D40141A06396AB94340160F6E1354070042624A878C3FA30042801F4699B0C9FB3F66876B3BEC4440405C7CB9ECB874FC41CC3A6D6D7058F641D092588EB51E21C0E06A2DE039433540D1846FA08699F2C100A0D710F955EA419277E50BA56154C0AC41D993DEF73A40B448F0F8EF90EC4185BBE5FBED90F6C1156B3305049B5CC0B43E2051971F3B406049B55702C5C4C180443E9F35C0B7C101020000C002000000DD1457968EBE50C08B3CBF07482E56C0A82148463003D9C12336CD57187DFEC1800D11CC2ADB2D40CF41D5A7C9A950C090EED075DDB0CBC1E8B4062CB3560242':::GEOGRAPHY,'\\x9c9b2d48':::BYTES,26:::OID,(-0.8428827144913749):::FLOAT8,e',\\' '::STRING),('18:49:08.988719':::TIME,1186:::OID,e'u\\x1a\\b'::STRING,B'11111100110101100101010110110010111000',NULL,'\\x435923fc88ac13':::BYTES,0.6958065032958984:::FLOAT8,''::NAME:::NAME,e'g\\x05pEG\\x1d,\\x03\\x19'::STRING,ARRAY['EZZ':::STRING:::NAME,e'\\x11qO\\x19#p':::STRING:::NAME],8.924687560078392295E+20:::DECIMAL,'01040000E0E61000000400000001010000C040E2BA2BB3B53140A8F644901D11504074581180C5EF01422A8C02C4A686F04101010000C04727831D17E054C090B8184DB93B504010A2F7F5419AFBC1F65792F8EDD8F1C101010000C056E73B71EF466140D0FF1F760B264F40748672390D02F9414A5BC295C670E7C101010000C0A02EF3A4F3F2204040217795C5304D4020768B48A887F741E60A7EE68383F0C1':::GEOGRAPHY,'\\x200953d461a9cb96':::BYTES,90004:::OID,(-1.005883226858427):::FLOAT8,e'%+!;6H\\x0f\\x10'::STRING) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()]: ERROR: relation \"seq3501\" (886): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 6374bd83f192bd509e50313acb431690b9c3f8ae:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_125958.784508835_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_125958.784508835_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 798, schema3074.table3756.col791_799, schema3074.table3756.col791_801 FROM schema1377.table1895, schema1470.table1789, schema3074.table3756 time:420.426951ms]
		  | I220710 13:12:54.503558 219 workload/pgx_helpers.go:79  [-] 1019  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema320.table3760" does not exist (SQLSTATE 42P01) pid:1473806 sql:INSERT INTO schema320.table3760 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:645.140776ms]
		  | I220710 13:13:01.622696 217 workload/pgx_helpers.go:79  [-] 1020  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: command is too large: 76517260 bytes (max: 67108864) (SQLSTATE XXUUU) pid:1610991 sql:CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791 time:2m1.258635508s]
		  | Error: ***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:
		  | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema320" "table998"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema1864" "table2162"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["public" "table791"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT schema_name\n\t\tFROM information_schema.schemata\n   WHERE schema_name = $1\n\t)" ["schema3074"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3074" "table3379"]] :false
		  | ===========================Previous statements [INSERT INTO schema1377.table1895 (col998_1004,col998_1000,col998_1008,col998_1011,col998_1005,col998_1002,col619_624,col619_629,col619_625,col619_620,col94_95,col94_101,col94_103,col94_97,col94_104,col94_99,col94_96) VALUES ((-696188699):::INT8,'11:14:49.454686-07:14:00':::TIMETZ,'8``<'::STRING,e'Z\r\x1e\x1dZ}FG'::STRING,ARRAY['3a3cd0cb-521d-409c-a898-bd2a8afcf300':::UUID],e'\x18'::"char",90000:::OID,'05:14:32.31676':::TIME,'1986-04-15 20:35:43.000845':::TIMESTAMP,false,'0107000040020000000101000040EC604F0AECDCFC41007E15ECEA94FE41A0684E824462FD41010400004006000000010100004016463B769AC6F2C12F23BAE2C4F1FDC193AD289D8DACFAC10101000040CC7880A9328EE9C14A56AC04F89501C2C0883A51297702420101000040165956328D62F4C110D875FC867BDE41366C309E6152F0410101000040BAF400D8D52AF9C1507F2B5CD5C6014280EE96EA9A4DD2C101010000407037928DA1A9CD41A84C7FC39747DFC13031F0C80F96D3C101010000402864A66658AFF5C1E4ED038EF544FE41D133FBEEF52AFEC1':::GEOMETRY,0.32986159773781043:::FLOAT8,0.39415409726750117:::FLOAT8,1.4332882165908813:::FLOAT8,(-0.014903925608096669):::FLOAT8,(-1.1400883197784424):::FLOAT8,'0103000020E6100000010000000D000000C83DA5BD33F746C0484D4DACEF7430C0A54C7CCCF24B5EC034F3A49943954CC0B8C58FF6482E42C099B78D2B426844C0802DF9517CFCFB3FE0066F6871BC47C03C19B18532C34F406CEC2D93DC2354C0C6C0F279146050408846A8BC88E052C09061F4792E71574048661376BA0C2340B602EBBDBFCF6240088ACB47A4A43440F4B6844936D8424028D78AD376984C40F071FE691B59264070D10F1123D154409988036663C361C0D6D199755F9555402F68E5C6982C66C0F4AFF45EF4054040C83DA5BD33F746C0484D4DACEF7430C0':::GEOGRAPHY) CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791]: ERROR: command is too large: 76517260 bytes (max: 67108864) (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 13:::FLOAT8,(-0.014903925608096669):::FLOAT8,(-1.1400883197784424):::FLOAT8,'0103000020E6100000010000000D000000C83DA5BD33F746C0484D4DACEF7430C0A54C7CCCF24B5EC034F3A49943954CC0B8C58FF6482E42C099B78D2B426844C0802DF9517CFCFB3FE0066F6871BC47C03C19B18532C34F406CEC2D93DC2354C0C6C0F279146050408846A8BC88E052C09061F4792E71574048661376BA0C2340B602EBBDBFCF6240088ACB47A4A43440F4B6844936D8424028D78AD376984C40F071FE691B59264070D10F1123D154409988036663C361C0D6D199755F9555402F68E5C6982C66C0F4AFF45EF4054040C83DA5BD33F746C0484D4DACEF7430C0':::GEOGRAPHY)",
		  |   "CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Received an unexpected execution error. Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema320\" \"table998\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema1864\" \"table2162\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table791\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT schema_name\\n\\t\\tFROM information_schema.schemata\\n   WHERE schema_name = $1\\n\\t)\" [\"schema3074\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3074\" \"table3379\"]] :false\n===========================Previous statements [INSERT INTO schema1377.table1895 (col998_1004,col998_1000,col998_1008,col998_1011,col998_1005,col998_1002,col619_624,col619_629,col619_625,col619_620,col94_95,col94_101,col94_103,col94_97,col94_104,col94_99,col94_96) VALUES ((-696188699):::INT8,'11:14:49.454686-07:14:00':::TIMETZ,'8``\u003c'::STRING,e'Z\\r\\x1e\\x1dZ}FG'::STRING,ARRAY['3a3cd0cb-521d-409c-a898-bd2a8afcf300':::UUID],e'\\x18'::\"char\",90000:::OID,'05:14:32.31676':::TIME,'1986-04-15 20:35:43.000845':::TIMESTAMP,false,'0107000040020000000101000040EC604F0AECDCFC41007E15ECEA94FE41A0684E824462FD41010400004006000000010100004016463B769AC6F2C12F23BAE2C4F1FDC193AD289D8DACFAC10101000040CC7880A9328EE9C14A56AC04F89501C2C0883A51297702420101000040165956328D62F4C110D875FC867BDE41366C309E6152F0410101000040BAF400D8D52AF9C1507F2B5CD5C6014280EE96EA9A4DD2C101010000407037928DA1A9CD41A84C7FC39747DFC13031F0C80F96D3C101010000402864A66658AFF5C1E4ED038EF544FE41D133FBEEF52AFEC1':::GEOMETRY,0.32986159773781043:::FLOAT8,0.39415409726750117:::FLOAT8,1.4332882165908813:::FLOAT8,(-0.014903925608096669):::FLOAT8,(-1.1400883197784424):::FLOAT8,'0103000020E6100000010000000D000000C83DA5BD33F746C0484D4DACEF7430C0A54C7CCCF24B5EC034F3A49943954CC0B8C58FF6482E42C099B78D2B426844C0802DF9517CFCFB3FE0066F6871BC47C03C19B18532C34F406CEC2D93DC2354C0C6C0F279146050408846A8BC88E052C09061F4792E71574048661376BA0C2340B602EBBDBFCF6240088ACB47A4A43440F4B6844936D8424028D78AD376984C40F071FE691B59264070D10F1123D154409988036663C361C0D6D199755F9555402F68E5C6982C66C0F4AFF45EF4054040C83DA5BD33F746C0484D4DACEF7430C0':::GEOGRAPHY) CREATE TABLE schema3074.table3379 AS SELECT schema320.table998.col998_1000, schema320.table998.col998_1012, schema320.table998.col998_1009, schema1864.table2162.col94_102, public.table791.col791_801, public.table791.col791_795, public.table791.col791_803 FROM schema320.table998, schema1864.table2162, public.table791]: ERROR: command is too large: 76517260 bytes (max: 67108864) (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 86e007dcb5cbde3501f138c1d768519db3487857:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_125752.352067431_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_125752.352067431_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 0103000020E61000000100000004000000B0D170B1610738C064D2E19FE0E139C09C2834EC1406574018DE66554F5631C069F66A83341360C0CA3FBD1D39F95440B0D170B1610738C064D2E19FE0E139C00105000020E610000006000000010200000003000000BCE7B97429F763C07409BA8DB9F25340704175B927B365C0FAC54AA1310256402EEE8CBEE10C58C0885C4AE641452E4001020000000300000094ED5B12925755407835053E35B83240104C01E1BC6662C0F8C95868263C534064FD6A32D5B860C0643436DAE6BA3F4001020000000800000046036576BC5F4BC0966B7D4782684EC0FCD5819D26735B4033C24CB7DEE152C04CC953C186EC514038EA55AB6E223FC0DCBC8F3C8C616140F7F4E95FC64642C0B045E7F4246C5F409AD9E7024D843EC086D23F6C4AEB6040603F70C6D00C15C000E9F9F23B540C40189E94957B314A408EC492CBCF6861C04A138ED0B5EF5140010200000002000000100E68721AB42D40CCA759A99EFD454080605376B9F234C09A9CB143CD33544001020000000300000090D8410CD85F2BC0592053C86F3A51C0BC3AE4D03383594088C3F2B9A4BD44C0EECFE37451B8634074901788195249C001020000000300000065C6486463E05BC0224209BC1AA14DC070507BAC13EC2FC06FD6743BF6B84CC014060FFFED134840C14BBA26C4D251C00106000020E6100000040000000103000000010000000B0000009CC8C65B8ADE58C014DC09ED2A3C54C0D83FB51098945B40DE729D290B5B53C0D8B3BAA867FD514060F56E6278131CC092A57444ADB5644010F7C4FC99101B40B84920DADE1E6140708D7A9F4E8821405872A9CDA08E5740B009151CDFC55040792E8868305465C04C769F2AA6EB5540DC0665BEAF925AC0A4E540A7EB8049400CEF272C310962C04800334E091841406BF817F9468E62C0C0D509C3ED902A409CC8C65B8ADE58C014DC09ED2A3C54C001030000000100000005000000000DBB3909AA114040B1AF96A1A03BC00C2F5BC998C55240655BC1A653DD41C0CC7210D4DB855E4084E76FC27BA83040686C8D3099D162C0007E550B29654B40000DBB3909AA114040B1AF96A1A03BC00103000000010000000500000000E111871FF8294052CEB06AD67B50C03877DE107CAD5B4047465A4F8B9455C0586FD4F591EB56405E16E489A2A846C07D2C4F05D97551C0E8C538EBC2AD444000E111871FF8294052CEB06AD67B50C0010300000001000000070000007A15058EDF2D62C070876A7D7FF811C0B84703B38A7866C09140270B35D955C000358B1B9CDF034073D083F9ACC355C0386793E1EC923740A8B7884655E04840F8C41679A30B46C0200E44CD6ED336406048C9AF28BB28C00E714FF9726156407A15058EDF2D62C070876A7D7FF811C00103000020E61000000100000006000000B42CFE0341FE49C0C819F3FF139031C0D4D19BB73C1D5140D48CF9D5C4CF4DC058DF313065863D4058A3BCAB506127C080BB7ACCA34C5840A08E7F12FCE92E4080043E04A96528C0143DC41DCE683D40B42CFE0341FE49C0C819F3FF139031C00102000020E610000004000000B4E9FC1AB14F554010D97AB8A76B1A4024264FD309655340760E5AE3DA9746C0D8865CAABFE85940B8AE3609E8302DC0CC2BF0B9324F6040009C6F9A00C6F8BF0102000020E61000000300000064CD7C2CEE224FC00A2639362F6042C07CA6FE325A5258401CA9EA145C4F51C0C8602468B143604024C163ED56FD46C0':::GEOGRAPHY,'-74 years -8 mons -8 days -00:29:52.117558':::INTERVAL,90004:::OID,869:::OID,ARRAY[7763035483519866360:::INT8,2147483647:::INT8],2951:::OID),('0101000020E6100000DCCFB6F269475040D899CDC9C1013B40':::GEOGRAPHY,'63 years 2 mons 535 days 21:55:46.356649':::INTERVAL,1184:::OID,20:::OID,ARRAY[(-1402262343001151102):::INT8],1043:::OID) CREATE TABLE IF NOT EXISTS schema2821.table2877 (col2877_2879 "char" NOT NULL, col2877_2880 UUID NOT NULL, col2877_2881 DATE NOT NULL, col2877_2882 REGROLE NULL, col2877_2883 INT8 NOT NULL, col2877_2884 GEOMETRY, col2877_2885 VARCHAR NOT NULL, col2877_2886 DECIMAL NOT NULL, col2877_2887 INT4 NULL, col2877_2888 STRING, col2877_2889 REGNAMESPACE NULL, col2877_2890 REGPROC, col2877_2891 GEOGRAPHY NOT NULL, col2877_2892 DECIMAL NOT NULL AS (col2877_2886 + (-7.597922143869865637E+22):::DECIMAL) VIRTUAL, col2877_2893 INT8 NULL AS (col2877_2887 + (-532726378):::INT8) STORED, col2877_2894 STRING NULL AS (CASE WHEN col2877_2889 IS NULL THEN e'CbS\x1c\x14x':::STRING ELSE e'<,;aA\b\x05':::STRING END) VIRTUAL) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema1281.table2726 (col358_362,col358_366,col358_360,col358_364,col358_365,col358_361,col358_363,col479_480) VALUES (1000:::OID,NULL,'-43 years -10 mons -603 days -10:45:44.351933':::INTERVAL,1186:::OID,e'\U00002603'::STRING,19:::OID,ARRAY[6020803765411495318:::INT8,(-6918499188159710745):::INT8],'1971-01-26':::DATE)]: ERROR: relation "seq2871" (804): expected matching namespace entry, found none (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 00100000004000000B0D170B1610738C064D2E19FE0E139C09C2834EC1406574018DE66554F5631C069F66A83341360C0CA3FBD1D39F95440B0D170B1610738C064D2E19FE0E139C00105000020E610000006000000010200000003000000BCE7B97429F763C07409BA8DB9F25340704175B927B365C0FAC54AA1310256402EEE8CBEE10C58C0885C4AE641452E4001020000000300000094ED5B12925755407835053E35B83240104C01E1BC6662C0F8C95868263C534064FD6A32D5B860C0643436DAE6BA3F4001020000000800000046036576BC5F4BC0966B7D4782684EC0FCD5819D26735B4033C24CB7DEE152C04CC953C186EC514038EA55AB6E223FC0DCBC8F3C8C616140F7F4E95FC64642C0B045E7F4246C5F409AD9E7024D843EC086D23F6C4AEB6040603F70C6D00C15C000E9F9F23B540C40189E94957B314A408EC492CBCF6861C04A138ED0B5EF5140010200000002000000100E68721AB42D40CCA759A99EFD454080605376B9F234C09A9CB143CD33544001020000000300000090D8410CD85F2BC0592053C86F3A51C0BC3AE4D03383594088C3F2B9A4BD44C0EECFE37451B8634074901788195249C001020000000300000065C6486463E05BC0224209BC1AA14DC070507BAC13EC2FC06FD6743BF6B84CC014060FFFED134840C14BBA26C4D251C00106000020E6100000040000000103000000010000000B0000009CC8C65B8ADE58C014DC09ED2A3C54C0D83FB51098945B40DE729D290B5B53C0D8B3BAA867FD514060F56E6278131CC092A57444ADB5644010F7C4FC99101B40B84920DADE1E6140708D7A9F4E8821405872A9CDA08E5740B009151CDFC55040792E8868305465C04C769F2AA6EB5540DC0665BEAF925AC0A4E540A7EB8049400CEF272C310962C04800334E091841406BF817F9468E62C0C0D509C3ED902A409CC8C65B8ADE58C014DC09ED2A3C54C001030000000100000005000000000DBB3909AA114040B1AF96A1A03BC00C2F5BC998C55240655BC1A653DD41C0CC7210D4DB855E4084E76FC27BA83040686C8D3099D162C0007E550B29654B40000DBB3909AA114040B1AF96A1A03BC00103000000010000000500000000E111871FF8294052CEB06AD67B50C03877DE107CAD5B4047465A4F8B9455C0586FD4F591EB56405E16E489A2A846C07D2C4F05D97551C0E8C538EBC2AD444000E111871FF8294052CEB06AD67B50C0010300000001000000070000007A15058EDF2D62C070876A7D7FF811C0B84703B38A7866C09140270B35D955C000358B1B9CDF034073D083F9ACC355C0386793E1EC923740A8B7884655E04840F8C41679A30B46C0200E44CD6ED336406048C9AF28BB28C00E714FF9726156407A15058EDF2D62C070876A7D7FF811C00103000020E61000000100000006000000B42CFE0341FE49C0C819F3FF139031C0D4D19BB73C1D5140D48CF9D5C4CF4DC058DF313065863D4058A3BCAB506127C080BB7ACCA34C5840A08E7F12FCE92E4080043E04A96528C0143DC41DCE683D40B42CFE0341FE49C0C819F3FF139031C00102000020E610000004000000B4E9FC1AB14F554010D97AB8A76B1A4024264FD309655340760E5AE3DA9746C0D8865CAABFE85940B8AE3609E8302DC0CC2BF0B9324F6040009C6F9A00C6F8BF0102000020E61000000300000064CD7C2CEE224FC00A2639362F6042C07CA6FE325A5258401CA9EA145C4F51C0C8602468B143604024C163ED56FD46C0':::GEOGRAPHY,'-74 years -8 mons -8 days -00:29:52.117558':::INTERVAL,90004:::OID,869:::OID,ARRAY[7763035483519866360:::INT8,2147483647:::INT8],2951:::OID),('0101000020E6100000DCCFB6F269475040D899CDC9C1013B40':::GEOGRAPHY,'63 years 2 mons 535 days 21:55:46.356649':::INTERVAL,1184:::OID,20:::OID,ARRAY[(-1402262343001151102):::INT8],1043:::OID) CREATE TABLE IF NOT EXISTS schema2821.table2877 (col2877_2879 \"char\" NOT NULL, col2877_2880 UUID NOT NULL, col2877_2881 DATE NOT NULL, col2877_2882 REGROLE NULL, col2877_2883 INT8 NOT NULL, col2877_2884 GEOMETRY, col2877_2885 VARCHAR NOT NULL, col2877_2886 DECIMAL NOT NULL, col2877_2887 INT4 NULL, col2877_2888 STRING, col2877_2889 REGNAMESPACE NULL, col2877_2890 REGPROC, col2877_2891 GEOGRAPHY NOT NULL, col2877_2892 DECIMAL NOT NULL AS (col2877_2886 + (-7.597922143869865637E+22):::DECIMAL) VIRTUAL, col2877_2893 INT8 NULL AS (col2877_2887 + (-532726378):::INT8) STORED, col2877_2894 STRING NULL AS (CASE WHEN col2877_2889 IS NULL THEN e'CbS\\x1c\\x14x':::STRING ELSE e'\u003c,;aA\\b\\x05':::STRING END) VIRTUAL) SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema1281.table2726 (col358_362,col358_366,col358_360,col358_364,col358_365,col358_361,col358_363,col479_480) VALUES (1000:::OID,NULL,'-43 years -10 mons -603 days -10:45:44.351933':::INTERVAL,1186:::OID,e'\\U00002603'::STRING,19:::OID,ARRAY[6020803765411495318:::INT8,(-6918499188159710745):::INT8],'1971-01-26':::DATE)]: ERROR: relation \"seq2871\" (804): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ e9ee21860458d997a8155734dc608cfcd050ef24:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_122518.447822259_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_122518.447822259_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | (VALUES( 19:::OID,24:::OID,e'\\x14'::\"char\",e':\\x07v#;\\x7fO':::STRING,0.48219192028045654:::FLOAT8,1043:::OID,'47.206.210.205/7':::INET,1014:::OID,'0.3708638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452) ) )" []] :false
		  | QUERY ["SELECT array[((col1449_1450, col1449_1460, col1449_1464, col1449_1456, col1449_1465, col1449_1454, col1449_1452, col1449_1459, col1449_1461, col1449_1463, col1449_1457))::STRING] FROM (VALUES(19:::OID,24:::OID,e'\\x14'::\"char\",e':\\x07v#;\\x7fO':::STRING,0.48219192028045654:::FLOAT8,1043:::OID,'47.206.210.205/7':::INET,1014:::OID,'0.3708638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452)" []] :[["(24,\x14,:\av#;\u007fO,1014,0.3708638548851013,infinity,18,0.48219192028045654,1043,47.206.210.205/7,19)"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'schema1251.table1449'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='schema1251' AND\n\t\t\t\t\t\t\t\t    table_name='table1449' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('schema1251.table1449', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'schema1251'\n\t\t           AND table_name = 'table1449'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[["schema1251" "table1449" "col1449_1452" "col1449_1452"]]
		  | ===========================Previous statements [INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (2205:::OID,24:::OID,17:::OID,NULL,'1977-08-18':::DATE,B'01111111111111111111111111111111111111111111111',90004:::OID,1114:::OID,24:::OID,(-0.3453367352485657):::FLOAT8,'5'::"char",90002:::OID,'\x95f6db6933b67c4ab4':::BYTES,'d01:835c:62d7:6c27:881f:37d6:1faf:e314/69':::INET),(1266:::OID,1700:::OID,1700:::OID,'22:53:21.995695':::TIME,'1972-01-16':::DATE,B'10110000000011101110101111010011001111111100110',23:::OID,1560:::OID,3802:::OID,(-0.09712880849838257):::FLOAT8,'g'::"char",2950:::OID,'\x73446952616f':::BYTES,'140.99.124.168/9':::INET) INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (24:::OID,18:::OID,18:::OID,'11:00:19.220298':::TIME,'infinity':::DATE,B'10011100011010111110111011011010011001110001111',1014:::OID,19:::OID,18:::OID,0.48219192028045654:::FLOAT8,e'\x14'::"char",1043:::OID,'\xeb':::BYTES,'47.206.210.205/7':::INET)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 08638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452) ) )\" []] :false\nQUERY [\"SELECT array[((col1449_1450, col1449_1460, col1449_1464, col1449_1456, col1449_1465, col1449_1454, col1449_1452, col1449_1459, col1449_1461, col1449_1463, col1449_1457))::STRING] FROM (VALUES(19:::OID,24:::OID,e'\\\\x14'::\\\"char\\\",e':\\\\x07v#;\\\\x7fO':::STRING,0.48219192028045654:::FLOAT8,1043:::OID,'47.206.210.205/7':::INET,1014:::OID,'0.3708638548851013'::FLOAT8,'infinity':::DATE,18:::OID) ) AS T(col1449_1457,col1449_1450,col1449_1460,col1449_1464,col1449_1459,col1449_1461,col1449_1463,col1449_1456,col1449_1465,col1449_1454,col1449_1452)\" []] :[[\"(24,\\x14,:\\av#;\\u007fO,1014,0.3708638548851013,infinity,18,0.48219192028045654,1043,47.206.210.205/7,19)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema1251.table1449'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema1251' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table1449' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema1251.table1449', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema1251'\\n\\t\\t           AND table_name = 'table1449'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[[\"schema1251\" \"table1449\" \"col1449_1452\" \"col1449_1452\"]]\n===========================Previous statements [INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (2205:::OID,24:::OID,17:::OID,NULL,'1977-08-18':::DATE,B'01111111111111111111111111111111111111111111111',90004:::OID,1114:::OID,24:::OID,(-0.3453367352485657):::FLOAT8,'5'::\"char\",90002:::OID,'\\x95f6db6933b67c4ab4':::BYTES,'d01:835c:62d7:6c27:881f:37d6:1faf:e314/69':::INET),(1266:::OID,1700:::OID,1700:::OID,'22:53:21.995695':::TIME,'1972-01-16':::DATE,B'10110000000011101110101111010011001111111100110',23:::OID,1560:::OID,3802:::OID,(-0.09712880849838257):::FLOAT8,'g'::\"char\",2950:::OID,'\\x73446952616f':::BYTES,'140.99.124.168/9':::INET) INSERT INTO schema1251.table1449 (col1449_1450,col1449_1451,col1449_1452,col1449_1453,col1449_1454,col1449_1455,col1449_1456,col1449_1457,col1449_1458,col1449_1459,col1449_1460,col1449_1461,col1449_1462,col1449_1463) VALUES (24:::OID,18:::OID,18:::OID,'11:00:19.220298':::TIME,'infinity':::DATE,B'10011100011010111110111011011010011001110001111',1014:::OID,19:::OID,18:::OID,0.48219192028045654:::FLOAT8,e'\\x14'::\"char\",1043:::OID,'\\xeb':::BYTES,'47.206.210.205/7':::INET)]: ERROR: internal error: cannot use CheckPrivilege without a txn (SQLSTATE XX000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 0e816efe5443764591a944edcb385c7b2762523e:

		Wraps: (2) output in run_131023.344529848_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | -] 1195  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "europe-west2" to database schemachange (SQLSTATE 42P12) pid:1488520 sql:ALTER DATABASE schemachange ADD REGION "europe-west2" time:64.34414ms]
		  | I220716 13:16:05.773356 239 workload/pgx_helpers.go:79  [-] 1196  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot drop schema "public" (SQLSTATE 3F000) pid:1527331 sql:DROP SCHEMA "public" CASCADE time:1.338759ms]
		  | I220716 13:16:05.887759 226 workload/pgx_helpers.go:79  [-] 1197  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: type "schemachange.schema1605.enum2798" already exists (SQLSTATE 42710) pid:2032719 sql:CREATE TYPE schema1605.enum2798 AS ENUM ('a') time:528.728157ms]
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198  pgx logger [error]: Query logParams=map[args:[public.table3548] err:ERROR: error in argument for $1: relation "public.table3548" does not exist (SQLSTATE 42P01) pid:1226535 sql:
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +SELECT array_agg(index_name)
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +  FROM (
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +SELECT
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +	index_name
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +FROM
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +	crdb_internal.table_indexes
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +WHERE
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +	index_type = 'primary'
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +	AND descriptor_id = $1::REGCLASS
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +       );
		  | I220716 13:16:06.703788 240 workload/pgx_helpers.go:79  [-] 1198 +	]
		  | I220716 13:16:06.720284 232 workload/pgx_helpers.go:79  [-] 1199  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: duplicate key value violates unique constraint "table3231_col3231_3240_col3231_3237_key" (SQLSTATE 23505) pid:1218423 sql:INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1983-07-28 23:42:56.000958':::TIMESTAMP,2202:::OID,''::STRING,'BOX(-0.8028798948299167 0.4866737668766704,0.25842982473176523 1.256152449899192)':::BOX2D,'{"b": [true, [{}, {"c": []}, true]], "bar": null, "baz": [[]], "foo": "$Kk7U&Sf,/", "|iHKGe&Y|)i": 0.21917494336233792}':::JSONB,90002:::OID) time:134.01211ms]
		  | Error: ***UNEXPECTED ERROR; Failed to generate a random operation
		  |  OpGen log: 
		  | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["public" "table3548"]] :true
		  | Stmts: 
		  | [INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1992-05-03 17:51:29.00024':::TIMESTAMP,869:::OID,e'\x1bo|QL>'::STRING,'BOX(-1.48848753568185 0.11413130096700558,-0.5829904756554781 1.2114179120739301)':::BOX2D,e'[{"\'-{\\"#qb$": [], "c": {"Zo&U": {"(;19~W5": 0.3397086645906202, "a": []}, "bar": {"{![Cg": {}}}, "foo": {"b": {}, "c": {"H)a7I": []}}}, {"Rf\\\\X": false, "}p>&I\'AiYD,{": {}}]':::JSONB,90004:::OID) INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1971-02-05 02:00:12.000346':::TIMESTAMP,1043:::OID,e'[rO\x15iq<W~'::STRING,'BOX(-0.9235027037328147 -1.2006760750506533,-0.8793129890189155 0.17191664750169722)':::BOX2D,'[[{}], [{"b": {}, "bar": {}, "u/RSD.r``}": [true]}, []], [{}], []]':::JSONB,17:::OID),('1974-08-28 05:54:34.000015':::TIMESTAMP,19:::OID,''::STRING,NULL,'1':::JSONB,1560:::OID)]
		  | : scanStringArray "\nSELECT array_agg(index_name)\n  FROM (\nSELECT\n\tindex_name\nFROM\n\tcrdb_internal.table_indexes\nWHERE\n\tindex_type = 'primary'\n\tAND descriptor_id = $1::REGCLASS\n       );\n\t" ["public.table3548"]: ERROR: error in argument for $1: relation "public.table3548" does not exist (SQLSTATE 42P01)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1983-07-28 23:42:56.000958':::TIMESTAMP,2202:::OID,''::STRING,'BOX(-0.8028798948299167 0.4866737668766704,0.25842982473176523 1.256152449899192)':::BOX2D,'{\"b\": [true, [{}, {\"c\": []}, true]], \"bar\": null, \"baz\": [[]], \"foo\": \"$Kk7U\u0026Sf,/\", \"|iHKGe\u0026Y|)i\": 0.21917494336233792}':::JSONB,90002:::OID)]: ERROR: duplicate key value violates unique constraint \"table3231_col3231_3240_col3231_3237_key\" (SQLSTATE 23505)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "13:15:59.206776",
		  |  "ops": [
		  |   "BEGIN",
		  |   "INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1992-05-03 17:51:29.00024':::TIMESTAMP,869:::OID,e'\\x1bo|QL\u003e'::STRING,'BOX(-1.48848753568185 0.11413130096700558,-0.5829904756554781 1.2114179120739301)':::BOX2D,e'[{\"\\'-{\\\\\"#qb$\": [], \"c\": {\"Zo\u0026U\": {\"(;19~W5\": 0.3397086645906202, \"a\": []}, \"bar\": {\"{![Cg\": {}}}, \"foo\": {\"b\": {}, \"c\": {\"H)a7I\": []}}}, {\"Rf\\\\\\\\X\": false, \"}p\u003e\u0026I\\'AiYD,{\": {}}]':::JSONB,90004:::OID)",
		  |   "INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1971-02-05 02:00:12.000346':::TIMESTAMP,1043:::OID,e'[rO\\x15iq\u003cW~'::STRING,'BOX(-0.9235027037328147 -1.2006760750506533,-0.8793129890189155 0.17191664750169722)':::BOX2D,'[[{}], [{\"b\": {}, \"bar\": {}, \"u/RSD.r``}\": [true]}, []], [{}], []]':::JSONB,17:::OID),('1974-08-28 05:54:34.000015':::TIMESTAMP,19:::OID,''::STRING,NULL,'1':::JSONB,1560:::OID)"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation\n OpGen log: \nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"public\" \"table3548\"]] :true\nStmts: \n[INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1992-05-03 17:51:29.00024':::TIMESTAMP,869:::OID,e'\\x1bo|QL\u003e'::STRING,'BOX(-1.48848753568185 0.11413130096700558,-0.5829904756554781 1.2114179120739301)':::BOX2D,e'[{\"\\'-{\\\\\"#qb$\": [], \"c\": {\"Zo\u0026U\": {\"(;19~W5\": 0.3397086645906202, \"a\": []}, \"bar\": {\"{![Cg\": {}}}, \"foo\": {\"b\": {}, \"c\": {\"H)a7I\": []}}}, {\"Rf\\\\\\\\X\": false, \"}p\u003e\u0026I\\'AiYD,{\": {}}]':::JSONB,90004:::OID) INSERT INTO schema2811.table3231 (col3231_3232,col3231_3233,col3231_3234,col3231_3235,col3231_3236,col3231_3237) VALUES ('1971-02-05 02:00:12.000346':::TIMESTAMP,1043:::OID,e'[rO\\x15iq\u003cW~'::STRING,'BOX(-0.9235027037328147 -1.2006760750506533,-0.8793129890189155 0.17191664750169722)':::BOX2D,'[[{}], [{\"b\": {}, \"bar\": {}, \"u/RSD.r``}\": [true]}, []], [{}], []]':::JSONB,17:::OID),('1974-08-28 05:54:34.000015':::TIMESTAMP,19:::OID,''::STRING,NULL,'1':::JSONB,1560:::OID)]\n: scanStringArray \"\\nSELECT array_agg(index_name)\\n  FROM (\\nSELECT\\n\\tindex_name\\nFROM\\n\\tcrdb_internal.table_indexes\\nWHERE\\n\\tindex_type = 'primary'\\n\\tAND descriptor_id = $1::REGCLASS\\n       );\\n\\t\" [\"public.table3548\"]: ERROR: error in argument for $1: relation \"public.table3548\" does not exist (SQLSTATE 42P01)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ e4cafeb8b1d586d091fb98e3e570650d7eeea294:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:896: output in run_125223.608565153_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_125223.608565153_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  FLOAT4 NOT NULL, col139_141 TIME NOT NULL, col139_142 NAME NULL, col139_143 JSONB NOT NULL, col139_144 INET NOT NULL, col139_145 INET NULL, col139_146 STRING NOT NULL AS (lower(CAST(col139_144 AS STRING))) VIRTUAL, INVERTED INDEX (col139_141, col139_142 gin_trgm_ops ASC) PARTITION BY LIST (col139_141) (PARTITION table139_part_0 VALUES IN (('24:00:00':::TIME,), ('23:37:14.196281':::TIME,), ('05:25:13.281703':::TIME,), ('01:53:05.700101':::TIME,), ('19:55:22.660942':::TIME,), ('13:09:51.902929':::TIME,), ('07:35:05.868837':::TIME,), ('19:11:32.353996':::TIME,)), PARTITION table139_part_1 VALUES IN (('17:55:44.07421':::TIME,), ('20:48:27.589646':::TIME,), ('07:08:53.715879':::TIME,), ('11:21:19.692067':::TIME,), ('13:59:06.021694':::TIME,), ('10:02:12.241381':::TIME,), ('18:14:46.021724':::TIME,), ('12:29:38.802148':::TIME,)), PARTITION table139_part_2 VALUES IN (('14:42:29.996344':::TIME,), ('11:23:05.141123':::TIME,), ('01:48:59.930289':::TIME,), ('18:27:24.136821':::TIME,), ('00:00:00':::TIME,), ('14:56:32.653137':::TIME,), ('00:08:26.334906':::TIME,), ('18:30:00.665433':::TIME,)), PARTITION table139_part_3 VALUES IN (('05:32:15.193743':::TIME,), ('03:57:30.432515':::TIME,), ('13:11:47.918144':::TIME,), ('08:25:41.942095':::TIME,), ('04:09:21.342126':::TIME,), ('23:12:19.158671':::TIME,), ('17:29:23.834196':::TIME,), ('10:52:17.605332':::TIME,)), PARTITION table139_part_4 VALUES IN (('11:17:36.747781':::TIME,), ('07:47:06.089525':::TIME,), ('10:55:02.190203':::TIME,), ('10:48:58.225295':::TIME,), ('12:45:42.754913':::TIME,), ('10:54:59.160654':::TIME,), ('11:49:36.699482':::TIME,)), PARTITION table139_part_5 VALUES IN (('21:49:23.451647':::TIME,), ('15:17:22.036342':::TIME,), ('23:57:56.831033':::TIME,), ('17:45:29.588652':::TIME,), ('07:19:57.297439':::TIME,), ('12:35:18.123245':::TIME,)), PARTITION table139_part_6 VALUES IN (('13:01:37.934688':::TIME,), ('11:13:07.842972':::TIME,), ('07:34:37.631871':::TIME,), ('15:16:40.572605':::TIME,), ('22:14:53.010854':::TIME,), ('15:14:52.795521':::TIME,), ('01:04:03.747686':::TIME,)), PARTITION table139_part_7 VALUES IN (('05:05:05.689702':::TIME,), ('15:28:25.631971':::TIME,), ('17:48:21.603086':::TIME,), ('14:14:47.692422':::TIME,), ('04:37:12.672538':::TIME,), ('17:32:54.435286':::TIME,)), PARTITION table139_part_8 VALUES IN (('00:07:54.421109':::TIME,), ('22:23:01.532495':::TIME,), ('03:54:00.602109':::TIME,), ('00:36:29.80034':::TIME,), ('22:32:02.953054':::TIME,), ('06:38:22.466239':::TIME,)), PARTITION table139_part_9 VALUES IN (('03:13:02.860947':::TIME,), ('15:23:33.758746':::TIME,), ('03:34:32.950878':::TIME,), ('08:20:43.90328':::TIME,), ('07:28:35.873363':::TIME,), ('12:49:27.320904':::TIME,), ('10:45:10.306046':::TIME,))), UNIQUE (col139_144) STORING (col139_140, col139_141, col139_142), UNIQUE (lower(CAST(col139_144 AS STRING)) DESC, lower(CAST(col139_143 AS STRING)), lower(CAST(col139_145 AS STRING)) ASC), UNIQUE (col139_142, col139_141 DESC, col139_140 DESC, col139_144 DESC), FAMILY (col139_144), FAMILY (col139_140, col139_141, col139_143, col139_142), FAMILY (col139_145)) INSERT INTO public.table139 (col139_140,col139_141,col139_142,col139_143,col139_144,col139_145) VALUES ((-0.2963637709617615):::FLOAT8,'12:44:43.837899':::TIME,e'\'6"'::NAME:::NAME,'[{"#FI": [[[]], []], "a": [["}|-#+s2rC", [], [], true], false]}, [], {"5RK": [0.0421417206374064]}]':::JSONB,'997d:8848:7d08:79d5:5a5a:f1f4:b2f5:2b1e/86':::INET,NULL),((-0.13457858562469482):::FLOAT8,'19:55:35.483005':::TIME,e'\''::NAME,e'{"9:zLDXA": {"(kh_/)Z\\"d": {}, "QvUGlWg~o@": {"OM$J91": {}}, "^,(g!!Wl": [[{"bar": [[]]}], true], "b": [null], "s8JVv8": [], "|}YJ[I-": {}}, "r!t^y,": []}':::JSONB,'0.0.0.0/0':::INET,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),((-1.0784741640090942):::FLOAT8,'00:00:00':::TIME,'c6EE'::NAME:::NAME,e'[[[[{"qQW!": {}}], {"H1\'@0z6z|P": [null, [], false, {}]}, [[]]], {}, null], []]':::JSONB,'195.61.126.28/5':::INET,NULL)]: ERROR: internal error: unexpected error from the vectorized engine: interface conversion: tree.Datum is *tree.DOidWrapper, not *tree.DString (SQLSTATE XX000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | , col139_142 NAME NULL, col139_143 JSONB NOT NULL, col139_144 INET NOT NULL, col139_145 INET NULL, col139_146 STRING NOT NULL AS (lower(CAST(col139_144 AS STRING))) VIRTUAL, INVERTED INDEX (col139_141, col139_142 gin_trgm_ops ASC) PARTITION BY LIST (col139_141) (PARTITION table139_part_0 VALUES IN (('24:00:00':::TIME,), ('23:37:14.196281':::TIME,), ('05:25:13.281703':::TIME,), ('01:53:05.700101':::TIME,), ('19:55:22.660942':::TIME,), ('13:09:51.902929':::TIME,), ('07:35:05.868837':::TIME,), ('19:11:32.353996':::TIME,)), PARTITION table139_part_1 VALUES IN (('17:55:44.07421':::TIME,), ('20:48:27.589646':::TIME,), ('07:08:53.715879':::TIME,), ('11:21:19.692067':::TIME,), ('13:59:06.021694':::TIME,), ('10:02:12.241381':::TIME,), ('18:14:46.021724':::TIME,), ('12:29:38.802148':::TIME,)), PARTITION table139_part_2 VALUES IN (('14:42:29.996344':::TIME,), ('11:23:05.141123':::TIME,), ('01:48:59.930289':::TIME,), ('18:27:24.136821':::TIME,), ('00:00:00':::TIME,), ('14:56:32.653137':::TIME,), ('00:08:26.334906':::TIME,), ('18:30:00.665433':::TIME,)), PARTITION table139_part_3 VALUES IN (('05:32:15.193743':::TIME,), ('03:57:30.432515':::TIME,), ('13:11:47.918144':::TIME,), ('08:25:41.942095':::TIME,), ('04:09:21.342126':::TIME,), ('23:12:19.158671':::TIME,), ('17:29:23.834196':::TIME,), ('10:52:17.605332':::TIME,)), PARTITION table139_part_4 VALUES IN (('11:17:36.747781':::TIME,), ('07:47:06.089525':::TIME,), ('10:55:02.190203':::TIME,), ('10:48:58.225295':::TIME,), ('12:45:42.754913':::TIME,), ('10:54:59.160654':::TIME,), ('11:49:36.699482':::TIME,)), PARTITION table139_part_5 VALUES IN (('21:49:23.451647':::TIME,), ('15:17:22.036342':::TIME,), ('23:57:56.831033':::TIME,), ('17:45:29.588652':::TIME,), ('07:19:57.297439':::TIME,), ('12:35:18.123245':::TIME,)), PARTITION table139_part_6 VALUES IN (('13:01:37.934688':::TIME,), ('11:13:07.842972':::TIME,), ('07:34:37.631871':::TIME,), ('15:16:40.572605':::TIME,), ('22:14:53.010854':::TIME,), ('15:14:52.795521':::TIME,), ('01:04:03.747686':::TIME,)), PARTITION table139_part_7 VALUES IN (('05:05:05.689702':::TIME,), ('15:28:25.631971':::TIME,), ('17:48:21.603086':::TIME,), ('14:14:47.692422':::TIME,), ('04:37:12.672538':::TIME,), ('17:32:54.435286':::TIME,)), PARTITION table139_part_8 VALUES IN (('00:07:54.421109':::TIME,), ('22:23:01.532495':::TIME,), ('03:54:00.602109':::TIME,), ('00:36:29.80034':::TIME,), ('22:32:02.953054':::TIME,), ('06:38:22.466239':::TIME,)), PARTITION table139_part_9 VALUES IN (('03:13:02.860947':::TIME,), ('15:23:33.758746':::TIME,), ('03:34:32.950878':::TIME,), ('08:20:43.90328':::TIME,), ('07:28:35.873363':::TIME,), ('12:49:27.320904':::TIME,), ('10:45:10.306046':::TIME,))), UNIQUE (col139_144) STORING (col139_140, col139_141, col139_142), UNIQUE (lower(CAST(col139_144 AS STRING)) DESC, lower(CAST(col139_143 AS STRING)), lower(CAST(col139_145 AS STRING)) ASC), UNIQUE (col139_142, col139_141 DESC, col139_140 DESC, col139_144 DESC), FAMILY (col139_144), FAMILY (col139_140, col139_141, col139_143, col139_142), FAMILY (col139_145)) INSERT INTO public.table139 (col139_140,col139_141,col139_142,col139_143,col139_144,col139_145) VALUES ((-0.2963637709617615):::FLOAT8,'12:44:43.837899':::TIME,e'\\'6\"'::NAME:::NAME,'[{\"#FI\": [[[]], []], \"a\": [[\"}|-#+s2rC\", [], [], true], false]}, [], {\"5RK\": [0.0421417206374064]}]':::JSONB,'997d:8848:7d08:79d5:5a5a:f1f4:b2f5:2b1e/86':::INET,NULL),((-0.13457858562469482):::FLOAT8,'19:55:35.483005':::TIME,e'\\''::NAME,e'{\"9:zLDXA\": {\"(kh_/)Z\\\\\"d\": {}, \"QvUGlWg~o@\": {\"OM$J91\": {}}, \"^,(g!!Wl\": [[{\"bar\": [[]]}], true], \"b\": [null], \"s8JVv8\": [], \"|}YJ[I-\": {}}, \"r!t^y,\": []}':::JSONB,'0.0.0.0/0':::INET,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),((-1.0784741640090942):::FLOAT8,'00:00:00':::TIME,'c6EE'::NAME:::NAME,e'[[[[{\"qQW!\": {}}], {\"H1\\'@0z6z|P\": [null, [], false, {}]}, [[]]], {}, null], []]':::JSONB,'195.61.126.28/5':::INET,NULL)]: ERROR: internal error: unexpected error from the vectorized engine: interface conversion: tree.Datum is *tree.DOidWrapper, not *tree.DString (SQLSTATE XX000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 3dcf715f37595a73e8504b5331493f75207c6524:

		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1972
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:896
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1581
		Wraps: (2) output in run_131753.284990293_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:396.443µs]
		  | I220719 13:28:00.325512 234 workload/pgx_helpers.go:79  [-] 1302  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema2555.table3841" does not exist (SQLSTATE 42P01) pid:1657702 sql:CREATE TABLE schema3198.table3860 AS SELECT "IrrelevantColumnName", schema1303.table2587.col2587_2591, schema1303.table2587.col2587_2593, schema2809.table3147.col3147_3148, schema2809.table3147.col3147_3151, schema2809.table3147.col3147_3150, schema2809.table3147.col3147_3149, schema2809.table3147.col3147_3152, schema2809.table3147.col3147_3154 FROM schema2555.table3841, schema1303.table2587, schema2809.table3147 time:176.743834ms]
		  | I220719 13:28:00.831970 227 workload/pgx_helpers.go:79  [-] 1303  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "us-west1" to database schemachange (SQLSTATE 42P12) pid:1509596 sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:177.165183ms]
		  | I220719 13:28:02.092115 240 workload/pgx_helpers.go:79  [-] 1304  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3657.table3865" does not exist (SQLSTATE 42P01) pid:1679658 sql:INSERT INTO schema3657.table3865 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:808.447µs]
		  | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305  pgx logger [error]: Query logParams=map[args:[] err:ERROR: referenced schema ID 1005: descriptor not found (SQLSTATE XXUUU) pid:1686650 sql:
		  | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +  SELECT schema_name, table_name
		  | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +    FROM [SHOW TABLES]
		  | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +   WHERE table_name ~ 'table[0-9]+'
		  | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +ORDER BY random()
		  | I220719 13:28:04.595631 232 workload/pgx_helpers.go:79  [-] 1305 +   LIMIT 1;]
		  | Error: ***UNEXPECTED ERROR; Failed to generate a random operation
		  |  OpGen log: 
		  |
		  | Stmts: 
		  | [CREATE TABLE schema2809.table3826 (col3826_3828 INT2 NOT NULL, col3826_3829 UUID, col3826_3830 REGROLE NULL, UNIQUE (col3826_3829, col3826_3830 ASC), INDEX (col3826_3828, col3826_3830) STORING (col3826_3829), UNIQUE (col3826_3829 ASC, col3826_3830 DESC, col3826_3828), INDEX (col3826_3830 ASC, col3826_3829, col3826_3828), INDEX (col3826_3829) PARTITION BY LIST (col3826_3829) (PARTITION table3826_part_0 VALUES IN (('00000000-0000-0000-0000-000000000000':::UUID,), ('15ad563c-19b9-40f2-90c8-e351cf96e0be':::UUID,), ('14e6a3a4-5c56-4211-a541-906b593b481a':::UUID,)), PARTITION table3826_part_1 VALUES IN (('382b745a-3db0-43aa-b015-7ab7c9ad1679':::UUID,), ('4c0cf83b-7c9f-4400-9457-4bb8636895df':::UUID,), ('e1ee0b87-de5a-4ad6-8716-9369d6ffb0bb':::UUID,)), PARTITION table3826_part_2 VALUES IN (('0d749848-c234-436b-86a0-7eff610b9c55':::UUID,), ('3bab0819-a490-40a6-9ea7-cf7635a8f30b':::UUID,), ('3061acf1-fbb2-44b9-b203-7352a335cb44':::UUID,)), PARTITION table3826_part_3 VALUES IN (('247beb63-fedb-4a82-9d9e-d86f27db3d6d':::UUID,), ('1d67cfa0-0db0-4e2e-8768-c19b725dab5f':::UUID,), ('3ab948c5-b41a-4071-a893-6ccec7aabb96':::UUID,)))) CREATE SCHEMA schema3842 AUTHORIZATION root SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema2809.table3147 (col3147_3148,col3147_3149,col3147_3150,col3147_3151,col3147_3152,col3147_3153,col3147_3154,col3147_3155) VALUES ((-7630.165712410323711):::DECIMAL,B'101100011011101010100111111',17:::OID,e'\x1fkg'::STRING,21:::OID,NULL,1560:::OID,B'011111111111111'),(2743276165876.467938:::DECIMAL,B'010011000111110110001110011',1083:::OID,e'^)x\x01\x1e"_D\x0b'::STRING,4089:::OID,'33 years 11 mons 539 days 20:35:09.254163':::INTERVAL,24:::OID,B'110001101011000'),(3661649782431.620875:::DECIMAL,B'111001110100001101001011111',1560:::OID,e'\x1d\x1b\x16hYb)nQ'::STRING,3802:::OID,'17 years 1 mon 987 days 19:05:02.435918':::INTERVAL,20:::OID,B'110011101000101')]
		  | : error getting random table name: ERROR: referenced schema ID 1005: descriptor not found (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 0 REGROLE NULL, UNIQUE (col3826_3829, col3826_3830 ASC), INDEX (col3826_3828, col3826_3830) STORING (col3826_3829), UNIQUE (col3826_3829 ASC, col3826_3830 DESC, col3826_3828), INDEX (col3826_3830 ASC, col3826_3829, col3826_3828), INDEX (col3826_3829) PARTITION BY LIST (col3826_3829) (PARTITION table3826_part_0 VALUES IN (('00000000-0000-0000-0000-000000000000':::UUID,), ('15ad563c-19b9-40f2-90c8-e351cf96e0be':::UUID,), ('14e6a3a4-5c56-4211-a541-906b593b481a':::UUID,)), PARTITION table3826_part_1 VALUES IN (('382b745a-3db0-43aa-b015-7ab7c9ad1679':::UUID,), ('4c0cf83b-7c9f-4400-9457-4bb8636895df':::UUID,), ('e1ee0b87-de5a-4ad6-8716-9369d6ffb0bb':::UUID,)), PARTITION table3826_part_2 VALUES IN (('0d749848-c234-436b-86a0-7eff610b9c55':::UUID,), ('3bab0819-a490-40a6-9ea7-cf7635a8f30b':::UUID,), ('3061acf1-fbb2-44b9-b203-7352a335cb44':::UUID,)), PARTITION table3826_part_3 VALUES IN (('247beb63-fedb-4a82-9d9e-d86f27db3d6d':::UUID,), ('1d67cfa0-0db0-4e2e-8768-c19b725dab5f':::UUID,), ('3ab948c5-b41a-4071-a893-6ccec7aabb96':::UUID,))))",
		  |   "CREATE SCHEMA schema3842 AUTHORIZATION root",
		  |   "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()",
		  |   "INSERT INTO schema2809.table3147 (col3147_3148,col3147_3149,col3147_3150,col3147_3151,col3147_3152,col3147_3153,col3147_3154,col3147_3155) VALUES ((-7630.165712410323711):::DECIMAL,B'101100011011101010100111111',17:::OID,e'\\x1fkg'::STRING,21:::OID,NULL,1560:::OID,B'011111111111111'),(2743276165876.467938:::DECIMAL,B'010011000111110110001110011',1083:::OID,e'^)x\\x01\\x1e\"_D\\x0b'::STRING,4089:::OID,'33 years 11 mons 539 days 20:35:09.254163':::INTERVAL,24:::OID,B'110001101011000'),(3661649782431.620875:::DECIMAL,B'111001110100001101001011111',1560:::OID,e'\\x1d\\x1b\\x16hYb)nQ'::STRING,3802:::OID,'17 years 1 mon 987 days 19:05:02.435918':::INTERVAL,20:::OID,B'110011101000101')"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED ERROR; Failed to generate a random operation\n OpGen log: \n\nStmts: \n[CREATE TABLE schema2809.table3826 (col3826_3828 INT2 NOT NULL, col3826_3829 UUID, col3826_3830 REGROLE NULL, UNIQUE (col3826_3829, col3826_3830 ASC), INDEX (col3826_3828, col3826_3830) STORING (col3826_3829), UNIQUE (col3826_3829 ASC, col3826_3830 DESC, col3826_3828), INDEX (col3826_3830 ASC, col3826_3829, col3826_3828), INDEX (col3826_3829) PARTITION BY LIST (col3826_3829) (PARTITION table3826_part_0 VALUES IN (('00000000-0000-0000-0000-000000000000':::UUID,), ('15ad563c-19b9-40f2-90c8-e351cf96e0be':::UUID,), ('14e6a3a4-5c56-4211-a541-906b593b481a':::UUID,)), PARTITION table3826_part_1 VALUES IN (('382b745a-3db0-43aa-b015-7ab7c9ad1679':::UUID,), ('4c0cf83b-7c9f-4400-9457-4bb8636895df':::UUID,), ('e1ee0b87-de5a-4ad6-8716-9369d6ffb0bb':::UUID,)), PARTITION table3826_part_2 VALUES IN (('0d749848-c234-436b-86a0-7eff610b9c55':::UUID,), ('3bab0819-a490-40a6-9ea7-cf7635a8f30b':::UUID,), ('3061acf1-fbb2-44b9-b203-7352a335cb44':::UUID,)), PARTITION table3826_part_3 VALUES IN (('247beb63-fedb-4a82-9d9e-d86f27db3d6d':::UUID,), ('1d67cfa0-0db0-4e2e-8768-c19b725dab5f':::UUID,), ('3ab948c5-b41a-4071-a893-6ccec7aabb96':::UUID,)))) CREATE SCHEMA schema3842 AUTHORIZATION root SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs() INSERT INTO schema2809.table3147 (col3147_3148,col3147_3149,col3147_3150,col3147_3151,col3147_3152,col3147_3153,col3147_3154,col3147_3155) VALUES ((-7630.165712410323711):::DECIMAL,B'101100011011101010100111111',17:::OID,e'\\x1fkg'::STRING,21:::OID,NULL,1560:::OID,B'011111111111111'),(2743276165876.467938:::DECIMAL,B'010011000111110110001110011',1083:::OID,e'^)x\\x01\\x1e\"_D\\x0b'::STRING,4089:::OID,'33 years 11 mons 539 days 20:35:09.254163':::INTERVAL,24:::OID,B'110001101011000'),(3661649782431.620875:::DECIMAL,B'111001110100001101001011111',1560:::OID,e'\\x1d\\x1b\\x16hYb)nQ'::STRING,3802:::OID,'17 years 1 mon 987 days 19:05:02.435918':::INTERVAL,20:::OID,B'110011101000101')]\n: error getting random table name: ERROR: referenced schema ID 1005: descriptor not found (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ f59620ec646d1181d358d0dc41ab60815ecf59c9:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:166,schemachange_random_load.go:58,test_runner.go:897: output in run_142724.345168039_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1981
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:163
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:58
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:897
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1571
		Wraps: (2) output in run_142724.345168039_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 0987 sql:SELECT sum(num_retries) + sum(num_auto_retries) FROM crdb_internal.cluster_transactions WHERE id=$1]
		  | I220828 14:39:32.873901 220144 workload/pgx_helpers.go:79  [-] 1866  pgx logger [error]: Query logParams=map[args:[170f8879fef0a36e0000000000000001] err:can't scan into dest[1]: cannot scan null into *string pid:1853638 sql:SELECT active_queries, kv_txn FROM crdb_internal.cluster_sessions WHERE session_id = $1]
		  | I220828 14:39:32.918455 221 workload/pgx_helpers.go:79  [-] 1867  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: transaction committed but schema change aborted with error: (XXUUU): job 791763898982465537 could not be loaded: job with ID 791763898982465537 does not exist (SQLSTATE XXA00) pid:1109351 sql:commit time:1.311683227s]
		  | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:
		  | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [QUERY: INSERT INTO schema1724.table2880 (col2089_2099,col2089_2097,col2089_2098) VALUES (NULL,1266:::OID,NULL),(NULL,2206:::OID,'1981-08-13 14:05:51.000334':::TIMESTAMP), Expected Errors: , Potential Errors:  QUERY: DROP SCHEMA "schema883" CASCADE, Expected Errors: , Potential Errors:  QUERY: SELECT t1.col3108_3111 AS col0,t2.col3108_3111 AS col1,t2.col3108_3112 AS col2,t2.col3108_3111 AS col3,t2.col3108_3112 AS col4 FROM schema1036.table3085 AS t0 ,schema2878.table3108 AS t1 ,schema2878.table3108 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO schema960.table3222 (col3222_3223,col3222_3224,col3222_3225,col3222_3226,col3222_3227,col3222_3228,col3222_3229,col3222_3230,col3222_3231,col3222_3232,col3222_3233,col3222_3234) VALUES (90004:::OID,869:::OID,'[{"baz": true}, false, [0.9247392964302076, {"+p)?ME#Igb``F": {}}]]':::JSONB,NULL,18:::OID,e'[[{}], [{"foo": [{"UP\\"<i3sF]": [], "\\\\)QRnj": "hX6``~$b{", "c": 2.1823215525789683}, "baz"], "lAL\\\\<,>{": "{iP%"}], [[], {}], false]':::JSONB,'e47a970c-e168-4337-a1f1-297363a38ccc':::UUID,'1984-01-19':::DATE,ARRAY[]:::BOX2D[],e'6\x01'::VARCHAR,(-10021):::INT8,3.4028234663852886e+38:::FLOAT8),(4096:::OID,90002:::OID,e'{"# %G?/ZU": {",)03p\\\\": {}}, "P+C": [{"\\"Hk-J": {}}, null, [true], 0.18354290333985984, null], "cy&rR": false, "r)J9B,6": []}':::JSONB,'1992-07-25':::DATE,1186:::OID,'[[[{}], {"baz": [], "foobar": [], "y>7sw~w": {}}, [{"a": [null]}, {}], []], {}, 0.6018873870111852]':::JSONB,'362ca0bc-871b-4058-96b2-b4dff1915ddc':::UUID,'1985-02-28':::DATE,ARRAY['BOX(-1.0181908860286981 0.18530395215814816,0.5254117218302146 1.034998708167156)':::BOX2D,'BOX(-0.28661207607902983 -0.5389349608185564,-0.015288337666203577 -0.40233266470478224)':::BOX2D,'BOX(-2.0543113996119904 0.7875088956272273,0.916865925285563 0.9893326932918927)':::BOX2D,'BOX(-1.2183186451741432 -0.7138806038744121,0.47261435420629283 -0.03558315795788117)':::BOX2D],'] oRWe,'::VARCHAR,(-22554):::INT8,(-0.2746864855289459):::FLOAT8),(90002:::OID,25:::OID,e'[{"I?uOviQg%)#2": "=5.n\\\\B", "V~u]r14e": {"MNHstWRZ": {"baz": "c", "foo": []}}}, [{"!q>Z": "_~\\\\( 8mU8c"}, []], {"<1[MSm.}": [], "a": []}]':::JSONB,'1994-12-17':::DATE,2205:::OID,e'[{"\'\'?hZZ@": [1.7963067695346675], "<!{$_30$U": [], "baz": null, "j%Xv>M": "b"}, {"mbk?": []}, [[]], false, {}]':::JSONB,'d1bfbad1-54cc-4e0e-b763-f4464294b9f3':::UUID,'1979-10-12':::DATE,ARRAY['BOX(-1.5615589742057119 -1.12256509236807,1.792201970494969 -0.5322828313219645)':::BOX2D,'BOX(0.31251457737123034 1.1537515644843743,1.2750033079224639 1.2541575071703341)':::BOX2D],e'\x18^(!k'::VARCHAR,(-32295):::INT8,0.6518183350563049:::FLOAT8), Expected Errors: , Potential Errors:  QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 791763898982465537 could not be loaded: job with ID 791763898982465537 does not exist (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | ": []}, [[]], false, {}]':::JSONB,'d1bfbad1-54cc-4e0e-b763-f4464294b9f3':::UUID,'1979-10-12':::DATE,ARRAY['BOX(-1.5615589742057119 -1.12256509236807,1.792201970494969 -0.5322828313219645)':::BOX2D,'BOX(0.31251457737123034 1.1537515644843743,1.2750033079224639 1.2541575071703341)':::BOX2D],e'\\x18^(!k'::VARCHAR,(-32295):::INT8,0.6518183350563049:::FLOAT8), Expected Errors: , Potential Errors: ",
		  |   "QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: ===========================Previous statements [QUERY: INSERT INTO schema1724.table2880 (col2089_2099,col2089_2097,col2089_2098) VALUES (NULL,1266:::OID,NULL),(NULL,2206:::OID,'1981-08-13 14:05:51.000334':::TIMESTAMP), Expected Errors: , Potential Errors:  QUERY: DROP SCHEMA \"schema883\" CASCADE, Expected Errors: , Potential Errors:  QUERY: SELECT t1.col3108_3111 AS col0,t2.col3108_3111 AS col1,t2.col3108_3112 AS col2,t2.col3108_3111 AS col3,t2.col3108_3112 AS col4 FROM schema1036.table3085 AS t0 ,schema2878.table3108 AS t1 ,schema2878.table3108 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO schema960.table3222 (col3222_3223,col3222_3224,col3222_3225,col3222_3226,col3222_3227,col3222_3228,col3222_3229,col3222_3230,col3222_3231,col3222_3232,col3222_3233,col3222_3234) VALUES (90004:::OID,869:::OID,'[{\"baz\": true}, false, [0.9247392964302076, {\"+p)?ME#Igb``F\": {}}]]':::JSONB,NULL,18:::OID,e'[[{}], [{\"foo\": [{\"UP\\\\\"\u003ci3sF]\": [], \"\\\\\\\\)QRnj\": \"hX6``~$b{\", \"c\": 2.1823215525789683}, \"baz\"], \"lAL\\\\\\\\\u003c,\u003e{\": \"{iP%\"}], [[], {}], false]':::JSONB,'e47a970c-e168-4337-a1f1-297363a38ccc':::UUID,'1984-01-19':::DATE,ARRAY[]:::BOX2D[],e'6\\x01'::VARCHAR,(-10021):::INT8,3.4028234663852886e+38:::FLOAT8),(4096:::OID,90002:::OID,e'{\"# %G?/ZU\": {\",)03p\\\\\\\\\": {}}, \"P+C\": [{\"\\\\\"Hk-J\": {}}, null, [true], 0.18354290333985984, null], \"cy\u0026rR\": false, \"r)J9B,6\": []}':::JSONB,'1992-07-25':::DATE,1186:::OID,'[[[{}], {\"baz\": [], \"foobar\": [], \"y\u003e7sw~w\": {}}, [{\"a\": [null]}, {}], []], {}, 0.6018873870111852]':::JSONB,'362ca0bc-871b-4058-96b2-b4dff1915ddc':::UUID,'1985-02-28':::DATE,ARRAY['BOX(-1.0181908860286981 0.18530395215814816,0.5254117218302146 1.034998708167156)':::BOX2D,'BOX(-0.28661207607902983 -0.5389349608185564,-0.015288337666203577 -0.40233266470478224)':::BOX2D,'BOX(-2.0543113996119904 0.7875088956272273,0.916865925285563 0.9893326932918927)':::BOX2D,'BOX(-1.2183186451741432 -0.7138806038744121,0.47261435420629283 -0.03558315795788117)':::BOX2D],'] oRWe,'::VARCHAR,(-22554):::INT8,(-0.2746864855289459):::FLOAT8),(90002:::OID,25:::OID,e'[{\"I?uOviQg%)#2\": \"=5.n\\\\\\\\B\", \"V~u]r14e\": {\"MNHstWRZ\": {\"baz\": \"c\", \"foo\": []}}}, [{\"!q\u003eZ\": \"_~\\\\\\\\( 8mU8c\"}, []], {\"\u003c1[MSm.}\": [], \"a\": []}]':::JSONB,'1994-12-17':::DATE,2205:::OID,e'[{\"\\'\\'?hZZ@\": [1.7963067695346675], \"\u003c!{$_30$U\": [], \"baz\": null, \"j%Xv\u003eM\": \"b\"}, {\"mbk?\": []}, [[]], false, {}]':::JSONB,'d1bfbad1-54cc-4e0e-b763-f4464294b9f3':::UUID,'1979-10-12':::DATE,ARRAY['BOX(-1.5615589742057119 -1.12256509236807,1.792201970494969 -0.5322828313219645)':::BOX2D,'BOX(0.31251457737123034 1.1537515644843743,1.2750033079224639 1.2541575071703341)':::BOX2D],e'\\x18^(!k'::VARCHAR,(-32295):::INT8,0.6518183350563049:::FLOAT8), Expected Errors: , Potential Errors:  QUERY: SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs(), Expected Errors: , Potential Errors: ]: ERROR: transaction committed but schema change aborted with error: (XXUUU): job 791763898982465537 could not be loaded: job with ID 791763898982465537 does not exist (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 3b16435371a43d603d193a1e2b480a23fba3f07a:

		  | runtime.selectgo(0xc0000a4f78, 0xc0000a4f44, 0xc001094100?, 0x0, 0x3f65800?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0000a4f08 sp=0xc0000a4dc8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002211310, {0x54f7c38, 0xc001094100})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0000a4fb8 sp=0xc0000a4f08 pc=0x1aa65ee
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0000a4fe0 sp=0xc0000a4fb8 pc=0x1aa68ee
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000a4fe8 sp=0xc0000a4fe0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 373441 [runnable]:
		  | runtime.gopark(0xc0011e4fb0?, 0x2?, 0xf8?, 0x4e?, 0xc0011e4fa4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0011e4e30 sp=0xc0011e4e10 pc=0x465216
		  | runtime.selectgo(0xc0011e4fb0, 0xc0011e4fa0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0011e4f70 sp=0xc0011e4e30 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0011e4fe0 sp=0xc0011e4f70 pc=0xd2e74c
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0011e4fe8 sp=0xc0011e4fe0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | C2FB41AC8D24261F76014200D0BE0EBFEEFA4180F722105E4994C188D027A7B3BDD441F4A8FE813D21FF41347F0CD6AECAFD41C89173517689D0410CEBD8A5EE7EED412CAEF2950DBCE041404B28C95C63CDC100F452BE2FA5AB4100B275870AC5C1410157883371BC00C2F6B5572151A7F4C1B2B2E56777B0F1410103000080010000000C0000004AD7BFDA010702C2B4BBE96B2C13FDC1C856A9E0100AF34170A7003A1B67C8C1462933DC38A7FAC130B31632FEC4E0C1C00E558FD2B6A0C17D019DC7B4F5FEC1335FF0ED03CCF2C1E02CEC1F1657DE41F21A41BA869CFEC11096A4E3ABABFE41F884EC071159F141BE489885430AFEC15AA35056FA3FFDC1C073327DEEEFB3C108183A89131AEDC18CCCBA10B7FFE1419CEF902F00B9E241B0B41C113449EBC1FC51150F9673EB4140BFBCB4FAE5A34168967BDFCB80D5C1B0C02A1D7FF4FD417022DE672BC2D8C1E04CE577AA94BDC1340623497F8A02421EB5A658B085E9C1FAB598725368004200EB1FEFFF7597C176B6C3DCE70AF5C1CCEC3ED08FE3E6C1DA8EFF744BB0F1414AD7BFDA010702C2B4BBE96B2C13FDC1C856A9E0100AF3410103000080010000000B000000A0891162391BE2416063B49D442DFDC1B495ABC69E28FF41880006C12E69F7416A4B838C24EBF2C1C8E53C9D350CE34192FEBA47C2ED00424CAB10C1F183F7C1745959909F95E8C1B6654CBAD0EE0042E1DB376FA369F2C1814313B206E801C2641DCD5C0675FC4148B46C776E8EDFC1E82CE454A903ED413AA4F12AD10C004244355C500291F4418E776D0F2315F741A0D270C71CF4B541E00309D14704D341103AC981417FFB415025257439D2CE41A0E6F08671CFB741DB81210662D3FDC1E856A294D8C4EAC1C02C15DEA13BC74100BDF8291056CCC1B0DE94D07E4001C24812963F8030DEC1908E6C617136C441A0891162391BE2416063B49D442DFDC1B495ABC69E28FF410103000080010000000C000000658EAB1A2C6B02C28067A696F75C984192A8053799BCFDC1B2DF19E019C5F6C100ECFFFBDF6277C10A9E8C2D4F57F9C1AC243F067E14F2C1A505F7A05FF401C234497F970987004260C6F1F353F3C641024F3679E231E3C1B83364C2F3F2F6C1A8E17FD3B257FB41400197DCE86EE7C15D21D4C1026FF4C1267CD61903B6F041E05C7796EB4BC04190C2C33C4C04E141A41ECBC1AED1FC418438BE9B592EFD41D8429968EDBAD1C120C7450A6F06B9414434F7AED691E34140511F36E34BEA4188ED8C03715FDE41E8333F92D7A9F34121CE41E8EDD7F2C1887DF706DCD9DAC1965CEEB0DD58F041CE49CFF1450301429A31884CBCE601C27CEF44745F2AED41480D305026B0F7C1658EAB1A2C6B02C28067A696F75C984192A8053799BCFDC10103000080010000000C0000000059F4F350C5E8C15C3031495B92DEC17CA63C2E5D24FB4199A6E73BB851FFC192649573926701C210C015635AC6CE4148B94FE8CC2FF7C13EBCAB40A277FBC104778240A80E01C2AC636315E469F5415BCE779A9180FDC16C03BEAE819AECC178749AEC20BFEB412CF10208B1E8E3C170BF84BBCC50DE417A5E24923A3C0242A4D2E074D6F3E14190C04754ED01CCC180D5D994D881BFC196E67516218CF741F082202CE6D7CBC14A7AC1855776E3C130D199CE513AEC41165AD52F5B42FBC124E9B4AB8DFEF8C15CB2A656ACBDFF41949D31E97302E841DCADBFB0B5C0F1C100D9911A2A64D2419428C553086D00C21F0022FAFEC301C2082477482B06DF414038FDD56C70C2410059F4F350C5E8C15C3031495B92DEC17CA63C2E5D24FB4101030000800100000006000000464E3293B54000C20EC23643D889F6C10FBA02CB74ACF0C1D8F40969E5B2D4C13545FAF3BDD0F2C1AE4FF0649BE8F9C12622C22360640042A4D6BB577EEEF241CA76CC4597B8F7416496417DE058F541301BDF56F390F44128C06F222168F7415E43E8763E5EF6C1F091970286A2F341AC3E4951E85CFE41464E3293B54000C20EC23643D889F6C10FBA02CB74ACF0C10103000080010000000A000000BE1453EF7FC0E9C100156E0EEC42A2C1F4A65202C117FEC144C16AA198FBFEC15B787D4C2FBDF3C138FD8B9B063ADD413CECC8E8AE1CEBC104634FA43AABF0C1F5733E66EF9102C2D0386679CB47DF4129C9D5762A4AF4C16C322D7AF5DCFFC150442BE804E4F54150835E862554DCC1DC740101138B00C2BAE57AEC8863F941B053E9671B3BC2C19C28954C8691FE418049F4E3441EE141940D967EDB7EFD410075940DE7C9C8C1A8474F9C526800C20AE5492E1A7E02425434439437E4FC417218BE27F1DBFEC16A2FF629CAF3F0419498D8798E26FE41BE1453EF7FC0E9C100156E0EEC42A2C1F4A65202C117FEC1':::GEOMETRY,2202:::OID,1560:::OID), Expected Errors: , Potential Errors: ]: ERROR: relation \"seq3541\" (816): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "14:01:37.348142",
		  |  "ops": null,
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "WATCH DOG: failed to get session information: can't scan into dest[1]: cannot scan null into *string\n"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "14:01:37.379094",
		  |  "ops": null,
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "WATCH DOG: failed to get session information: can't scan into dest[1]: cannot scan null into *string\n"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ d35c174b71065264b8a3910df3f488d10741f788:

		  | I220830 13:59:12.309875 228 workload/pgx_helpers.go:79  [-] 1740  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3256.table4424" does not exist (SQLSTATE 42P01) pid:1414074 sql:INSERT INTO schema3256.table4424 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:349.260748ms]
		  | I220830 13:59:12.317142 278172 workload/pgx_helpers.go:79  [-] 1741  pgx logger [error]: Query logParams=map[args:[1710239df6eab8620000000000000001] err:can't scan into dest[1]: cannot scan null into *string pid:1880538 sql:SELECT active_queries, kv_txn FROM crdb_internal.cluster_sessions WHERE session_id = $1]
		  | I220830 13:59:13.096869 229 workload/pgx_helpers.go:79  [-] 1742  pgx logger [error]: Query logParams=map[args:[] err:ERROR: relation "public.table4426" does not exist (SQLSTATE 42P01) pid:1676029 sql:SELECT t0.col4238_4245 AS col0,t0.col4238_4245 AS col1,t0.col4238_4244 AS col2,t0.col4238_4240 AS col3,t0.col4238_4240 AS col4 FROM schema3523.table4238 AS t0 ,public.table4426 AS t1 ,schema4427.table4428 AS t2  FETCH FIRST 1 ROWS ONLY]
		  | I220830 13:59:14.106330 216 workload/pgx_helpers.go:79  [-] 1743  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: cannot add region "us-west1" to database schemachange (SQLSTATE 42P12) pid:2025457 sql:ALTER DATABASE schemachange ADD REGION "us-west1" time:366.088µs]
		  | I220830 13:59:15.082626 224 workload/pgx_helpers.go:79  [-] 1744  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "schema3523.table4433" does not exist (SQLSTATE 42P01) pid:1423779 sql:INSERT INTO schema3523.table4433 (IrrelevantColumnName) VALUES ("IrrelevantValue") time:4.669346ms]
		  | I220830 13:59:15.193300 277802 workload/pgx_helpers.go:79  [-] 1745  pgx logger [error]: Query logParams=map[args:[1710239df74abab00000000000000001] err:can't scan into dest[1]: cannot scan null into *string pid:1857984 sql:SELECT active_queries, kv_txn FROM crdb_internal.cluster_sessions WHERE session_id = $1]
		  | I220830 13:59:16.048989 226 workload/pgx_helpers.go:79  [-] 1746  pgx logger [error]: Exec logParams=map[args:[] err:ERROR: relation "seq4408" (1042): expected matching namespace entry, found none (SQLSTATE XXUUU) pid:1631576 sql:commit time:262.266381ms]
		  | Error: ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:
		  | Expected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3523" "table4238"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3523" "table4238"]] :true
		  | QUERY ["SELECT EXISTS (\n\tSELECT table_name\n    FROM information_schema.tables \n   WHERE table_schema = $1\n     AND table_name = $2\n   )" ["schema3523" "table4238"]] :true
		  | ===========================Previous statements [QUERY: CREATE SEQUENCE IF NOT EXISTS schema3256.seq4408, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (90004:::OID,e'7.K\x07~'::NAME:::NAME,'-71 years -10 mons -436 days -08:35:40.778842':::INTERVAL,''::CHAR,701:::OID),(1700:::OID,e'\x1c*i'::NAME:::NAME,'-1 years -4 mons -550 days -10:38:42.211373':::INTERVAL,'x'::CHAR,18:::OID), Expected Errors: , Potential Errors:  QUERY: SELECT t1.col4238_4245 AS col0,t2.col4238_4239 AS col1,t2.col4238_4243 AS col2,t1.col4238_4242 AS col3,t0.col4238_4242 AS col4,t0.col4238_4243 AS col5 FROM schema3523.table4238 AS t0 ,schema3523.table4238 AS t1 ,schema3523.table4238 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]: ERROR: relation "seq4408" (1042): expected matching namespace entry, found none (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 8} lock=true stat=PENDING rts=1661867950.072283708,0 wto=false gul=1661867950.572283708,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "13:59:10.217362",
		  |  "ops": [
		  |   "BEGIN",
		  |   "QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (1560:::OID,e'\\x17CkRq'::NAME:::NAME,'80 years 8 mons 378 days 01:58:44.972277':::INTERVAL,e'\\x02'::CHAR,1560:::OID),(700:::OID,e';[lh\\t\\x07L\\x1f'::NAME:::NAME,'-51 years -9 mons -511 days -02:54:43.912576':::INTERVAL,'7'::CHAR,4089:::OID),(1042:::OID,e'9\\x1a\\x18'::NAME:::NAME,'-5 years -7 mons -183 days -09:23:35.871636':::INTERVAL,e'\\t'::CHAR,701:::OID), Expected Errors: , Potential Errors: ",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "TXN RETRY ERROR; ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/104/2/1): \"sql txn\" meta={id=1550ed73 key=/Table/1008/1/\"\"/80 years 8 mons 378 days 01:58:44.972277/\"\\x02\"/\"\\x17CkRq\"/1560/\"w\\x0f\\th\\rWS'{\"/0 pri=0.04391046 epo=0 ts=1661867952.632632671,2 min=1661867950.215964378,0 seq=12} lock=true stat=PENDING rts=1661867950.215964378,0 wto=false gul=1661867950.715964378,0 (SQLSTATE 40001)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "13:59:07.251598",
		  |  "ops": [
		  |   "BEGIN",
		  |   "QUERY: CREATE SEQUENCE IF NOT EXISTS schema3256.seq4408, Expected Errors: , Potential Errors: ",
		  |   "QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (90004:::OID,e'7.K\\x07~'::NAME:::NAME,'-71 years -10 mons -436 days -08:35:40.778842':::INTERVAL,''::CHAR,701:::OID),(1700:::OID,e'\\x1c*i'::NAME:::NAME,'-1 years -4 mons -550 days -10:38:42.211373':::INTERVAL,'x'::CHAR,18:::OID), Expected Errors: , Potential Errors: ",
		  |   "QUERY: SELECT t1.col4238_4245 AS col0,t2.col4238_4239 AS col1,t2.col4238_4243 AS col2,t1.col4238_4242 AS col3,t0.col4238_4242 AS col4,t0.col4238_4243 AS col5 FROM schema3523.table4238 AS t0 ,schema3523.table4238 AS t1 ,schema3523.table4238 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200",
		  |   "COMMIT"
		  |  ],
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "***UNEXPECTED COMMIT ERROR; Received an unexpected commit error Dumping state before death:\nExpected errors: Potential exec errors: Expected commit errors: Potential commit errors: ===========================Executed queries for generating errors: QUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3523\" \"table4238\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3523\" \"table4238\"]] :true\nQUERY [\"SELECT EXISTS (\\n\\tSELECT table_name\\n    FROM information_schema.tables \\n   WHERE table_schema = $1\\n     AND table_name = $2\\n   )\" [\"schema3523\" \"table4238\"]] :true\n===========================Previous statements [QUERY: CREATE SEQUENCE IF NOT EXISTS schema3256.seq4408, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3523.table4238 (col4238_4239,col4238_4240,col4238_4241,col4238_4242,col4238_4243) VALUES (90004:::OID,e'7.K\\x07~'::NAME:::NAME,'-71 years -10 mons -436 days -08:35:40.778842':::INTERVAL,''::CHAR,701:::OID),(1700:::OID,e'\\x1c*i'::NAME:::NAME,'-1 years -4 mons -550 days -10:38:42.211373':::INTERVAL,'x'::CHAR,18:::OID), Expected Errors: , Potential Errors:  QUERY: SELECT t1.col4238_4245 AS col0,t2.col4238_4239 AS col1,t2.col4238_4243 AS col2,t1.col4238_4242 AS col3,t0.col4238_4242 AS col4,t0.col4238_4243 AS col5 FROM schema3523.table4238 AS t0 ,schema3523.table4238 AS t1 ,schema3523.table4238 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]: ERROR: relation \"seq4408\" (1042): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ b316a5ed5fe7253d113174d9d95ddebf1143b4e4:

		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 368902 [runnable]:
		  | runtime.gopark(0xc000f11f78?, 0x3?, 0x0?, 0x30?, 0xc000f11f4a?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc000f11dc8 sp=0xc000f11da8 pc=0x465216
		  | runtime.selectgo(0xc000f11f78, 0xc000f11f44, 0xc0010a2b00?, 0x0, 0x3f71820?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc000f11f08 sp=0xc000f11dc8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002bbcc30, {0x5506a98, 0xc0010a2b00})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000f11fb8 sp=0xc000f11f08 pc=0x1aab6ae
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000f11fe0 sp=0xc000f11fb8 pc=0x1aab9ae
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000f11fe8 sp=0xc000f11fe0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 368989 [runnable]:
		  | runtime.gopark(0xc00222ef78?, 0x3?, 0x0?, 0x30?, 0xc00222ef4a?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00222edc8 sp=0xc00222eda8 pc=0x465216
		  | runtime.selectgo(0xc00222ef78, 0xc00222ef44, 0xc0010a2b00?, 0x0, 0x3f71820?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00222ef08 sp=0xc00222edc8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002868960, {0x5506a98, 0xc0010a2b00})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc00222efb8 sp=0xc00222ef08 pc=0x1aab6ae
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc00222efe0 sp=0xc00222efb8 pc=0x1aab9ae
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00222efe8 sp=0xc00222efe0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |  (col4955_4961, col4955_4957, col4955_4956, col4955_4967, col4955_4958, col4955_4963, col4955_4968, col4955_4960, col4955_4966, col4955_4962, col4955_4965)= ( SELECT  (col4955_4961, col4955_4957, col4955_4956, col4955_4967, col4955_4958, col4955_4963, col4955_4968, col4955_4960, col4955_4966, col4955_4962, col4955_4965) FROM (VALUES( 'BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)':::BOX2D,'24:00:00':::TIME,'SL':::STRING,'23:24:43.616041':::STRING,'23:24:43.616041':::STRING,e'\\\\x0f\\\\x1f\\\\x1b[^u\u003eg9':::STRING,1043:::OID,false,'24:00:00':::STRING,701:::OID,'23:24:43.616041':::TIME) ) AS T(col4955_4961,col4955_4956,col4955_4967,col4955_4968,col4955_4966,col4955_4965,col4955_4957,col4955_4958,col4955_4963,col4955_4960,col4955_4962) ) )\" []] :false\nQUERY [\"SELECT array[((col4955_4961, col4955_4957, col4955_4956, col4955_4967, col4955_4958, col4955_4963, col4955_4968, col4955_4960, col4955_4966, col4955_4962, col4955_4965))::STRING] FROM (VALUES('BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)':::BOX2D,'24:00:00':::TIME,'SL':::STRING,'23:24:43.616041':::STRING,'23:24:43.616041':::STRING,e'\\\\x0f\\\\x1f\\\\x1b[^u\u003eg9':::STRING,1043:::OID,false,'24:00:00':::STRING,701:::OID,'23:24:43.616041':::TIME) ) AS T(col4955_4961,col4955_4956,col4955_4967,col4955_4968,col4955_4966,col4955_4965,col4955_4957,col4955_4958,col4955_4963,col4955_4960,col4955_4962)\" []] :[[\"(\\\"BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)\\\",1043,24:00:00,SL,f,24:00:00,23:24:43.616041,701,23:24:43.616041,23:24:43.616041,\\x0f\\x1f\\x1b[^u\u003eg9)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema4574.table4955'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema4574' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4955' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema4574.table4955', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema4574'\\n\\t\\t           AND table_name = 'table4955'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema4574.table4955 (col4955_4956,col4955_4957,col4955_4958,col4955_4959,col4955_4960,col4955_4961,col4955_4962) VALUES ('24:00:00':::TIME,1043:::OID,false,1042:::OID,701:::OID,'BOX(-1.6594455775631447 -0.2781458012779048,-1.4804816641738205 0.77814576626752)':::BOX2D,'23:24:43.616041':::TIME), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "14:22:17.366748",
		  |  "ops": null,
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "WATCH DOG: failed to get session information: timeout: context canceled\n"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ c080f7ac2f5be11aedd787dceba77cca7df18a16:

		  | runtime.gopark(0xc001528f78?, 0x3?, 0x0?, 0x30?, 0xc001528f4a?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001528dc8 sp=0xc001528da8 pc=0x465216
		  | runtime.selectgo(0xc001528f78, 0xc001528f44, 0xc001372100?, 0x0, 0x3f763c0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001528f08 sp=0xc001528dc8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0028e6000, {0x550cc68, 0xc001372100})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc001528fb8 sp=0xc001528f08 pc=0x1a90b2e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc001528fe0 sp=0xc001528fb8 pc=0x1a90e2e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001528fe8 sp=0xc001528fe0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 260035 [runnable]:
		  | runtime.gopark(0xc0011767b0?, 0x2?, 0xf8?, 0x66?, 0xc0011767a4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001176630 sp=0xc001176610 pc=0x465216
		  | runtime.selectgo(0xc0011767b0, 0xc0011767a0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001176770 sp=0xc001176630 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0011767e0 sp=0xc001176770 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0011767e8 sp=0xc0011767e0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 259134 [runnable]:
		  | runtime.gopark(0xc00205af78?, 0x3?, 0x0?, 0x30?, 0xc00205af4a?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00205adc8 sp=0xc00205ada8 pc=0x465216
		  | runtime.selectgo(0xc00205af78, 0xc00205af44, 0xc001372100?, 0x0, 0x3f763c0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00205af08 sp=0xc00205adc8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002819130, {0x550cc68, 0xc001372100})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc00205afb8 sp=0xc00205af08 pc=0x1a90b2e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc00205afe0 sp=0xc00205afb8 pc=0x1a90e2e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00205afe8 sp=0xc00205afe0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | T8,e'\\\\'::CHAR),(869:::OID,'01070000A0E61000000600000001010000A0E610000078232150B11234C0F40846488B144140B4B786E7B10CF5C101010000A0E610000078A1E4D644F54640497632D536AA54C0B85290D47233F9C101060000A0E61000000200000001030000800100000009000000E87C9176AC6649C0D431A1E53C103BC0ACB25448152EFF41E06FDB2115E01CC040F727751A4AF53F062DAFC50EB9F5C140E943BF19B209C00F730C260BF749C070EACC26FB54EB4160AAF26F3C432D40786A6174DFFE35C044F2796C331EFF41301665F8A7EF564019944CD6FBC842C07A5C042F7D01F84124D3A3BA81B14C40B8D002ACA3014740B8B566EA8115FCC1686ABCBDEEE747C0E23BD2140A395540A80AD0E1E825DC41079C97DA643651C0A225A244F11C56407DD452E9D889F6C1E87C9176AC6649C0D431A1E53C103BC0ACB25448152EFF410103000080010000000800000032B54540758C5AC0BC8BCD447AEE4C40705DDA0D0FC3CCC177DA1DD2218862C0F067E87706E347405C933FDF7480EBC1389068D2F3433CC0D4C3DA891F964740DC49EAE13C12FFC1E4F95021EEF765401E8BF560FE4D564088BAC64DC9CED641DCCB8FC089D25340EEB4998DEA085640D2802FFECB47F5C120695247D37743C042C04CFFE172544090F748EE7A1AC3C1B0F9EEEF39654DC0CECF9A995E4D56404E6259E9C580E2C132B54540758C5AC0BC8BCD447AEE4C40705DDA0D0FC3CCC101050000A0E610000004000000010200008007000000E1116249ED995AC06C2C37797B5032C00015EDDF87D699C1F76E041972A358C0CE5CE13136363EC06413DC8C2405E541604CB7A392004EC03AD7415222014FC0BB872925A58700C23808E938F74C32C014B86837B2A636C0A601C783961EF4C1A8BFCCF1C8634840D1FC8CFD89DD55C016578A60BD80F2412CB633566DF45E4078923C029FCF3FC0645C76D02431F2412406A366C3245A40B0FF74BD4AB81B40C0483398517FF8C1010200008004000000208E3AC53ABD4240B8B5BE9E19A246C0EC4C5CE61F5700428AFD6CD649C56340305365F1E388314044219536AFEEFA4106FEBDD2B2E762401055A4CDA2A7444028D312E394FCF1414644628AA14E58C08C14BF046C04444094C25304632E01C2010200008002000000DA8AEEA5CBC75AC0FFDAE3C3141E4AC026389877CC61F3415EF79E2E8344524094116ABB60E853C0ECA8BBCF18ACE7C101020000800300000025E39C8DD57665C004197F8F455953C06484CADEE01DE641F0691A7BDA7F20C00059C6BE6375FC3F5419AC8345E2F5414AD33D3732965FC06263001FCC2C5240A0DF93968E7AF1C101030000A0E6100000010000000A000000E3DF28A5FC7665C0F015734266443AC03F3F36716A2102C2465C5541DA884EC0A4162F8AEC6D48C05409AC3E3F6AF841E8E9215A24FE3FC03845AC1797842AC0A09F085B1D84D8411A68250188D86540ED33C3750EB047C0D88E40E0E6AFEAC1B6BED12651FC52408CBAD1032DA54E40E437746BE17AE84190FE32FA624139400401160F5A9E5540204178D89E80BFC1D4DF5274C93D41C0E08F073E009C524012763F64F3F4F34123C6619F20D758C0C08D1051D0E55440728435B62B34F141EE7C035E16A75AC0988F9333B2115540E88B50E5EC0ED441E3DF28A5FC7665C0F015734266443AC03F3F36716A2102C201040000A0E610000002000000010100008084F1B16835C454C084D45729903835406CA136EF0E46FD410101000080EA952F8D65805EC0C844526C897F4F4058E0B141710BE041':::GEOGRAPHY,'04:23:28.782631':::TIME,'cbfc86f3-c3d9-4723-b562-4740e29fabd5':::UUID,'0d847842-566c-41dd-8183-fd1f8bdcbc36':::UUID,2187169389959324096:::INT8,B'000001101110110101000101010101010',1186:::OID,'0104000020E6100000030000000101000000081F0D0D9A4333409EAD6E073D1C424001010000001EB6F32EDC2366404C1DD30AEDAF3CC001010000007C29A7CF0DBD5140A006077214E90240':::GEOGRAPHY,'2005-11-26 15:30:46.000852+00:00':::TIMESTAMPTZ,'24:00:00-15:59:00':::TIMETZ,'63 years 3 mons 675 days 03:44:01.766508':::INTERVAL,21:::OID,ARRAY['18:00:11.435092':::TIME,'19:06:05.940257':::TIME],0.41053306360871844:::FLOAT8,'0'::CHAR),(1184:::OID,'0102000060E61000000300000040433FACE35048C0A77DD090001644C03B610381F841F0C1E3225450D5F252C00A751814F4B051C01E6E80A482A4FCC18846503C5F055440E1DE1CF9F99554C034E16C6C6E4EF241':::GEOGRAPHY,'13:59:26.56293':::TIME,'7f76bf35-399d-46b5-abd8-477ef84b9e1e':::UUID,'678d6b1d-530b-478c-94cc-4cb105e306c4':::UUID,357741825856761355:::INT8,B'011000010100011001011100000111101',1184:::OID,'01040000A0E6100000020000000101000080EC6F4C74A73964C013563C96D9764BC0646D316B7B23F7C10101000080E4D9CB19049740C02066625300B30740306D7E8A1B42CBC1':::GEOGRAPHY,'2017-07-14 08:50:06.000775+00:00':::TIMESTAMPTZ,'08:37:00.070212+02:32:00':::TIMETZ,'6 years 10 mons 213 days 07:53:00.391277':::INTERVAL,19:::OID,ARRAY['00:13:45.26256':::TIME,'13:09:40.062221':::TIME],'+Inf':::FLOAT8,'3'::CHAR), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 4dcb32c0346e20a95847763f89b9b0796d9ed4dc:

		  |
		  | goroutine 449077 [runnable]:
		  | runtime.gopark(0xc0000a7f78?, 0x3?, 0x0?, 0x30?, 0xc0000a7f4a?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0000a7dc8 sp=0xc0000a7da8 pc=0x465216
		  | runtime.selectgo(0xc0000a7f78, 0xc0000a7f44, 0xc000f02dc0?, 0x0, 0x3f769e0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0000a7f08 sp=0xc0000a7dc8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc003206190, {0x550d408, 0xc000f02dc0})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0000a7fb8 sp=0xc0000a7f08 pc=0x1a90b2e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0000a7fe0 sp=0xc0000a7fb8 pc=0x1a90e2e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000a7fe8 sp=0xc0000a7fe0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 449979 [runnable]:
		  | runtime.gopark(0xc002dcdfb0?, 0x2?, 0xf8?, 0xde?, 0xc002dcdfa4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc002dcde30 sp=0xc002dcde10 pc=0x465216
		  | runtime.selectgo(0xc002dcdfb0, 0xc002dcdfa0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc002dcdf70 sp=0xc002dcde30 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc002dcdfe0 sp=0xc002dcdf70 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc002dcdfe8 sp=0xc002dcdfe0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 450193 [runnable]:
		  | runtime.gopark(0xc0027197b0?, 0x2?, 0x60?, 0x96?, 0xc0027197a4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc002719630 sp=0xc002719610 pc=0x465216
		  | runtime.selectgo(0xc0027197b0, 0xc0027197a0, 0xc000880df0?, 0x0, 0xc00163f0e0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc002719770 sp=0xc002719630 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0027197e0 sp=0xc002719770 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0027197e8 sp=0xc0027197e0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | mons 849 days 09:07:15.24373':::INTERVAL,0.4116644884997049971:::DECIMAL,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,'1 year 7 mons 925 days 00:15:40.084379':::INTERVAL) ) AS T(col5418_5429,col5418_5425,col5418_5433,col5418_5435,col5418_5430,col5418_5421,col5418_5422,col5418_5427,col5418_5432,col5418_5431,col5418_5423)\" []] :[[\"(\\\"1 year 7 mons 925 days 00:15:40.084379\\\",\\\"2003-07-02 22:06:29.000174+00:00\\\",1560,4096,~\\x16eF,\\\"62 years 11 mons 849 days 09:07:15.24373\\\",-0.5100394236031163782,9bb933f3-8aaa-4cae-b3cf-c09fd3240b84,-0.9217039121028213753,ffffffff-ffff-ffff-ffff-ffffffffffff,\\\"-73 years -7 mons -692 days -18:26:57.373198\\\")\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema2653.table5418'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema2653' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table5418' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema2653.table5418', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema2653'\\n\\t\\t           AND table_name = 'table5418'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema2653.table5418 (col5418_5419,col5418_5420,col5418_5421,col5418_5422,col5418_5423,col5418_5424,col5418_5425,col5418_5426,col5418_5427,col5418_5428,col5418_5429,col5418_5430,col5418_5431,col5418_5432,col5418_5433,col5418_5434,col5418_5435) VALUES ('[false, [0.0335919891811407], {}, []]':::JSONB,'294276-12-31 23:59:59.999999+00:00':::TIMESTAMPTZ,2202:::OID,e'B8=\\x03\\x0e'::NAME:::NAME,'-69 years -3 mons -400 days -04:12:29.915842':::INTERVAL,'eaef:b976:d581:34d1:3971:e4d2:9e65:f096/81':::INET,'57 years 8 mons 297 days 02:46:46.541525':::INTERVAL,1042:::OID,'19 years 6 mons 383 days 16:31:51.365574':::INTERVAL,1042:::OID,'9cbebe5e-c572-43cc-8786-843e3e6c541d':::UUID,23:::OID,'266b423e-de2d-421e-aa6a-5d0dcc2c4569':::UUID,(-3.636366667979596381E+27):::DECIMAL,6053688165.121697478:::DECIMAL,700:::OID,'-2000-01-01 00:00:00+00:00':::TIMESTAMPTZ),(e'[[true, {\"LK##HTIVF\": {\"PbYyY\u003csM\": 1.1330497642449417}}, {\"9W[M\\'\": {\"foobar\": []}, \"}8Uz;X[K\": null}, \"N_YsP\", true, [], {}], false]':::JSONB,NULL,4096:::OID,e'~\\x16eF'::NAME:::NAME,'1 year 7 mons 925 days 00:15:40.084379':::INTERVAL,'213.191.185.165/16':::INET,'-73 years -7 mons -692 days -18:26:57.373198':::INTERVAL,90004:::OID,'62 years 11 mons 849 days 09:07:15.24373':::INTERVAL,1560:::OID,'9bb933f3-8aaa-4cae-b3cf-c09fd3240b84':::UUID,1560:::OID,'ffffffff-ffff-ffff-ffff-ffffffffffff':::UUID,0.4116644884997049971:::DECIMAL,(-0.9217039121028213753):::DECIMAL,1009:::OID,'2003-07-02 22:06:29.000174+00:00':::TIMESTAMPTZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ b0f13c9bbef3e6628471d887672be7c7658f6511:

		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 258009 [runnable]:
		  | runtime.gopark(0xc001883fb0?, 0x2?, 0xf8?, 0x3e?, 0xc001883fa4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc001883e30 sp=0xc001883e10 pc=0x465216
		  | runtime.selectgo(0xc001883fb0, 0xc001883fa0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc001883f70 sp=0xc001883e30 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc001883fe0 sp=0xc001883f70 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc001883fe8 sp=0xc001883fe0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 257938 [runnable]:
		  | runtime.gopark(0xc00159efb0?, 0x2?, 0xc7?, 0x52?, 0xc00159efa4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00159ee30 sp=0xc00159ee10 pc=0x465216
		  | runtime.selectgo(0xc00159efb0, 0xc00159efa0, 0xc0009a6e70?, 0x0, 0xc00010ff20?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00159ef70 sp=0xc00159ee30 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00159efe0 sp=0xc00159ef70 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00159efe8 sp=0xc00159efe0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 257988 [runnable]:
		  | runtime.gopark(0xc00106e7b0?, 0x2?, 0xf8?, 0xe6?, 0xc00106e7a4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc00106e630 sp=0xc00106e610 pc=0x465216
		  | runtime.selectgo(0xc00106e7b0, 0xc00106e7a0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc00106e770 sp=0xc00106e630 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00106e7e0 sp=0xc00106e770 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00106e7e8 sp=0xc00106e7e0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | t\\t\\t\\t\\t\\t\\t\\t    table_name='table5153' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema3739.table5153', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema3739'\\n\\t\\t           AND table_name = 'table5153'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n===========================Previous statements [QUERY: DROP TABLE schema4495.table4674 RESTRICT, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema2742.table5048 (col5048_5049,col5048_5050,col5048_5051,col5048_5052,col5048_5053,col5048_5054,col5048_5055,col5048_5056,col5048_5057,col5048_5058,col5048_5059,col5048_5060,col5048_5061,col5048_5062,col5048_5063,col5048_5064) VALUES (NULL,'2008-01-09 15:17:45.000867+00:00':::TIMESTAMPTZ,0.05700961500406265:::FLOAT8,ARRAY[e'\\x00':::STRING,e'\\t\\n9f*FhC':::STRING,e'w:\\x18T\\x1a\\x0b\\r\\x05':::STRING],'0101000040000000000000F87F000000000000F87F000000000000F87F':::GEOMETRY,2205:::OID,'02184a97-839b-42c6-a437-1bf3480111e5':::UUID,'8685884d-a646-4f69-b124-2e25f75d7f43':::UUID,90000:::OID,'1977-01-28':::DATE,'24:00:00':::TIME,''::VARCHAR,'1970-08-30':::DATE,'2012-12-31 18:18:56.000849+00:00':::TIMESTAMPTZ,(-2.1002867221832275):::FLOAT8,3802:::OID), Expected Errors: , Potential Errors:  QUERY: CREATE SCHEMA schema5287 AUTHORIZATION root, Expected Errors: , Potential Errors:  QUERY: ALTER SEQUENCE schema3739.seq5207 RENAME TO schema3739.seq5207, Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema3739.table5153 (col5153_5155,col5153_5156,col5153_5157,col5153_5158,col5153_5159,col5153_5160,col5153_5161,col5153_5162,col5153_5163,col5153_5164,col5153_5165,col5153_5166,col5153_5167,col5153_5168) VALUES ('55 years 11 mons 138 days 06:30:45.120108':::INTERVAL,'1975-06-01':::DATE,1.0330995321273804:::FLOAT8,B'11100001111',1894786193:::INT8,'BOX(-1.768983998993107 -0.17063296603575273,0.7357922491711375 0.4416433630277291)':::BOX2D,'}'::\"char\",'2482b987-b89c-4be7-8143-8a4ab54121de':::UUID,1266:::OID,90002:::OID,90000:::OID,ARRAY[e'\\'':::STRING,e'\\x0f':::STRING,'R':::STRING],3802:::OID,'82 years 222 days 04:51:46.34076':::INTERVAL),('-52 years -8 mons -31 days -24:40:02.464009':::INTERVAL,'1991-10-29':::DATE,(-0.11074919253587723):::FLOAT8,B'10011110011',1806223354:::INT8,'BOX(-0.7807303332530433 0.3384270663732014,-0.12424586671507853 1.2347847070465505)':::BOX2D,'X'::\"char\",'9b060de9-0564-464c-b1b2-26083a979845':::UUID,21:::OID,1082:::OID,1082:::OID,ARRAY[e'\\n':::STRING,'A':::STRING,'t':::STRING,e'\\x1a':::STRING],1266:::OID,'-25 years -8 mons -376 days -08:33:38.118848':::INTERVAL),('52 years 6 mons 661 days 18:09:36.704261':::INTERVAL,'1981-09-06':::DATE,(-0.25688132643699646):::FLOAT8,B'00010000100',(-232383820):::INT8,'BOX(-1.509601589655769 -0.1631420385226183,0.20403427394885937 -0.07889506018150583)':::BOX2D,';'::\"char\",'8f8b2e9f-66dc-4c88-b021-23aab76f63d3':::UUID,18:::OID,26:::OID,4089:::OID,ARRAY[',':::STRING,'}':::STRING,e'\\x0b':::STRING],17:::OID,'-32 years -359 days -18:18:34.919879':::INTERVAL), Expected Errors: , Potential Errors: ]: ERROR: schema \"schema5287\" (1265): expected matching namespace entry, found none (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ e5917605f751c569ddd90062250a40cafd7ccf30:

		  |  1449.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1450.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1450.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1450.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1450.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1451.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1451.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1451.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1451.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1452.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1452.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1452.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1452.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1453.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1453.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1453.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1453.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1454.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1454.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1454.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1454.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1455.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1455.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1455.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1455.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1456.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1456.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1456.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1456.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1457.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  | _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  |  1457.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1457.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1457.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1458.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1458.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1458.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1458.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		  |  1459.0s        0            0.0            4.4      0.0      0.0      0.0      0.0 opOk
		  |  1459.0s        0            0.0            0.7      0.0      0.0      0.0      0.0 txnCmtErr
		  |  1459.0s        0            0.0            0.5      0.0      0.0      0.0      0.0 txnOk
		  |  1459.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 txnRbk
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 2372698da1dfacb90f60c6a63f2c1298d1db16b8:

		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00060f7e8 sp=0xc00060f7e0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 230458 [runnable]:
		  | github.com/jackc/puddle.(*Resource).Destroy.func1()
		  | 	github.com/jackc/puddle/external/com_github_jackc_puddle/pool.go:71 fp=0xc0012edfe0 sp=0xc0012edfd8 pc=0xe0d840
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0012edfe8 sp=0xc0012edfe0 pc=0x495701
		  | created by github.com/jackc/puddle.(*Resource).Destroy
		  | 	github.com/jackc/puddle/external/com_github_jackc_puddle/pool.go:71 +0x85
		  |
		  | goroutine 230161 [runnable]:
		  | runtime.gopark(0xc0022d1778?, 0x3?, 0x0?, 0x30?, 0xc0022d174a?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0022d15c8 sp=0xc0022d15a8 pc=0x465216
		  | runtime.selectgo(0xc0022d1778, 0xc0022d1744, 0xc002250be8?, 0x0, 0x3f7a6a0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0022d1708 sp=0xc0022d15c8 pc=0x474cf2
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002cabe00, {0x5510e28, 0xc0010ea500})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0022d17b8 sp=0xc0022d1708 pc=0x1a91e6e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0022d17e0 sp=0xc0022d17b8 pc=0x1a9216e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0022d17e8 sp=0xc0022d17e0 pc=0x495701
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 230409 [runnable]:
		  | runtime.gopark(0xc0014b27b0?, 0x2?, 0xf8?, 0x26?, 0xc0014b27a4?)
		  | 	GOROOT/src/runtime/proc.go:361 +0xd6 fp=0xc0014b2630 sp=0xc0014b2610 pc=0x465216
		  | runtime.selectgo(0xc0014b27b0, 0xc0014b27a0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x772 fp=0xc0014b2770 sp=0xc0014b2630 pc=0x474cf2
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0014b27e0 sp=0xc0014b2770 pc=0xd2f2ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0014b27e8 sp=0xc0014b27e0 pc=0x495701
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | \n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema6065.table6545\"]] :[]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema6065.table6545'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema6065' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table6545' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema6065.table6545', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema6065'\\n\\t\\t           AND table_name = 'table6545'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema6065.table6545 (col6545_6546,col6545_6547,col6545_6548,col6545_6549,col6545_6550,col6545_6551,col6545_6552,col6545_6553,col6545_6554,col6545_6555,col6545_6556,col6545_6557,col6545_6558,col6545_6559,col6545_6560,col6545_6561,col6545_6562,col6545_6563,col6545_6564) VALUES (4267112677082135895:::INT8,NULL,1700:::OID,'1ac1daf3-3b06-4fcd-9f6f-cb77735bd1cb':::UUID,19:::OID,(-2147483648):::INT8,4096:::OID,'1984-01-05':::DATE,'00:48:01.211766':::TIME,B'1111101010110111111110111101010010111001001',1043:::OID,'1993-07-04':::DATE,'871c:215f:fc2c:6b6c:5e2e:67e0:367f:ab78/73':::INET,'0'::\"char\",B'101101000111000111100000010','1975-03-15':::DATE,e'o\\x02'::STRING,(-0.5978192090988159):::FLOAT8,'[null, \"O``]@\", [{\"bar\": 1.5328551685767675}], null]':::JSONB), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 8, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 34dc56fbb5789b39be47b110bf22332c7f5654f6:

		  | runtime.gopark(0xc0018dafb0?, 0x2?, 0x18?, 0x0?, 0xc0018dafa4?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc0018dae30 sp=0xc0018dae10 pc=0x466c16
		  | runtime.selectgo(0xc0018dafb0, 0xc0018dafa0, 0xc000a106b0?, 0x0, 0xc002d0b200?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc0018daf70 sp=0xc0018dae30 pc=0x476adc
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc0018dafe0 sp=0xc0018daf70 pc=0xd497ac
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0018dafe8 sp=0xc0018dafe0 pc=0x4981e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 427855 [runnable]:
		  | runtime.gopark(0xc0027b6f78?, 0x3?, 0x0?, 0x30?, 0xc0027b6f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc0027b6dc8 sp=0xc0027b6da8 pc=0x466c16
		  | runtime.selectgo(0xc0027b6f78, 0xc0027b6f44, 0xc001316c40?, 0x0, 0x3fea260?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc0027b6f08 sp=0xc0027b6dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002b76af0, {0x5b894a8, 0xc001316c40})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0027b6fb8 sp=0xc0027b6f08 pc=0x1ac236e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0027b6fe0 sp=0xc0027b6fb8 pc=0x1ac266e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0027b6fe8 sp=0xc0027b6fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 424696 [runnable]:
		  | runtime.gopark(0xc003069f78?, 0x3?, 0x0?, 0x30?, 0xc003069f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc003069dc8 sp=0xc003069da8 pc=0x466c16
		  | runtime.selectgo(0xc003069f78, 0xc003069f44, 0xc001316c40?, 0x0, 0x3fea260?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc003069f08 sp=0xc003069dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002a51e50, {0x5b894a8, 0xc001316c40})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc003069fb8 sp=0xc003069f08 pc=0x1ac236e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc003069fe0 sp=0xc003069fb8 pc=0x1ac266e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc003069fe8 sp=0xc003069fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | a)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: CREATE TYPE schema4777.enum5397 AS ENUM ('sfaa', 'a'), Expected Errors: , Potential Errors:  QUERY: SELECT t0.col4619_4632 AS col0,t1.col4619_4627 AS col1,t1.col4619_4631 AS col2,t0.col4619_4625 AS col3,t1.col4619_4632 AS col4,t1.col4619_4621 AS col5,t0.col4619_4622 AS col6,t1.col4619_4622 AS col7,t0.col4619_4628 AS col8,t0.col4619_4629 AS col9,t1.col4619_4628 AS col10,t1.col4619_4624 AS col11,t1.col4619_4632 AS col12,t0.col4619_4630 AS col13 FROM public.table4619 AS t0 ,public.table4619 AS t1  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200 QUERY: INSERT INTO public.table4619 (col4619_4620,col4619_4621,col4619_4622,col4619_4623,col4619_4624,col4619_4625,col4619_4626,col4619_4627,col4619_4628) VALUES (25:::OID,'0101000020E6100000000000000000F87F000000000000F87F':::GEOGRAPHY,'1998-07-03 06:57:55.000061':::TIMESTAMP,'010700000009000000010200000003000000006AB210FBD9A1C16A0309C8A156F1C100A4999BD3ED6CC1D8A03B3B0754D04130AA9C5A9262E7413003CEBF50C0EC410105000000040000000102000000020000005891CBEC522C0042AA89E6845C3FF141C0097D6A588ABA4164FDCDB58002F6410102000000050000000005C41313C9FB411414D37B9568E441B0965D019670C841F88C8795252B00426B91038BF792F3C180BA7B674F0A00429E7238ECF75AF3C182AEFCD973DAF8418E472EE38EF1F1C1C4324416BAB3EE4101020000000200000070690B698ABFDA417C74C5405B26EE4168A1AEA7EE49D9C1204DEB69BC37EE410102000000050000003613DB1428E8F8411512ADD4CA9202C2F249189E3960F9411E23B6C4BE34FAC1C48453DEB61EF8413046510B222CCBC1B8CDAB68367EE64140D46447337DA4C1F246506D54F100C29084848E5CFEE9410105000000050000000102000000070000000BF5E861BBBCF4C10E5E4FBA6EC3F7C18FC6BBA428C9F1C1F553EBA80291FCC1C83830AD0CFDD04134CF09B2BE3DEBC1A6BA14DFB5680042389F33E55A96F7C142EC571F8F3B01424CF765B43ABAF2C18E6CC796B038E0C1A0A4F0582E8FE9419AE19A17E42DE4C1943594FA426DE64101020000000200000010FACC7DF544FD417622AC1B6C89EDC1FCB4C5E4AF02DCC178812FC41161F2410102000000020000004E1377382167E6C1525CEA1FD484FBC16C13BB178E18ED41CB57E118F089F8C101020000000600000066E493358EA0F841B65ED0E6040D01C284D1B5CDAFBCFD41B0B06CC8FEC7C34164B4BAC713CFEA41D03D14D1578DCF41042AFE9579460042DA5BEA776F82F14150528FE2D8CFD2C1680114FDFA69FA41082D3AC2486FE0C1F08700ACCF93DE41010200000002000000F48F3B67802EE341680956FEF7F7E94100BEDED08AB3D1C130AF1B2E1B9AC24101010000001E2AB5601B86F94186A2585BF4310142010600000002000000010300000001000000090000004B284E13A309F8C170516A5D91ECE5C19583E59FC4ADF6C12FCE2A1CF8C4FEC1ACBD0890EBFFEA417FDC908D5C5C01C2EC1FACF230DAF3410EA60490CF3DFBC1B800764FB3F6F7413216A94434E4F3C164F403F73B7F0042D009C361A04B024228F4C4F91F53D0C1009B833CF98285C164EC198FD34CF2C1C0D4E7FF834FA7414B284E13A309F8C170516A5D91ECE5C10103000000010000000D000000AE4B0892EF35FBC1789B8F6BE394FFC1D8A430EAF7C8E5C168B158DA1A35F6C10C44D8EC0760EA41E3256097CD87F0C15020361717AEFC41E02CCB55B68802C28A3504909E51F141407C04740F01EFC12AC02A36BAAFFA4181770181171EF5C1467EE9AB623DF2417E39485F732AE7C196B93D6C90160142002088331201AAC1687A363978B5E74106D5218CCAABF041800B7E7DC049AE4150B4011DFFABF44197E4E8F6425AFEC1CCB6E05FDC5400423BBAFF9446D9F8C1E602B836DE62F941AE4B0892EF35FBC1789B8F6BE394FFC1010300000001000000050000009C1B0F0DAE68FBC1102475BEBEADD4C1021FFB297D06F841B45D8E49A5C4EC41C063143CCC8FC4C108B2631DF195F341B001231A8FCADBC1ECBAE551ED47EE419C1B0F0DAE68FBC1102475BEBEADD4C101010000003EE829BCBE8BF14118E758ED32A5D041010400000006000000010100000080131776790CE041840C56DD8584E5C1010100000038F66EE50926E4C1685CFF8FC188F8410101000000E83453E24C61EA4148F8FC441A79EA410101000000D4B07E5CF498D6C1387C762A2F0DE0410101000000AA560FE68B44F8413E3EF85AEA1F02C201010000004AC6B4B30ACB01C2145BFD169527E3410101000000A08AAB6A6431BFC15AC8A336F782F341':::GEOMETRY,2202:::OID,1043:::OID,2205:::OID,(-2168):::INT8,(-700623831):::INT8), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ a0bfa6dafcc206301d3a21887c374db63b377075:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:165,schemachange_random_load.go:59,test_runner.go:928: output in run_142521.580064015_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:1974
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:162
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:59
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:928
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) output in run_142521.580064015_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 84)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149)" []] :false
		  | QUERY ["SELECT COUNT (*) > 0 FROM (SELECT * FROM public.table5138 WHERE (col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152)= ( SELECT  (col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152) FROM (VALUES( 'Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149) ) )" []] :false
		  | QUERY ["SELECT array[((col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152))::STRING] FROM (VALUES('Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149)" []] :[["(\"80 years 323 days 16:47:01.421334\",01:15:32.898716+10:48:00,\"\",hn,\"box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\",869,\"BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\",Hn)"]]
		  | QUERY ["\n\t\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\n\t\t  FROM (\n\t\t        SELECT conname, conkey, confkey, conrelid, confrelid\n\t\t          FROM pg_constraint\n\t\t         WHERE contype = 'f'\n\t\t           AND conrelid = 'public.table5138'::REGCLASS::INT8\n\t\t       ) AS con\n\t\t\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \n\t\t\t\t      WHERE table_schema ='public' AND\n\t\t\t\t\t\t\t\t    table_name='table5138' AND\n\t\t\t\t\t\t\t\t\t\t(crdb_internal.is_constraint_active('public.table5138', constraint_name) = true)\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\n\t\t  JOIN (\n\t\t        SELECT column_name, ordinal_position, column_default\n\t\t          FROM information_schema.columns\n\t\t         WHERE table_schema = 'public'\n\t\t           AND table_name = 'table5138'\n\t\t       ) AS child ON conkey[1] = child.ordinal_position\n\t\t  JOIN (\n\t\t        SELECT pc.oid,\n\t\t               cols.table_schema,\n\t\t               cols.table_name,\n\t\t               cols.column_name,\n\t\t               cols.ordinal_position\n\t\t          FROM pg_class AS pc\n\t\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\n\t\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\n\t\t       ) AS parent ON (\n\t\t                       con.confkey[1] = parent.ordinal_position\n\t\t                       AND con.confrelid = parent.oid\n\t\t                      )\n\t\t WHERE child.column_name != 'rowid';\n" []] :[]
		  |
		  | ===========================
		  | Previous statements [QUERY: INSERT INTO public.table5138 (col5138_5149,col5138_5150,col5138_5151,col5138_5152,col5138_5153,col5138_5154) VALUES ('BOX(-1.3510600150968646 0.7064702248189542,-0.2299858840764486 1.4366532836712582)':::BOX2D,'55 years 8 mons 65 days 07:33:21.024335':::INTERVAL,1560:::OID,'P'::STRING,NULL,'11:49:02.79375-01:04:00':::TIMETZ),('BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D,'80 years 323 days 16:47:01.421334':::INTERVAL,869:::OID,'Hn'::STRING,1161285865:::INT8,'01:15:32.898716+10:48:00':::TIMETZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 4, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152) FROM (VALUES( 'Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149) ) )\" []] :false\nQUERY [\"SELECT array[((col5138_5150, col5138_5154, col5138_5156, col5138_5155, col5138_5157, col5138_5151, col5138_5149, col5138_5152))::STRING] FROM (VALUES('Hn'::STRING,'80 years 323 days 16:47:01.421334':::INTERVAL,'01:15:32.898716+10:48:00':::TIMETZ,'':::STRING,'hn':::STRING,'box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::STRING,869:::OID,'BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D) ) AS T(col5138_5152,col5138_5150,col5138_5154,col5138_5156,col5138_5155,col5138_5157,col5138_5151,col5138_5149)\" []] :[[\"(\\\"80 years 323 days 16:47:01.421334\\\",01:15:32.898716+10:48:00,\\\"\\\",hn,\\\"box(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\\\",869,\\\"BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)\\\",Hn)\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table5138'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table5138' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table5138', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table5138'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO public.table5138 (col5138_5149,col5138_5150,col5138_5151,col5138_5152,col5138_5153,col5138_5154) VALUES ('BOX(-1.3510600150968646 0.7064702248189542,-0.2299858840764486 1.4366532836712582)':::BOX2D,'55 years 8 mons 65 days 07:33:21.024335':::INTERVAL,1560:::OID,'P'::STRING,NULL,'11:49:02.79375-01:04:00':::TIMETZ),('BOX(-0.6887541798148256 0.024416595305017497,-0.5146334906409737 0.2511425777080384)':::BOX2D,'80 years 323 days 16:47:01.421334':::INTERVAL,869:::OID,'Hn'::STRING,1161285865:::INT8,'01:15:32.898716+10:48:00':::TIMETZ), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		  | {
		  |  "workerId": 0,
		  |  "clientTimestamp": "14:35:55.508552",
		  |  "ops": null,
		  |  "expectedExecErrors": "",
		  |  "expectedCommitErrors": "",
		  |  "message": "WATCH DOG: failed to get session information: timeout: context canceled\n"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 4780ee15194e7d22b661ac92254b13cd2b71dcb1:

		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 232326 [runnable]:
		  | runtime.gopark(0xc001389fb0?, 0x2?, 0x20?, 0x0?, 0xc001389fa4?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc001389e30 sp=0xc001389e10 pc=0x466c16
		  | runtime.selectgo(0xc001389fb0, 0xc001389fa0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc001389f70 sp=0xc001389e30 pc=0x476adc
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc001389fe0 sp=0xc001389f70 pc=0xd4cccc
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001389fe8 sp=0xc001389fe0 pc=0x4981e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 230672 [runnable]:
		  | runtime.gopark(0xc002393fb0?, 0x2?, 0x60?, 0x3e?, 0xc002393fa4?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc002393e30 sp=0xc002393e10 pc=0x466c16
		  | runtime.selectgo(0xc002393fb0, 0xc002393fa0, 0xc001bdb130?, 0x0, 0xc001860ba0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc002393f70 sp=0xc002393e30 pc=0x476adc
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc002393fe0 sp=0xc002393f70 pc=0xd4cccc
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002393fe8 sp=0xc002393fe0 pc=0x4981e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 232338 [runnable]:
		  | runtime.gopark(0xc00145ffb0?, 0x2?, 0xc7?, 0x6c?, 0xc00145ffa4?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc00145fe30 sp=0xc00145fe10 pc=0x466c16
		  | runtime.selectgo(0xc00145ffb0, 0xc00145ffa0, 0xc001f5aae0?, 0x0, 0xc0012cdd40?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc00145ff70 sp=0xc00145fe30 pc=0x476adc
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc00145ffe0 sp=0xc00145ff70 pc=0xd4cccc
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00145ffe8 sp=0xc00145ffe0 pc=0x4981e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 65E135D40178A3233055953C0B848CD97C08ED1C1026C36145ABEF9412EF91B6C14BB6440AA4D00B0E7E350400C490929713FEDC13A92FBDB59E6F9C1743B79B76EA54640C43C7A4551204F4080C15D693899D04124A1B9498090F14180DF5EC3C57E09C06CAE2A4154403D40685BABB6A800D2C182174226780DFFC1506B6E4D10EA41C05A7E61D9D06146C0780CD2B02B47FAC140A105101EB6A3C101030000C0010000000900000094E2BAFF40895FC09183FB929C6546C00027B0B84E5D8D41A23FD5BEFFABE7C1C0FD9D8D91E343C070BBF0B1CD253CC0B03AB2D52084F441D0C3D5D4E9EFCE41208459F8257F2D40D80A19E6592536C0C6F8C38B407BF3C160F6267381F2B9C17043A00C32F54840A226FD55D9AF52C06081D8923CACC7C136DE3735D11AE6C1B4F845C8099365403438C48C69DD354000631E1C0653F1418AA61C0E5A3301C26AA77736C06B64403C27A47DBD0739404A3F369B59E400C29C303FFD0D95FE410EA990A90E0550C0D22DDA7902544140C889712BAB66D3C166A23EC9AFCCF541962A6CCC75FD5DC0C491DE1A666F424050605649B903D5C148C0B40FD594D84194E2BAFF40895FC09183FB929C6546C00027B0B84E5D8D41A23FD5BEFFABE7C101030000C0010000000B000000C48BABA1144162C0BA60FB28F08B54C01CCBD8DFE6E3EC4140238F005289E741568ABAD387C75FC02C0491A0C8A655C0D0B13DCFAC61F941B45A55F1A66CD7C1809BB9C4444017C0E1331EFB4A9443C040C45114C7B4F7C1A4F9DEE40280FF4120A7D257698F3840FA445DC059E63FC0D0BFB64EB37DE8C1E051EA71165BC2419222AC5EBD4450401C530F598DDC49C0D8EF42778BF0FE4122F4DD1E29C3ECC15C7A1C7A57FA6540908FB2D4C01D2DC0FA7A307407B80142A8A7745D4E3CE741BCC7A46351D65240187417EB251D35401E621AC592C1E7C115FC8CDB938CF8C1981F17BF20755A4018638DAFFE7153401E300FEA8FB8F041ACB84DA34B930142C054648E886C0CC028C7C9EEB4592140A4BD39E93227E3416858A8140F87D0C1985E7285C66145C0B49B650ED29D3040787B93FF3CADDBC1C626F238E3540242C48BABA1144162C0BA60FB28F08B54C01CCBD8DFE6E3EC4140238F005289E74101030000C0010000000C000000AB48115BB4155DC0ACC38146D44430C06CC65C04A36CDEC135A9536A3131FAC1E2156CBA58B560C0B4E706022F5C50C05CBBE203E281E0C15200B1C34E3700C29C4CFBAC2A4D45C0F0FC1D75FB2534C0CC26EAEC912BEAC1849567D3E3DF00C24E03E6D7140750C01081EF27955656C0FE70CD6D995BE6C1F84B6A3440AAE4410C0C555B398D5B404EF3A2A6D4B646C0E074135D0426EDC1403C98342CB9BB41B8C986AF6FFA5C4080317B74C70EFE3FE0245129CF3CD7C153061EB9FB0401C26A49114124A96240C8EABC05A3B355402063D7296F3DF541D0A2822371CFCEC1E864ECF7ABCF354074499423DDAE4340E0B40CEC0A36D24188DEEC0D62000042CCFA82C1156752C0E4E252D06BCA4840AFC962FD60C2F1C16CA4CE940227F0C1005072F73AE644C0CC81AE89E5623640FEF4C1B8E53D00C2F0C077322269C0C1FCEF3DA674E860C0885C9CECFC605340C08D392DC961A0C10058B1328040DE41AB48115BB4155DC0ACC38146D44430C06CC65C04A36CDEC135A9536A3131FAC101030000C001000000090000004EABFCFDEEA258C046356B07222A50C0B002AD6C5DB4E4411F08114FFCF7F2C19019CF1399F242C0D1CCB2C528DF49C03EE4DC8E76D50042CEFBF0753BBB0142F4C7A66FB68A5940F9D0CCD0C34146C066B3D49D017E00422C0E6CE17C47F2C14C62501388005B4073C69282802F43C08AFE72B550F0F7418F484B196D4E00C24020FD0C07A35940BCD4CCEA488B30C0E0498F5DE69DFB4190F9CEE3E483FA4180DEF831A45A654098790C2B5916444041A880B5055700C2DE7AD1B75C9AF3C15682667ED85C4FC0ECE480D80D164E402C67CE3D6242E64160EC45DFF2DEE041CCE48696E8B661C0E8E0F38958124340205BE6B49A4FD441B140E0738977FFC14EABFCFDEEA258C046356B07222A50C0B002AD6C5DB4E4411F08114FFCF7F2C101030000E0E6100000010000000C000000039774A7310566C080216A4A1D6451C06060E026E548C9C1523812EDA127014210A5593F645A54C07CEA84F2971748C0E04A242A3B94CCC124728F3BD340F841D64158F6B1FE51C00CD09121E98E47C08C264B2791BAFF4134501BEB005BFE41102330B14C5436C05086FEA32F9A51C0FCB58B98FCBCF241005CDE5FE025CE41FC5D711A2C625740A2CED6577F0856C0488D21B196BFE2C166F0FC5E3063F4C15CB1302878DC454047DF5581F70445C0710D3718E93AF0C108F86DAF5F810242CE9219757FFD634064133CA6E5EF3AC01E7A60251983E3C1F807D79B14B7D0C1A8F25886B9615740AC3F852DFF0353407E1855136F92F74154C1CA069A4DFB41585BF0CEB17837C050B9D19F1E694A401C9CEC0F92E8F84159445F3FDE0A00C2F8FEBB23821A50C03C1F962277C94D40D0AC68C67B23FD41C85140B5DD2BDFC1C3AD8A8CC4D257C0341EA4AB255E474000453DACA899B8C140746A926C0BA5C1039774A7310566C080216A4A1D6451C06060E026E548C9C1523812EDA1270142':::GEOGRAPHY,'1976-08-31 06:38:20.000497+00:00':::TIMESTAMPTZ,'0101000020E610000058006723731F6340141D3A6807BF4A40':::GEOGRAPHY,B'0001',700:::OID), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 4780ee15194e7d22b661ac92254b13cd2b71dcb1:

		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).isConnectionActive(0xc00295ef50, {0x5b9b188, 0xc001f28700})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:70 +0x2c5 fp=0xc0026ccf08 sp=0xc0026cce50 pc=0x1aca665
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00295ef50, {0x5b9b188, 0xc001f28700})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:100 +0x110 fp=0xc0026ccfb8 sp=0xc0026ccf08 pc=0x1aca8f0
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0026ccfe0 sp=0xc0026ccfb8 pc=0x1acabce
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0026ccfe8 sp=0xc0026ccfe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 361790 [runnable]:
		  | runtime.gopark(0xc0012d9f78?, 0x3?, 0x0?, 0x30?, 0xc0012d9f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc0012d9dc8 sp=0xc0012d9da8 pc=0x466c16
		  | runtime.selectgo(0xc0012d9f78, 0xc0012d9f44, 0xc001f28700?, 0x0, 0x3ff6880?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc0012d9f08 sp=0xc0012d9dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc002a58f00, {0x5b9b188, 0xc001f28700})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc0012d9fb8 sp=0xc0012d9f08 pc=0x1aca8ce
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc0012d9fe0 sp=0xc0012d9fb8 pc=0x1acabce
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0012d9fe8 sp=0xc0012d9fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 362698 [runnable]:
		  | runtime.gopark(0xc000ee4f78?, 0x3?, 0x0?, 0x30?, 0xc000ee4f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc000ee4dc8 sp=0xc000ee4da8 pc=0x466c16
		  | runtime.selectgo(0xc000ee4f78, 0xc000ee4f44, 0xc001f28700?, 0x0, 0x3ff6880?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc000ee4f08 sp=0xc000ee4dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc001d4d130, {0x5b9b188, 0xc001f28700})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000ee4fb8 sp=0xc000ee4f08 pc=0x1aca8ce
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000ee4fe0 sp=0xc000ee4fb8 pc=0x1acabce
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000ee4fe8 sp=0xc000ee4fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | lse\nQUERY [\"SELECT array[((col4667_4670))::STRING] FROM (VALUES(1560:::OID) ) AS T(col4667_4670)\" []] :[[\"1560\"]]\nQUERY [\"SELECT num_nulls(col4667_4669, col4667_4670, col4667_4671) \u003e 0 FROM (VALUES(1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4669,col4667_4670,col4667_4671)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema2196.table4667 WHERE (col4667_4669, col4667_4670, col4667_4671)= ( SELECT  (col4667_4669, col4667_4670, col4667_4671) FROM (VALUES( 1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4669,col4667_4670,col4667_4671) ) )\" []] :false\nQUERY [\"SELECT array[((col4667_4669, col4667_4670, col4667_4671))::STRING] FROM (VALUES(1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4669,col4667_4670,col4667_4671)\" []] :[[\"(1042,1560,ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff)\"]]\nQUERY [\"SELECT num_nulls(col4667_4671) \u003e 0 FROM (VALUES('ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4671)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema2196.table4667 WHERE (col4667_4671)= ( SELECT  (col4667_4671) FROM (VALUES( 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET) ) AS T(col4667_4671) ) )\" []] :true\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema2196.table4667'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema2196' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4667' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema2196.table4667', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema2196'\\n\\t\\t           AND table_name = 'table4667'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema2196.table4667 (col4667_4668,col4667_4669,col4667_4670,col4667_4671) VALUES ('21:01:08.892855':::TIME,26:::OID,2202:::OID,'4da9:7de5:f95b:48b1:2cb4:6d67:91b5:511a/58':::INET), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema2196.table4667 (col4667_4668,col4667_4669,col4667_4670,col4667_4671) VALUES ('19:08:25.259379':::TIME,26:::OID,25:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET), Expected Errors: , Potential Errors:  QUERY: INSERT INTO schema2196.table4667 (col4667_4668,col4667_4669,col4667_4670,col4667_4671) VALUES ('15:48:16.396418':::TIME,700:::OID,2950:::OID,'5d9c:87cb:d9f2:fa54:e793:78c3:5a6c:9c80/1':::INET),('23:30:34.155906':::TIME,1042:::OID,1560:::OID,'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff':::INET),('06:04:28.279302':::TIME,1560:::OID,19:::OID,'dabd:5b8d:5c07:c150:7a3e:a601:16e6:6007/56':::INET), Expected Errors: 23505, Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ e786cba2b137a671d3846cf7a33e7b9dea2854e6:

		  | goroutine 301350 [runnable]:
		  | runtime.gopark(0xc001dcafb0?, 0x2?, 0x0?, 0x0?, 0xc001dcafa4?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc001dcae30 sp=0xc001dcae10 pc=0x466c16
		  | runtime.selectgo(0xc001dcafb0, 0xc001dcafa0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc001dcaf70 sp=0xc001dcae30 pc=0x476adc
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc001dcafe0 sp=0xc001dcaf70 pc=0xd5000c
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001dcafe8 sp=0xc001dcafe0 pc=0x4981e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | goroutine 302286 [runnable]:
		  | runtime.gopark(0xc001547f78?, 0x3?, 0x0?, 0x30?, 0xc001547f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc001547dc8 sp=0xc001547da8 pc=0x466c16
		  | runtime.selectgo(0xc001547f78, 0xc001547f44, 0xc000fd6568?, 0x0, 0x4012820?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc001547f08 sp=0xc001547dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00252b630, {0x5bb8328, 0xc0011ae080})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc001547fb8 sp=0xc001547f08 pc=0x1acf4ae
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc001547fe0 sp=0xc001547fb8 pc=0x1acf7ae
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001547fe8 sp=0xc001547fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  | panic: dumping stacks, we failed to terminate threads on time.
		  |
		  | goroutine 302420 [running]:
		  | panic({0x397efe0, 0x5b80860})
		  | 	GOROOT/src/runtime/panic.go:987 +0x3ba fp=0xc000f06f08 sp=0xc000f06e48 pc=0x463bba
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00274a5a0, {0x5bb8328, 0xc0011ae080})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:96 +0x175 fp=0xc000f06fb8 sp=0xc000f06f08 pc=0x1acf535
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000f06fe0 sp=0xc000f06fb8 pc=0x1acf7ae
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000f06fe8 sp=0xc000f06fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | IMESTAMP)) AS t(col4894_4895,col4894_4896,col4894_4897,col4894_4898,col4894_4899,col4894_4900);\" []] :[[\"\\bGEWY\"]]\nQUERY [\"SELECT num_nulls(col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897) \u003e 0 FROM (VALUES(1043:::OID,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'-2.4307332038879395'::FLOAT8,'1990-10-20 01:57:42.00058':::TIMESTAMP,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL) ) AS T(col4894_4896,col4894_4899,col4894_4904,col4894_4900,col4894_4897)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM schema3925.table4894 WHERE (col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897)= ( SELECT  (col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897) FROM (VALUES( 1043:::OID,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'-2.4307332038879395'::FLOAT8,'1990-10-20 01:57:42.00058':::TIMESTAMP,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL) ) AS T(col4894_4896,col4894_4899,col4894_4904,col4894_4900,col4894_4897) ) )\" []] :false\nQUERY [\"SELECT array[((col4894_4896, col4894_4899, col4894_4904, col4894_4900, col4894_4897))::STRING] FROM (VALUES(1043:::OID,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'-2.4307332038879395'::FLOAT8,'1990-10-20 01:57:42.00058':::TIMESTAMP,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL) ) AS T(col4894_4896,col4894_4899,col4894_4904,col4894_4900,col4894_4897)\" []] :[[\"(1043,\\\"-35 years -4 mons -4 days -21:58:48.967541\\\",-2.4307332038879395,\\\"1990-10-20 01:57:42.00058\\\",\\\"59 years 11 mons 491 days 22:21:03.698774\\\")\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'schema3925.table4894'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='schema3925' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4894' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('schema3925.table4894', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'schema3925'\\n\\t\\t           AND table_name = 'table4894'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: INSERT INTO schema3925.table4894 (col4894_4895,col4894_4896,col4894_4897,col4894_4898,col4894_4899,col4894_4900) VALUES ('09:20:42.364954-06:06:00':::TIMETZ,1184:::OID,'50 years 4 mons 562 days 02:01:31.545969':::INTERVAL,(-0.41339758038520813):::FLOAT8,'-41 years -668 days -17:29:34.473611':::INTERVAL,'1970-11-03 03:19:17.000794':::TIMESTAMP),('08:37:57.269533-11:22:00':::TIMETZ,1043:::OID,'59 years 11 mons 491 days 22:21:03.698774':::INTERVAL,(-0.8522640466690063):::FLOAT8,'-35 years -4 mons -4 days -21:58:48.967541':::INTERVAL,'1990-10-20 01:57:42.00058':::TIMESTAMP), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 1, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 0eaeeb773474716753781289788fdd087fb9b166:

		  | runtime.gopark(0xc000fa5f78?, 0x3?, 0x0?, 0x30?, 0xc000fa5f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc000fa5dc8 sp=0xc000fa5da8 pc=0x466c16
		  | runtime.selectgo(0xc000fa5f78, 0xc000fa5f44, 0xc00126eb80?, 0x0, 0x4010ca0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc000fa5f08 sp=0xc000fa5dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc001372d70, {0x5bb4908, 0xc00126eb80})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000fa5fb8 sp=0xc000fa5f08 pc=0x1acf44e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000fa5fe0 sp=0xc000fa5fb8 pc=0x1acf74e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000fa5fe8 sp=0xc000fa5fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 370714 [runnable]:
		  | runtime.gopark(0xc000fa3f78?, 0x3?, 0x0?, 0x30?, 0xc000fa3f4a?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc000fa3dc8 sp=0xc000fa3da8 pc=0x466c16
		  | runtime.selectgo(0xc000fa3f78, 0xc000fa3f44, 0xc00126eb80?, 0x0, 0x4010ca0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc000fa3f08 sp=0xc000fa3dc8 pc=0x476adc
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc0026e9400, {0x5bb4908, 0xc00126eb80})
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:90 +0xee fp=0xc000fa3fb8 sp=0xc000fa3f08 pc=0x1acf44e
		  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start.func1()
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x2e fp=0xc000fa3fe0 sp=0xc000fa3fb8 pc=0x1acf74e
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000fa3fe8 sp=0xc000fa3fe0 pc=0x4981e1
		  | created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
		  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:123 +0x139
		  |
		  | goroutine 370820 [runnable]:
		  | runtime.gopark(0xc002256fb0?, 0x2?, 0x20?, 0x0?, 0xc002256fa4?)
		  | 	GOROOT/src/runtime/proc.go:363 +0xd6 fp=0xc002256e30 sp=0xc002256e10 pc=0x466c16
		  | runtime.selectgo(0xc002256fb0, 0xc002256fa0, 0x0?, 0x0, 0x0?, 0x1)
		  | 	GOROOT/src/runtime/select.go:328 +0x7bc fp=0xc002256f70 sp=0xc002256e30 pc=0x476adc
		  | github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1()
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:47 +0x6c fp=0xc002256fe0 sp=0xc002256f70 pc=0xd5016c
		  | runtime.goexit()
		  | 	GOROOT/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002256fe8 sp=0xc002256fe0 pc=0x4981e1
		  | created by github.com/jackc/pgconn/internal/ctxwatch.(*ContextWatcher).Watch
		  | 	github.com/jackc/pgconn/internal/ctxwatch/external/com_github_jackc_pgconn/internal/ctxwatch/context_watcher.go:46 +0x12c
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | lic.table4473 WHERE (col4473_4486)= ( SELECT  (col4473_4486) FROM (VALUES( 25:::OID) ) AS T(col4473_4486) ) )\" []] :false\nQUERY [\"SELECT array[((col4473_4486))::STRING] FROM (VALUES(25:::OID) ) AS T(col4473_4486)\" []] :[[\"25\"]]\nQUERY [\"SELECT num_nulls(col4473_4486) \u003e 0 FROM (VALUES(701:::OID) ) AS T(col4473_4486)\" []] :false\nQUERY [\"SELECT COUNT (*) \u003e 0 FROM (SELECT * FROM public.table4473 WHERE (col4473_4486)= ( SELECT  (col4473_4486) FROM (VALUES( 701:::OID) ) AS T(col4473_4486) ) )\" []] :false\nQUERY [\"SELECT array[((col4473_4486))::STRING] FROM (VALUES(701:::OID) ) AS T(col4473_4486)\" []] :[[\"701\"]]\nQUERY [\"\\n\\t\\tSELECT array[parent.table_schema, parent.table_name, parent.column_name, child.column_name]\\n\\t\\t  FROM (\\n\\t\\t        SELECT conname, conkey, confkey, conrelid, confrelid\\n\\t\\t          FROM pg_constraint\\n\\t\\t         WHERE contype = 'f'\\n\\t\\t           AND conrelid = 'public.table4473'::REGCLASS::INT8\\n\\t\\t       ) AS con\\n\\t\\t\\tJOIN ( SELECT CONSTRAINT_NAME from information_schema.table_constraints \\n\\t\\t\\t\\t      WHERE table_schema ='public' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t    table_name='table4473' AND\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t(crdb_internal.is_constraint_active('public.table4473', constraint_name) = true)\\n           ) AS tc ON conname = tc.CONSTRAINT_NAME\\n\\t\\t  JOIN (\\n\\t\\t        SELECT column_name, ordinal_position, column_default\\n\\t\\t          FROM information_schema.columns\\n\\t\\t         WHERE table_schema = 'public'\\n\\t\\t           AND table_name = 'table4473'\\n\\t\\t       ) AS child ON conkey[1] = child.ordinal_position\\n\\t\\t  JOIN (\\n\\t\\t        SELECT pc.oid,\\n\\t\\t               cols.table_schema,\\n\\t\\t               cols.table_name,\\n\\t\\t               cols.column_name,\\n\\t\\t               cols.ordinal_position\\n\\t\\t          FROM pg_class AS pc\\n\\t\\t          JOIN pg_namespace AS pn ON pc.relnamespace = pn.oid\\n\\t\\t          JOIN information_schema.columns AS cols ON (pc.relname = cols.table_name AND pn.nspname = cols.table_schema)\\n\\t\\t       ) AS parent ON (\\n\\t\\t                       con.confkey[1] = parent.ordinal_position\\n\\t\\t                       AND con.confrelid = parent.oid\\n\\t\\t                      )\\n\\t\\t WHERE child.column_name != 'rowid';\\n\" []] :[]\n\n===========================\nPrevious statements [QUERY: DROP SCHEMA \"schema1254\" CASCADE, Expected Errors: , Potential Errors:  QUERY: CREATE TYPE schema1884.enum4643 AS ENUM ('saad', 'sff', 'sfs', 'af', 'a'), Expected Errors: , Potential Errors:  QUERY: INSERT INTO public.table4473 (col4473_4475,col4473_4476,col4473_4477,col4473_4478,col4473_4479,col4473_4480,col4473_4481,col4473_4482,col4473_4483,col4473_4484,col4473_4485,col4473_4486,col4473_4487,col4473_4488,col4473_4489) VALUES ('BOX(-1.5414831460613732 -1.6678590931387252,1.8135343231405447 0.05206416541382236)':::BOX2D,2206:::OID,e'\\x00'::CHAR,B'01011','17:12:13.425157':::TIME,'3000-01-01 00:00:00+00:00':::TIMESTAMPTZ,(-2147483648):::INT8,NULL,'1984-04-10 12:26:43.000985':::TIMESTAMP,'\\x00':::BYTES,e';zf\\x04'::STRING,90001:::OID,1042:::OID,(-1842093163):::INT8,(-1.6645545959472656):::FLOAT8),('BOX(-1.9285756037988755 -0.22387667790538246,0.9358243017520091 -0.19399173084054755)':::BOX2D,3802:::OID,':'::CHAR,B'01000','09:11:08.305014':::TIME,'1978-07-31 13:58:33.000484+00:00':::TIMESTAMPTZ,(-3905200269684727671):::INT8,true,'1984-07-03 08:16:13.000177':::TIMESTAMP,'\\x2e62c2':::BYTES,'\"'::STRING,25:::OID,1043:::OID,2147483647:::INT8,0.47048094868659973:::FLOAT8),('BOX(-1.150100758472085 -0.3469992550086455,-0.6873019350113381 -0.16640408981756305)':::BOX2D,19:::OID,'B'::CHAR,B'01000','19:02:36.961733':::TIME,'2020-11-22 10:01:04.000248+00:00':::TIMESTAMPTZ,2147483647:::INT8,false,'2013-05-20 06:30:26.000857':::TIMESTAMP,'\\x58b71d02642942':::BYTES,'jp7e'::STRING,701:::OID,2950:::OID,(-442943547):::INT8,(-1.545388102531433):::FLOAT8), Expected Errors: , Potential Errors: ]: ERROR: queryOid: descriptor is being dropped (SQLSTATE XXUUU)"
		  | }
		Wraps: (4) SSH_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 255
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.SSH (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 6, 2022
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 16, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ 762c1b86fe1c3a70338cee4a91c9e2e4c5e0fcfe:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_134446.781924515_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ bb4f8cd5396e73c31d370c0d0ac1881263c5156f:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_131806.397728074_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ a3b578466bcdcc5d2bbea628d33c337d87b9f4b8:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_133511.803815656_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.schemachange/random-load failed with artifacts on master @ c5decaeb69155399647909163451f3d8ca7858b6:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
(schemachange_random_load.go:165).runSchemaChangeRandomLoad: output in run_135358.653649291_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: SSH_PROBLEM: exit status 255

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@fqazi fqazi mentioned this issue Oct 28, 2022
Merged
@craig craig bot closed this as completed in 068845f Oct 28, 2022
@exalate-issue-sync exalate-issue-sync bot added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed T-sql-schema-deprecated Use T-sql-foundations instead labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. sync-me-8 T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workload/schemachanger: re-enable the schema changer workload fqazi/cockroach
3 participants
@jlinder @fqazi @cockroach-teamcity