roachtest: schemachange/random-load failed

[cockroach-teamcity]

roachtest.schemachange/random-load failed with artifacts on release-22.2 @ 78e7ab8398a655b612cf7c9f8f31d904da5caa84:

test artifacts and logs in: /artifacts/schemachange/random-load/run_1
	schemachange_random_load.go:162,schemachange_random_load.go:56,test_runner.go:930: output in run_140821.311903847_n1_workload_run_schemachange: ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned: COMMAND_PROBLEM: exit status 1
		(1) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).RunE
		  | 	main/pkg/cmd/roachtest/cluster.go:2018
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runSchemaChangeRandomLoad
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:159
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerSchemaChangeRandomLoad.func1
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/schemachange_random_load.go:56
		  | main.(*testRunner).runTest.func2
		  | 	main/pkg/cmd/roachtest/test_runner.go:930
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (2) output in run_140821.311903847_n1_workload_run_schemachange
		Wraps: (3) ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json returned
		  | stderr:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  |          )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n\t\t" ["schema3428.table4163"]] :[]
		  | QUERY ["\n    WITH tab_json AS (\n                    SELECT crdb_internal.pb_to_json(\n                            'desc',\n                            descriptor\n                           )->'table' AS t\n                      FROM system.descriptor\n                     WHERE id = $1::REGCLASS\n                  ),\n         columns_json AS (\n                        SELECT json_array_elements(t->'columns') AS c FROM tab_json\n                      ),\n         columns AS (\n                    SELECT (c->>'id')::INT8 AS col_id,\n                           IF(\n                            (c->'inaccessible')::BOOL,\n                            c->>'computeExpr',\n                            c->>'name'\n                           ) AS expr\n                      FROM columns_json\n                 ),\n         indexes_json AS (\n                         SELECT json_array_elements(t->'indexes') AS idx\n                           FROM tab_json\n                         UNION ALL SELECT t->'primaryIndex' FROM tab_json\n                      ),\n         unique_indexes AS (\n                            SELECT idx->'name' AS name,\n                                   json_array_elements(\n                                    idx->'keyColumnIds'\n                                   )::STRING::INT8 AS col_id\n                              FROM indexes_json\n                             WHERE (idx->'unique')::BOOL\n                        ),\n         index_exprs AS (\n                        SELECT name, expr\n                          FROM unique_indexes AS idx\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\n                     )\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\n    FROM index_exprs\n   WHERE expr != 'rowid'\nGROUP BY name;\n" ["schema3428.table4163"]] :[]
		  | Stmts: 
		  | [QUERY: SELECT t1.col3649_3651 AS col0,t2.col2952_2956 AS col1,t0.col3649_3660 AS col2,t0.col3649_3667 AS col3,t2.col2952_2953 AS col4,t1.col5004_5005 AS col5,t2.col1126_1129 AS col6,t0.col3649_3667 AS col7,t0.col3649_3656 AS col8,t2.col2952_2956 AS col9,t2.col1126_1128 AS col10,t1.col3649_3655 AS col11,t0.col3649_3654 AS col12,t1.col5004_5009 AS col13,t2.col2952_2958 AS col14 FROM schema5648.table5749 AS t0 ,schema5648.table5749 AS t1 ,schema3428.table4163 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]
		  | : ERROR: unknown schema "[865]" (SQLSTATE 3F000)
		  |
		  | stdout:
		  | <... some data truncated by circular buffer; go to artifacts for details ...>
		  | 003e'columns') AS c FROM tab_json\\n                      ),\\n         columns AS (\\n                    SELECT (c-\u003e\u003e'id')::INT8 AS col_id,\\n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\\t\\t\" [\"schema3428.table4163\"]] :[]\nQUERY [\"\\n    WITH tab_json AS (\\n                    SELECT crdb_internal.pb_to_json(\\n                            'desc',\\n                            descriptor\\n                           )-\u003e'table' AS t\\n                      FROM system.descriptor\\n                     WHERE id = $1::REGCLASS\\n                  ),\\n         columns_json AS (\\n                        SELECT json_array_elements(t-\u003e'columns') AS c FROM tab_json\\n                      ),\\n         columns AS (\\n                    SELECT (c-\u003e\u003e'id')::INT8 AS col_id,\\n                           IF(\\n                            (c-\u003e'inaccessible')::BOOL,\\n                            c-\u003e\u003e'computeExpr',\\n                            c-\u003e\u003e'name'\\n                           ) AS expr\\n                      FROM columns_json\\n                 ),\\n         indexes_json AS (\\n                         SELECT json_array_elements(t-\u003e'indexes') AS idx\\n                           FROM tab_json\\n                         UNION ALL SELECT t-\u003e'primaryIndex' FROM tab_json\\n                      ),\\n         unique_indexes AS (\\n                            SELECT idx-\u003e'name' AS name,\\n                                   json_array_elements(\\n                                    idx-\u003e'keyColumnIds'\\n                                   )::STRING::INT8 AS col_id\\n                              FROM indexes_json\\n                             WHERE (idx-\u003e'unique')::BOOL\\n                        ),\\n         index_exprs AS (\\n                        SELECT name, expr\\n                          FROM unique_indexes AS idx\\n                               INNER JOIN columns AS c ON idx.col_id = c.col_id\\n                     )\\n  SELECT ARRAY['(' || array_to_string(array_agg(expr), ', ') || ')'] AS final_expr\\n    FROM index_exprs\\n   WHERE expr != 'rowid'\\nGROUP BY name;\\n\" [\"schema3428.table4163\"]] :[]\nStmts: \n[QUERY: SELECT t1.col3649_3651 AS col0,t2.col2952_2956 AS col1,t0.col3649_3660 AS col2,t0.col3649_3667 AS col3,t2.col2952_2953 AS col4,t1.col5004_5005 AS col5,t2.col1126_1129 AS col6,t0.col3649_3667 AS col7,t0.col3649_3656 AS col8,t2.col2952_2956 AS col9,t2.col1126_1128 AS col10,t1.col3649_3655 AS col11,t0.col3649_3654 AS col12,t1.col5004_5009 AS col13,t2.col2952_2958 AS col14 FROM schema5648.table5749 AS t0 ,schema5648.table5749 AS t1 ,schema3428.table4163 AS t2  FETCH FIRST 1 ROWS ONLY, Expected Errors: , Potential Errors: 53200]\n: ERROR: unknown schema \"[865]\" (SQLSTATE 3F000)"
		  | }
		Wraps: (4) COMMAND_PROBLEM
		Wraps: (5) Node 1. Command with error:
		  | ``````
		  | ./workload run schemachange --verbose=1 --tolerate-errors=false  --histograms=perf/stats.json --max-ops 5000 --concurrency 20 --txn-log /mnt/data1/cockroach/transactions.json
		  | ``````
		Wraps: (6) exit status 1
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

• roachtest: schemachange/random-load failed #89146 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema]
• roachtest: schemachange/random-load failed #82133 roachtest: schemachange/random-load failed [C-test-failure O-roachtest O-robot T-sql-schema branch-master sync-me-8]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-20285

[ajwerner]

@fqazi is there a backport for this?

[fqazi]

This one was resolved via: #89883 on 22.2 @postamar @ajwerner. This specific failure was a workload bug