pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed

[cockroach-teamcity]

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2.0-rc @ d6d1d4840b9238b8d7fb120e88ace29cd6dd8912:

Fatal error:

panic: connection has timed out &{0xc0066795e0 0xc010e2afc0 17a011ce0754ad010000000000000001 0xc010a4cc60 { true} { false} 0}

Stack:

goroutine 3149308 [running]:
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc00b000ea0, {0x7ab9138, 0xc0103c7e00})
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:115 +0x26d
created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:133 +0x139

Log preceding fatal error

   "sql": "DROP SCHEMA \"public\" CASCADE",
   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}
{
 "workerId": 0,
 "clientTimestamp": "11:24:57.099016",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER DATABASE schemachange ADD REGION \"us-east1\"",
   "expectedExecErr": "42P12"
  }
 ],
 "expectedExecErrors": "42P12",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot add region \"us-east1\" to database schemachange (SQLSTATE 42P12)"
}
{
 "workerId": 0,
 "clientTimestamp": "11:24:57.505288",
 "ops": [
  "BEGIN",
  {
   "sql": "SELECT 'validating all objects', crdb_internal.validate_multi_region_zone_configs()"
  },
  "COMMIT"
 ],
 "expectedExecErrors": "",
 "expectedCommitErrors": "",
 "message": ""
}
{
 "workerId": 0,
 "clientTimestamp": "11:24:57.620516",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE INDEX \"IrrelevantName\" ON public.table108 (\"IrrelevantColumn\" ASC)",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table108\" does not exist (SQLSTATE 42P01)"
}

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [dropping enum value broken if concurrent with drop schema] #113859 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [dropping enum value broken if concurrent with drop schema] [C-test-failure O-robot O-schema-testing P-1 T-sql-foundations branch-release-23.2 db-cy-23]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #113655 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-remote-execution O-robot T-sql-foundations branch-master]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-34480

[rafiss]

This branch does not have #114853 backported to it

[rafiss]

@chrisseto do you think it's possible to backport #114853, or is not worth dealing with all the merge conflicts?

[cockroach-teamcity]

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.2.0-rc @ 0b0851f59e6c37706c5df624fa3a579faa46d736:

Fatal error:

panic: connection has timed out &{0xc00b267180 0xc004357b60 17a4f78ae6b04fb60000000000000001 0xc008271380 { true} { false} 0}

Stack:

goroutine 5637522 [running]:
github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).watchLoop(0xc017428ea0, {0x786e888, 0xc00a8a6050})
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:115 +0x25c
created by github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWatchDog).Start in goroutine 2180199
	github.com/cockroachdb/cockroach/pkg/workload/schemachange/watch_dog.go:133 +0x12b

Log preceding fatal error

   "expectedExecErr": "3F000"
  }
 ],
 "expectedExecErrors": "3F000",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot drop schema \"public\" (SQLSTATE 3F000)"
}
{
 "workerId": 0,
 "clientTimestamp": "10:20:15.372866",
 "ops": [
  "BEGIN",
  {
   "sql": "CREATE TABLE public.table168 (col168_169 BIT(34) NOT NULL, col168_170 JSONB NULL, PRIMARY KEY (col168_169 DESC), INDEX (col168_169 DESC) STORING (col168_170), INDEX (col168_169, col168_170 DESC), UNIQUE (col168_170, col168_169 DESC), UNIQUE (col168_169 ASC) STORING (col168_170), INVERTED INDEX (col168_169 DESC, col168_170 ASC), INDEX (lower(CAST(col168_169 AS STRING)) DESC, col168_169) NOT VISIBLE, FAMILY (col168_170), FAMILY (col168_169))",
   "potentialExecErr": "0A000,42601,42704,42P16"
  },
  {
   "sql": "SELECT t2.col168_169 AS col0,t0.col168_169 AS col1,t1.col168_169 AS col2,t2.col168_169 AS col3,t0.col168_169 AS col4,t2.col168_170 AS col5 FROM public.table168 AS t0 ,public.table168 AS t1 ,public.table168 AS t2  FETCH FIRST 1 ROWS ONLY",
   "potentialExecErr": "53100,53200,57014"
  },
  {
   "sql": "SELECT t1.col168_170 AS col0,t2.col168_170 AS col1,t1.col168_170 AS col2 FROM public.table168 AS t0 ,public.table168 AS t1 ,public.table168 AS t2  FETCH FIRST 1 ROWS ONLY",
   "potentialExecErr": "53100,53200,57014"
  },
  {
   "sql": "CREATE SEQUENCE IF NOT EXISTS public.seq171"
  },
  {
   "sql": "ALTER DATABASE schemachange ADD REGION \"us-east1\"",
   "expectedExecErr": "42P12"
  }
 ],
 "expectedExecErrors": "42P12",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: cannot add region \"us-east1\" to database schemachange (SQLSTATE 42P12)"
}
{
 "workerId": 0,
 "clientTimestamp": "10:20:16.966675",
 "ops": [
  "BEGIN",
  {
   "sql": "ALTER INDEX public.table172@\"IrrelevantConstraintName\" RENAME TO \"OtherConstraintName\"",
   "expectedExecErr": "42P01"
  }
 ],
 "expectedExecErrors": "42P01",
 "expectedCommitErrors": "",
 "message": "ROLLBACK; Successfully got expected execution error.: ERROR: relation \"public.table172\" does not exist (SQLSTATE 42P01)"
}

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [dropping enum value broken if concurrent with drop schema] #113859 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [dropping enum value broken if concurrent with drop schema] [C-test-failure O-robot O-schema-testing P-1 T-sql-foundations branch-release-23.2 db-cy-23]
• pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #113655 pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [C-test-failure O-remote-execution O-robot T-sql-foundations branch-master]

This test on roachdash | Improve this report!

[rafiss]

This failure seems to have stabilized.