roachtest: schemachange/mixed-versions failed [descriptor ID generator unavailable, migration in progress]

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 5bccb34dc1d1a62645e638d66cf6288149e23507:

(mixedversion.go:594).Run: mixed-version test failure while running step 21 (run "run schemachange workload and validation in mixed version"): full command output in run_061531.355009638_n4_COCKROACHRANDOMSEED7.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed [attempting to append refresh spans after the tracked timestamp has moved forward] #122274 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.2 release-blocker]
• roachtest: schemachange/mixed-versions failed [missing version gate for UDF calling UDF] #121821 roachtest: schemachange/mixed-versions failed [issue renaming and using a function in the same txn] [C-test-failure GA-blocker O-roachtest O-robot P-1 T-sql-foundations branch-master]
• roachtest: schemachange/mixed-versions failed [enum removal not yet in 23.2 release] #121411 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-24.1]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-37823

[fqazi]

We need to tolerate, the following error in this workload:
Error: ***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: descriptor ID generator unavailable, migration in progress, retry later (SQLSTATE XXUUU)

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-23.1 @ 331c98abb3143d38507e71cdb6352a502d6b3875:

(mixedversion.go:594).Run: mixed-version test failure while running step 23 (run "run schemachange workload and validation after upgrade has finalized"): full command output in run_061842.886033022_n4_cockroach-debug-doct.log: COMMAND_PROBLEM: exit status 125
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed #122572 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-master release-blocker]
• roachtest: schemachange/mixed-versions failed [attempting to append refresh spans after the tracked timestamp has moved forward] #122274 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-kv branch-release-23.2 release-blocker]
• roachtest: schemachange/mixed-versions failed [enum removal not yet in 23.2 release] #121411 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot P-1 T-sql-foundations branch-release-24.1]

This test on roachdash | Improve this report!

[rafiss]

latest failure is

Wraps: (6) Node 4. Command with error:
  | ```
  | ./cockroach debug doctor examine cluster --url {pgurl:4}
  | ```
  | stdout:Examining 435 descriptors and 265 namespace entries...
  | Examining 1678 jobs...
  | job 961533470234443779: running schema change GC refers to missing table descriptor(s) [638]; existing descriptors that still need to be dropped [644]; job safe to delete: false.
  |
  | stderr:ERROR: validation failed
  | Failed running "debug doctor examine cluster"

this seems similar to #123477 or #122675. closing as a duplicate.