roachtest: schemachange/mixed-versions failed [attempting to append refresh spans after the tracked timestamp has moved forward]

[cockroach-teamcity]

roachtest.schemachange/mixed-versions failed with artifacts on release-23.2.6-rc @ 7b037e25facb77ea370d9023c642626fdc1d5ba9:

(mixedversion.go:592).Run: mixed-version test failure while running step 15 (run "run schemachange workload and validation in mixed version"): full command output in run_064801.653542578_n4_COCKROACHRANDOMSEED4.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/schemachange/mixed-versions/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_metamorphicBuild=false
• ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: schemachange/mixed-versions failed [debug doctor validation: existing descriptors that still need to be dropped] #124722 roachtest: schemachange/mixed-versions failed [C-test-failure O-roachtest O-robot T-sql-foundations branch-release-23.1.22-rc]
• roachtest: schemachange/mixed-versions failed [functions cannot have more than 100 arguments] #124390 roachtest: schemachange/mixed-versions failed [functions cannot have more than 100 arguments] [C-test-failure O-roachtest O-robot P-2 T-sql-foundations branch-master]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-39075

[rafiss]

 "message": "***UNEXPECTED ERROR; Failed to generate a random operation: Collect: CollectRows: \"WITH \\\"descriptors\\\" AS (SELECT\\n\\t\\tdescriptor.id,\\n\\t\\t\\\"parentSchemaID\\\" AS schema_id,\\n\\t\\tnamespace.name AS name,\\n\\t\\tcrdb_internal.pb_to_json('desc', descriptor) AS descriptor\\n\\tFROM system.descriptor\\n\\tJOIN system.namespace ON namespace.id = descriptor.id\\n\\tWHERE \\\"parentID\\\" = (SELECT id FROM system.namespace WHERE name = current_database() AND \\\"parentID\\\" = 0)\\n\\t),\\n\\\"enums\\\" AS (SELECT id, schema_id, name, descriptor-\u003e'type' AS descriptor FROM descriptors WHERE descriptor ? 'type'),\\n\\\"enum_members\\\" AS (SELECT *, jsonb_array_elements(descriptor-\u003e'enumMembers') AS member FROM enums)\\nSELECT\\n\\t\\t\\t\\tquote_ident(schema_id::REGNAMESPACE::TEXT) || '.' || quote_ident(name) AS name,\\n\\t\\t\\t\\tquote_literal(member-\u003e\u003e'logicalRepresentation') AS value,\\n\\t\\t\\t\\tCOALESCE(member-\u003e\u003e'direction' = 'REMOVE', false) AS dropping,\\n\\t\\t\\t\\tCOALESCE(json_array_length(descriptor-\u003e'referencingDescriptorIds') \u003e 0, false) AS has_references\\n\\t\\t\\tFROM enum_members\\n\\t\" []: ERROR: internal error: attempting to append refresh spans after the tracked timestamp has moved forward. batchTimestamp: 1716965296.670323131,0 refreshedTimestamp: 1716965296.766832731,1 (SQLSTATE XX000)",

We saw this before in #122274. This was supposed to be fixed in #123150 and #123154. It looks like the change is in the 23.2.6-rc branch, as well as the v23.1.22 predecessor version. cc @yuzefovich do you know if this error can occur in another way?

[yuzefovich]

I'll need to think about it, but I'm quite certain that it's not a regression, so I'm dropping the release blocker label.

[yuzefovich]

Actually, #123154 is only included in 23.1.22+, but the test was using 23.1.21, so I'm closing this as a dup of #122274