upgrades: index validation should ignore CreatedAtNanos field

[postamar]

Presently, a cluster upgrade step which adds a new index may fail during validation because the validation expects the CreatedAtNanos to be zero and the field may already have a non-zero value.

Right now the only place this can happen in existing releases is the upgrade step in 22.1 which adds an index to system.statement_diagnostics_requests.

Jira issue: CRDB-18121

[blathers-crl]

Hi @postamar, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[postamar]

The fix itself should be extremely straightforward: zero the CreatedAtNanos field values in hasIndex. The fix should however be accompanied by a regression test which should catch this as well as future instances of this class of bugs, where we add a new field with non-deterministic contents but forget to zero it in this upgrade validation logic.

@ajwerner suggested we take inspiration from https://github.com/cockroachdb/cockroach/blob/614c2b131220db8de85f193c2d059bb7da7e7685/pkg/migration/migrations/retry_jobs_with_exponential_backoff_external_test.go
I agree. I'll add that we shouldn't tie the test to a specific upgrade step, instead we should reframe it as a unit test to cover upgrades.ValidateSchemaExists. We'll have to mock a system table upgrade somehow. This could be done by creating a new, unrelated table for the purposes of the test and piping operations defined in test cases through something that does the same thing as migrateTable does.

[blathers-crl]

Hi @postamar, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}