migrations: add migration to wait on in-flight schema changes

[stevendanna]

Eventually, we want to stop accepting non-MVCC AddSSTables. To do
this, we need all CREATE INDEX schema changes to use the new
mvcc-compatible process. To make this more tenable, we want to drain
in-flight schema changes during a migration.

Release note: Finalizing an upgrade to 22.2 requires that all
in-flight schema changes enter a terminal state. This may mean that
finalization takes as long as the longest-running schema change.

Release justification: Critical migration to ensure that 22.2's assumptions
about non-MVCC operations are true.

[cockroach-teamcity]

This change is 

[stevendanna]

We'd likely want to wait on this until after #73878 and until we are ready to pull the trigger on only using the new AddSStable and new backfiller.

[ajwerner]

though I agree about waiting

Reviewed 10 of 12 files at r1.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @dt and @stevendanna)

---

pkg/migration/migrations/wait_for_schema_changes.go, line 73 at r1 (raw file):

		id, ok := d.(*tree.DInt)
		if !ok {
			return errors.Newf("unexpected type for id column: %T (expected DInt)", d)

make these errors into assertion failures?

Code quote:

			return errors.Newf("unexpected number of columns: %d (expected 1)", len(datums))
		}
		d := datums[0]
		id, ok := d.(*tree.DInt)
		if !ok {
			return errors.Newf("unexpected type for id column: %T (expected DInt)", d)

---

pkg/migration/migrations/wait_for_schema_changes_test.go, line 258 at r1 (raw file):

				// TODO(ssd): Some tests may succeed just
				// because we haven't actually started the
				// migrations yet. The sleep is a poor way to

plumb a channel into a testing knob?

Code quote:

				// TODO(ssd): Some tests may succeed just
				// because we haven't actually started the
				// migrations yet. The sleep is a poor way to
				// help ensure the migration starts before the
				// schema change finishes.
				time.Sleep(10 * time.Millisecond)

[stevendanna]

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner and @dt)

---

pkg/migration/migrations/wait_for_schema_changes.go, line 73 at r1 (raw file):

Previously, ajwerner wrote…

make these errors into assertion failures?

Done.

---

pkg/migration/migrations/wait_for_schema_changes_test.go, line 258 at r1 (raw file):

Previously, ajwerner wrote…

plumb a channel into a testing knob?

Done. Along with some new interval knobs to speed up the test a bit.

[ajwerner]

Are we going to do this? What would we get out of it?

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner, @dt, and @samiskin)

[stevendanna]

@ajwerner That's a good question. I wonder what @dt and @erikgrinaker think. IIRC, we wanted to do this to lower the impact of #76934. If we aren't going to do that this release, then it might be nice to not do this migration in this release and preserve the ability to flip back to the old index backfill code if we really need to.

[stevendanna]

@dt @ajwerner I rebased this against master and also added a commit that removes the option for old-style backfills.

[stevendanna]

bors r=dt

[stevendanna]

bors cancel

[craig]

Canceled.

[stevendanna]

bors r=dt

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[stevendanna]

bors r=dt

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[stevendanna]

bors retry

[craig]

Build failed:

• Bazel Essential CI (Cockroach)

[stevendanna]

bors retry

Last two failures:

• spanconfigtestutils: copy gc job testing knobs to tenant #86558
• sql: attempt to deflake TestRaceWithIndexBackfillMerge #86554

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)