schemachanger: Refactor tests for concurrent schema changer behaviors

[Xiang-Gu]

1. It cleans up some redundant tests about concurrent schema changer behavior and refactor in a new simpler, cleaner test
2. It adds an integration style test for testing concurrent schema change behaviors where we run many schema changes for an extended period of time and assert that all of they eventually succeed and the descriptors end up in the expected state.

Fix #108140
Fix #107223

Epic: None
Release note: None

[cockroach-teamcity]

This change is 

[Xiang-Gu]

CI tests are flaky tests unrelated to this PR. Ready for a look!

[rafiss]

nice work! quick question, do you know if the new test would have been able to catch the bug described here? #106933

[Xiang-Gu]

@rafiss yes but with a different error! So I looked at that PR that fixes the bug. It essentially added two lines as listed below. So I did the following:

1. Run the test as in the PR, it passes;
2. Remove one line and re-run this test, rather quickly, it fails with error

    schemachanger_test.go:770: pq: internal error: executing declarative schema change PreCommitPhase stage 2 of 2 with 3 MutationType ops (rollback=false) for DROP INDEX: error executing PreCommitPhase stage 2 of 2 with 3 MutationType ops: *scop.SetJobStateOnDescriptor: ... unexpected schema change job ID 889775506765152257 on table 107, expected 0

3. Add back that line, re-run the test, and it passes.
4. Remove another one line and re-run this test, rather quickly, it fails with the same type of error again.
5. Add back this other line, re-run the test, and it passes.

[fqazi]

@Xiang-Gu This is excellent stuff! A couple of suggestions for improvements, the transaction one can be ignored.

Reviewed 2 of 2 files at r1, 1 of 2 files at r2, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @Xiang-Gu)

---

pkg/sql/schemachanger/schemachanger_test.go line 635 at r2 (raw file):

			scName = newSCName
			t.Logf("RENAME SCHEMA TO %v", newSCName)
		} else if isPQErrWithCode(err, pgcode.UndefinedDatabase) {

Would it make sense to use catalog queries here instead and transactions, that way, we get retry errors instead. But I think we can't because of declarative based on our slack conversation.

---

pkg/sql/schemachanger/schemachanger_test.go line 643 at r2 (raw file):

	// A goroutine that renames table `testdb.testsc.t` randomly.
	go repeatWorkWithInterval("rename-tbl-worker", renameTblInterval, func() error {

ctxgroup is a cleaner pattern here and would simplify the code

[Xiang-Gu]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @fqazi)

---

pkg/sql/schemachanger/schemachanger_test.go line 643 at r2 (raw file):

Previously, fqazi (Faizan Qazi) wrote…

ctxgroup is a cleaner pattern here and would simplify the code

I explored ctxgroup but its functionality is not suitable for our case here.
Namely, ctxgourp allows us to spawn workers, and wait for them to complete. Here we want the main routine to let workers work indefinitely and signal them to finish once a time up fires.

[Xiang-Gu]

TFTR!
bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)