sql: deflake and unskip TestTenantStatementTimeoutAdmissionQueueCancelation

[cucaroach]

Make sure we get 4 blockers parked in the admission control queues and no more and retry the
main query until it gets parked and booted from it, before we expected the main query to always
reach the admission control q but that isn't reliable.
Fixes: #78494

[cockroach-teamcity]

This change is 

[yuzefovich]

There appears to be a data race.

Reviewed 1 of 2 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @cucaroach)

---

-- commits line 3 at r1:
nit: can you add some words for what the problem was and how it's fixed now?

[cucaroach]

I couldn't reproduce the race, the test ran 100,000 w/o hanging though. I think I understand and fixed the race but its a bit of a guess.

[yuzefovich]

Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @cucaroach)

---

pkg/sql/run_control_test.go line 962 at r2 (raw file):

	// till we get one into the KV layer.
	for atomic.LoadUint64(&hitMainQuery) == 0 {
		_, err := client.DB.ExecContext(context.Background(), `SELECT * FROM foo`)

nit: can / should we check err here? IIUC we should be getting err == nil while hitMainQuery == 0. We could check that non-nil error is only seen once or something like that.

[cucaroach]

Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @yuzefovich)

---

pkg/sql/run_control_test.go line 962 at r2 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: can / should we check err here? IIUC we should be getting err == nil while hitMainQuery == 0. We could check that non-nil error is only seen once or something like that.

Actually it errors every time with a timeout error, my flaw was in thinking the 500ms was enough time to guarantee it would get parked in the admission control queue and get cancelled there but every once in a blue moon it doesn't get that far and gets cancelled sooner. I'll make it clear and add a require.Error().

[cucaroach]

bors r+

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[cucaroach]

blathers backport 23.1 22.2

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 5bf7940 to blathers/backport-release-22.2-99876: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2 failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}