scrun: added a cluster setting to skip planner sanity checks in prod

[Xiang-Gu]

Previously, there is a niche case where we might run into a backward incompatible issue (see #91528). We decided to fix it by relaxing the sanity check that caused the issue and backport the change to relase-22.2, so this backward incompatibility issue is contained only between release-22.2.0 and master, which we think should be a rare occurrance in the wild.

Fixes: #91528
Release note: None

[cockroach-teamcity]

This change is 

[postamar]

Is it possible to test this? I'm concerned of the possibility that while BuildStages no longer throws an error now, ValidateStages might still be.

[Xiang-Gu]

The only way to test I can think of is in a roachtest; in fact, we do have such a roachtest but it's currently disabled (It will be revived soon when build version is v23.1).

I tested it manually:

• Apply this PR to both master and release-22.2.
• Then do the following commands:

roachprod create local -n 2
git checkout release-22.2
dev build cockroach
mv cockroach cockroach-22.2
roachprod put local:1 ./cockroach-22.2
git checkout master
dev build cockroach
roachprod put local:2 ./cockroach
roachprod start local:1 --binary=cockroach-22.2
roachprod start local:2 --binary=cockroach
roachprod run local:2 -- touch data/DISABLE_STARTING_BACKGROUND_JOBS
roachprod sql local:2 -- -e 'create table t (i int primary key);'
roachprod sql local:2 -- -e 'drop table t;'

The last command to drop the table succeeded, which means the job for drop table t is backward compatible and can be adopted and completed by a node running v22.2:

(Note that it took >30s to finish that last command; I suspect it's related to how we detect and adopt pending jobs in our framework, instead of the command itself taking that long)

[postamar]

Sorry I forgot about this PR! Approved, of course.

(Note that it took >30s to finish that last command; I suspect it's related to how we detect and adopt pending jobs in our framework, instead of the command itself taking that long)

I agree. I noticed the same thing when setting pausepoints in demo.

[Xiang-Gu]

TFTR!

bors r+

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 255ceba to blathers/backport-release-22.2-92774: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 22.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}