Revert "kvprober: metamorphically enable / configure kvprober"

[nvanbenschoten]

This reverts (most of) commit 769ba1c.

That commit metamorphically enabled kvprober. This has been observed to be destabliziing to unit tests. When the metamorphic constant is enabled (50% of the time) and when kvprober is fast enough, random ranges will see extra requests that they aren’t expecting. This adds nondeterminism which can trip up tests in any number of different ways.

All of the following flakes have been tracked back to kvprober:

Fixes #107864.
Fixes #108242.
Fixes #108441.
Fixes #108349.
Fixes #108124.
Closes #108366.

Release note: None

[blathers-crl]

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[cockroach-teamcity]

This change is 

[andrewbaptist]

I agree that this adds too much non-determinism into tests, especially at the KV layer and below. We could consider a change like this for SQL tests, but I'm not sure how useful that would be in practice.

[joshimhoff]

Sorry for the trouble & thx for tracking this down & fixing.

To be clear about it, enabling kvprober leads to test failures due to the fact that the tests do very specific things and assume that there is no / minimal background querying of ranges, etc. But we do not see an bad behavior caused by kvprober. Right or wrong?

[nvanbenschoten]

TFTRs!

bors r=andrewbaptist,joshimhoff

To be clear about it, enabling kvprober leads to test failures due to the fact that the tests do very specific things and assume that there is no / minimal background querying of ranges, etc. But we do not see an bad behavior caused by kvprober. Right or wrong?

Yes, that's correct. We saw no indication that kvprober itself had issues.

[craig]

Build failed (retrying...):

• Bazel Essential CI (Cockroach)

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)