admission: replace soft slots mechanism with elastic cpu tokens

[irfansharif]

Is your feature request related to a problem? Please describe.

We introduced a notion of soft slots in admission control (#78519) for additional threads for Pebble compaction compression. We envisioned granting these "squishy" slots to background activities and permit work only under periods of low load. In working through #86638 (as part of #75066), we observed experimentally that the moderate-slots count was not sensitive enough to scheduling latency, and consequently latency observed by foreground traffic. Elastic CPU tokens, the kind now being used for backups, offer an alternative to soft slots. This issue tracks:

• Replacing the use of soft slots in additional pebble compaction compression threads
• Running some sanity check experiments as part of this (perhaps those we ran in admission: introduce soft slots for speculative concurrency #78519)
• Removing the code around soft/moderate load slots (it's better to minimize the number of mechanisms in the admission package)

Additional context

Some internal discussion here. Also #87883 could help highlight the (lack of) sensitivity to scheduling latency, perhaps worth checking again. Note that the elastic CPU limiter is disabled for 22.2, so if we're looking to do this in 22.2 (likely too late in the release), we'd have to rework the feature gates slightly to only block the integration with ExportRequests.

+cc @sumeerbhola, @andrewbaptist, @bananabrick.

Jira issue: CRDB-19649

Epic CRDB-20293

[nicktrav]

@bananabrick - in #90791, it looks like Cockroach got the vendor bump with your recent changes, but the work granter code in pkg/server/config.go was commented out. I assume this means the work granting mechanism is effectively disabled on master right now.

What's left to do here?

[irfansharif]

What's left to do here?

Just deleting some dead AC code.