Describe backup priority + admission control

[shermanCRL]

Matt Sherman (shermanCRL) commented:

In v22.1, and later patch releases of v21.2, backups will operate with a different priority than previous versions:

• backupccl: mark backup ExportRequests as Bulk priority cockroach#79086

Let’s document what users should expect here, and link to the broader context of admission control. cc @stevendanna @dt

Jira Issue: DOC-3528

[exalate-issue-sync]

Kathryn Hancox (kathancox) commented:
After talking to David, going to push this back until a docs issue comes through relating to a more significant change around backup priority for v23.1. The change in v22.1 is minor.

In v22.1, we changed to send with a different priority, but this would only trigger if full resource exhaustion was happening.

The change in 23.1, is much more significant, which is that the requests now actively leave spare resources available, to ensure they are there for foreground traffic. (That mode was available in 22.2 as well but defaulted off.)

So, I will update docs around this with v23.1.

[exalate-issue-sync]

Kathryn Hancox (kathancox) commented:
Main two PRs for the elastic CPU integration for backups/cdc:

cockroachdb/cockroach#86638 - for backups
cockroachdb/cockroach#89709 - for cdc initial scans

Checked in for v22.2, but only enabled in 23.1

Detailed blog post on topic, this section might be the most useful: https://www.cockroachlabs.com/blog/rubbing-control-theory/#35-experimentation-and-analysis

From Aditya: is that the KV requests sent by backups are allotted fixed slices of on-CPU time to read the required keys/values. Once the request has exhausted its on-CPU time it paginates and will have to be resumed at a later time once it can be allocated a new on-CPU time slice. This pagination allows for other processes such as foreground traffic to continue almost unaffected since there is a cap on how much CPU a backup will be allowed to hog.

It seems as though we can do a similar thing to CDC where we reference this on the Advanced Config page: https://www.cockroachlabs.com/docs/stable/advanced-changefeed-configuration.html