backupccl: ExportRequest with MaxUserPriority times out when it encounters an intent

[adityamaru]

Context:

Backup gives each ExportRequest 5 minutes to complete before it cancels the corresponding context. ExportRequest is initially sent with header.WaitPolicy = lock.WaitPolicyError which means that it fails with a WriteIntentError when it sees an intent written by another transaction. For a minute, we retry the export if we see a WriteIntentError, after which we send the request with header.UserPriority = roachpb.MaxUserPriority and header.WaitPolicy = lock.WaitPolicyBlock, which is expected to push any other lower priority transaction out of the way.

Issue:

The ExportRequest sent with a higher priority does not seem to push the low priority txn on its final attempt, and therefore times out after 5 minutes. Following is a test reproducing this behavior:

func TestExportRequestHighPriority(t *testing.T) {
	defer leaktest.AfterTest(t)()
	defer log.Scope(t).Close(t)

	const numAccounts = 1
	tc, sqlDB, _, cleanupFn := backupRestoreTestSetup(t, singleNode, numAccounts, InitManualReplication)
	defer cleanupFn()

	_, t10sql := serverutils.StartTenant(t, tc.Server(0), base.TestTenantArgs{
		TenantID: roachpb.MakeTenantID(10),
	})

	_, err := t10sql.Exec(`CREATE TABLE foo (id INT)`)
	require.NoError(t, err)

	ctx := context.Background()
	g := ctxgroup.WithContext(ctx)
	g.Go(func() error {
		_, err := t10sql.Exec(`
BEGIN;
INSERT INTO foo VALUES (1);
`)
		return err
	})

	sqlDB.Exec(t, `BACKUP TENANT 10 INTO 'nodelocal://1/foo'`)
	require.NoError(t, g.Wait())
}

Jira issue: CRDB-17016

[blathers-crl]

cc @cockroachdb/bulk-io

[adityamaru]

@nvanbenschoten has a fix in mind, following which I'll check in a backup level test. This issue might be what we are seeing in https://github.com/cockroachlabs/support/issues/1661 (v21.2.12) and potentially https://github.com/cockroachlabs/support/issues/1570.