backupccl: flake in TestClusterRestoreFailCleanup/retry-during-custom-system-table-restore/settings

[yuzefovich]

On this bors run.

=== RUN   TestClusterRestoreFailCleanup/retry-during-custom-system-table-restore/settings
*
* ERROR: a panic has occurred!
* expected *DOid, found tree.dNull
* (1) attached stack trace
*   -- stack trace:
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:884
*   | [...repeated from below...]
* Wraps: (2) assertion failure
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.MustBeDOid
*   | 	github.com/cockroachdb/cockroach/pkg/sql/sem/tree/datum.go:5244
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.roleIDSeqRestoreFunc
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/system_schema.go:362
*   | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*restoreResumer).restoreSystemTables.func3
*   | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/restore_job.go:2947
...

Jira issue: CRDB-22347

[blathers-crl]

cc @cockroachdb/disaster-recovery

[msbutler]

pretty sure this is #92848. I saw a similar error here #91991 (comment)

[rytaft]

This test flaked again in a slightly different way:

Failed
=== RUN   TestClusterRestoreFailCleanup
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/a3d5b6281df336d2ed5d8f2bd8c98231/logTestClusterRestoreFailCleanup3603781448
    test_log_scope.go:79: use -show-logs to present logs inline
    full_cluster_backup_restore_test.go:618: error executing 'BACKUP TO 'nodelocal://1/missing-ssts'': pq: failed to run backup: writing forward-compat metadata sst: pebble: keys must be added in strictly increasing order: "stats/\x8c\x88"/0,0#0,SET, "stats/\x8c\x88"/0,0#0,SET
    testutils.go:199: no Invalid Descriptors
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/a3d5b6281df336d2ed5d8f2bd8c98231/logTestClusterRestoreFailCleanup3603781448
--- FAIL: TestClusterRestoreFailCleanup (227.24s)

Let me know if you want me to open a separate issue.

[msbutler]

aha, just fixed that failure mode here #93837. Ty for flagging!

[stevendanna]

I'm going to close this as we've fixed the issues implicated here.