slow 8tb restore

[tbg]

Describe the problem

I ran #98767, i.e. the 8tb restore test but not using a weird RAID0 (but instead using gp3 at 125mb/s). This passed 7/10, but there were three failures documented here for further investigation.

Full artifacts: https://drive.google.com/drive/folders/18HBSubR9dTZG48gcW6AdU_PSG3rX-RVs?usp=sharing (tar -xjv <file>)

Run 6

Everything looks balanced.

the nodes are consuming quite a bit of CPU

RAM, leaseholders, etc, is balanced. No LSM inversion. Admission control not active as far as I can tell.

The ingest seems to hit some sort of blocker, you can see that live bytes just stops growing at the previous rate:

At the same time of this change, the logcommit latencies go from the ~200ms range into mostly the single digits. So we're just not throwing much at the cluster any more.

This seems like the kind of investigation the DR team could take the lead on - I'm unsure how to glean from the artifacts the progress the restore job is making. It's possible we just ended up distributing the load poorly (or something like that), or that the s3 data source slowed down, etc.

I glanced through the logs and the cluster looks happy - we just stop loading it at the same rate at some point:

Run 8

run_003404.366381060_n1_cockroach-sql-insecu: 00:40:18 cluster.go:1991: > result: ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05T17:30:00Z' " returned: COMMAND_PROBLEM: ssh verbose log retained in ssh_003404.373614968_n1_cockroach-sql-insecu.log: exit status 1
(1) ./cockroach sql --insecure -e "RESTORE  FROM LATEST IN 's3://cockroach-fixtures/backups/tpc-e/customers=500000/v22.2.1/inc-count=48?AUTH=implicit' AS OF SYSTEM TIME '2023-01-05T17:30:00Z' " returned
  | stderr:
  | ERROR: importing 14408 ranges: splitting key /Table/140/1/200001370685517/"CMPT": unable to find store 4 in range r14863:/Table/140/1/2000013{07912147/"CMPT"-76653510/"SBMT"} [(n6,s6):1, (n1,s1):7, (n10,s10):8, next=9, gen=453, sticky=1679448977.911505452,0]
  | Failed running "sql"

Run 9

This looks much like run 6, did not investigate further.

Jira issue: CRDB-25755

[blathers-crl]

cc @cockroachdb/disaster-recovery