db: add ViaBackingFileDownload option to DownloadSpan

[dt]

No description provided.

[cockroach-teamcity]

This change is 

[dt]

There's some additional work to do here as discussed offline, but wanted to throw this up in case it is useful and/or if we want to merge it in a disabled state so that the CRDB-side or limiter etc work can be unblocked and then we come back to add the de-dupe and robust tests.

[RaduBerinde]

Left some comments. Like we discussed, it's ok to merge this for now even if we decide to switch to a provider-based operation later.

Reviewable status: 0 of 3 files reviewed, 2 unresolved discussions (waiting on @dt and @itsbilal)

---

compaction.go line 2759 at r1 (raw file):

	}
	metaCopy.FileNum = d.mu.versions.getNextFileNum()
	metaCopy.Virtual = false

This is hacky, we should do

if metaCopy.Virtual {
  // Even though this will be a virtual SST, we use the same number for the FileNum and the backing DiskFileNum.
  metaCopy.InitProviderBacking(base.DiskFileNum(metaCopy.FileNum))
} else {
  metaCopy.InitPhysicalBacking()
}

---

compaction.go line 2811 at r1 (raw file):

				n := min(size-pos, int64(len(buf)))
				readErr := r.ReadAt(context.TODO(), buf, pos)
				if readErr != nil && readErr != io.EOF {

io.EOF isn't an ok error here, if we got the size wrong we'd be writing garbage at the end