c2c: add fingerprinting for internal testing

[adityamaru]

The following technical approach was replaced with:

• builtins: add crdb_internal.fingerprint builtin #91124)

...but the goal is the same.

Previous plan

crdb_internal.scan(crdb_internal.tenant_span($1)) returns the raw keys and values from the specified span. We would like to extend this generator to return timestamped ordered revisions of each key. While this is useful as a standalone tool, the motivation behind this change is to drive the on-demand fingerprinting that we are developing for c2c replication. At a high level this primitive will be executed by each processor on a pre-defined chunk of spans, the output of which will be fed to a checksum algorithm and sent downstream in the DistSQL flow.

As part of this issue, we should investigate whether we can add a mode to ExportRequest (KV request that is already capable of reading and returning timestamp-ordered revisions of keys) that does not write these keys to an SST but returns them in a kvBuf slice that we can read from. We should benchmark, and trace what parts of ExportRequest that are being used during backup are "slow" and "unnecessary" for simply reading and returning revisions of all rows in a given timebound.

Epic: CRDB-21075

Jira issue: CRDB-20208

[blathers-crl]

cc @cockroachdb/disaster-recovery

[adityamaru]

From an offline conversation with @dt and @stevendanna:

1. batcheval: ExportRequest should use a kvBuf to transport bytes instead of an SST #90450 is likely not a hard requirement for using ExportRequest as the driver to read key revisions.

2. What if we pushed down hashing into evalExport?

Until now we were planning to use a primitive such as crdb_internal.scan() to first fetch all the KVs with revisions in a given window of time, and then hash batches of those KVs. This would likely be done in a DistSQL flow that uses PartitionSpans to divvy work based on the leaseholders and then spins up processors on each node of the source cluster to hash their respective batches. The batched approach means that we would have to preserve the batching when generating the fingerprint on the destination cluster too since the hash would depend on the order of the keys. While this gives us more observability into what batch our fingerprinting failed on there is a simpler and potentially cheaper alternative we would like to consider.

Each ExportRequest constructs an SST and ships it as part of the ExportResponse. For the purposes of fingerprinting, we could teach ExportRequest to funnel KVs into an fnv64 hasher instead of the sstWriter and return a XOR'ed hash of the span as part of the ExportResponse. This would eliminate the cost of transferring SSTs over the wire. Furthermore, ExportRequests are splittable and so we can lean entirely on the DistSender to split, send and combine all the responses (XOR the hashes) without setting up a DistSQL flow. Since we are hashing every KV (along with its timestamp) the outputted hash becomes order agnostic and can be run on the destination cluster without having to maintain any complicated batching logic.

[adityamaru]

This can be marked as done once some of our C2C tests uses the crdb_internal.fingerprint builtin.

[adityamaru]

Based on an oflfine discussion with @livlobo we are closing the epic linked in this issue since crdb_internal.fingerprint has been implemented and tested. Fallout from tests (existing and future) that will use this tool should be tracked in the more appropriate epic linked here - #93078