[WIP] [RFC] add cryptographic hash to seekable format

[tomberek]

very much WIP, first time working on zstd, looking for feedback/thoughts

My intention is explicitly to trade some compression ratio in order to use the seekable format and perform content-addressing of each frame's uncompressed content. Perhaps a --rysyncable-frames mode? Adding a cryptographic checksum into the Seek Table would produce a format very similar to zchunk/casync/zsync but still be a valid .zst. This optimized format (similar to the niche served by Cloud Optimized GeoTIFF) could be hosted by a server supporting HTTP Range requests and clients can easily perform dedup, and binary diffs.

• update implementation zstdseek_compress.c, (status: PoC functioning)
• update implementation zstdseek_decompress.c (status: nothing)
• update examples/seekable_compression.c (status: framesize-based PoC functioning)
• provide metrics (roughly seems to increase size, including the new index, by ~10%)

Alternatively: can make a "Checksum Type" field out of a bit or two.

[Cyan4973]

The idea is interesting but likely needs to be fleshed out.
Closing, due to lack of activity.

[tomberek]

Still interested, but need assistance.

[silvanshade]

I'm also interested in this for chonker.

In that library I define a format which is essentially content-defined chunked slices of bytes compressed with zstd and hashed with BLAKE3:

• header: https://github.com/silvanshade/chonker/blob/40d435fe7b27db4c5d4e04b35469e0843c3a2448/crates/chonker/src/archive.rs#L19-L36
• metadata:
  https://github.com/silvanshade/chonker/blob/40d435fe7b27db4c5d4e04b35469e0843c3a2448/crates/chonker/src/archive/meta.rs#L19-L24
• chunks:
  https://github.com/silvanshade/chonker/blob/40d435fe7b27db4c5d4e04b35469e0843c3a2448/crates/chonker/src/archive/chunk.rs#L6-L12

It is similar to bita but about 4x as fast (due to use of rayon/zstd/BLAKE3/rkyv instead of bita's brotli/BLAKE2/protobuf).

I would consider using BLAKE3 over SHA256 or SHA512 since it can be significantly faster, even in single-threaded form (even when the SHA256 and SHA512 are hardware accelerated). On Zen4 I've seen around a 3-4x increase, depending on input size. And for multi-threaded BLAKE3, it can scale almost linearly up to memory bandwidth.

Having a hashed seekable format at the zstd level would make this approach much nicer, and open up more use cases.