Add support for BLAKE3 blobs #1630
redsolver
started this conversation in
Protocol (atproto)
Replies: 2 comments
-
a wild n0 team enters the chat. @dholms, is it peer pressure if it's really just the right thing to do 😉? |
Beta Was this translation helpful? Give feedback.
0 replies
-
Yup I completely agree with this 👍 In all likelihood, that's the plan when we start allowing larger blobs |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
At the moment all ATProto media blobs use the SHA256 hash function. This works well, because images and other blobs are limited to 1 MiB in size and fully downloaded before being processed. In the future, app.bsky or other (custom) lexicons will add support for new media types, including video. In most cases, 1 MiB blob sizes will no longer be enough for that and just increasing the limit isn't really a good solution either because it's not possible to only stream/download a small part of a video file and verify it. The BLAKE3 hash function solves this, because it uses a merkle tree construction and only needs a small amount of additional metadata (depending on group size) to verify a chunk anywhere in the blob. In addition to this feature, it's also a lot more efficient to compute and verify compared to SHA256, which is great for client and server performance when processing larger files. Implementations and bindings are available for all modern programming languages by now.
Here are some more details: https://github.com/BLAKE3-team/BLAKE3
Beta Was this translation helpful? Give feedback.
All reactions