feat: send core "haves" bitfield on first connect #254
Conversation
|
This ended up being a bit more work than originally estimated. What this does:
The bitfield are compressed using run-length-encoding. The implementation is mainly copied from I am leaving the use of these bitfields to a follow-up PR. We will use them for evaluating sync progress / the need for sync before actually replicating all cores between peers. We can either:
I think (1) is the preferable option, we just need to make sure it happens before the bitfield changes for the peer. There is a lot in this PR. Some notes to help what to focus on:
As part of this PR I also updated how extensions are handled, by handing off decoding/encoding to hypercore (rather than encoding/decoding in our own code). There is also a small fix to handling the existing extensions (for sharing core keys) that was uncovered as part of this work. |
| // Export the appropriate version of `quickbit-universal` as the plain import | ||
| // may resolve to an older version in some environments |
There was a problem hiding this comment.
purely out of curiosity, what's this about?
There was a problem hiding this comment.
Don't actually know. This is copied from hypercore code
* main: feat: send core "haves" bitfield on first connect (#254)
* fix findPeer bug (also in #254) * feat: share all core keys via extension messages Fixes #251 This adds new fields to the project extension to share keys for other namespaces and updates tests which manually added cores (manual adding is no longer needed, but do need to wait for cores to be added)
* main: chore: condense setup code in MemberAPI tests (#267) chore: replace queries opt in MemberAPI with dataType (#266) feat: add getMany method to member api (#263) feat: add getById method to member api (#262) chore: Update @digidem/types and remove patch (#269) feat: share all core keys via extension messages (#264) feat: send core "haves" bitfield on first connect (#254) fix: invite.encryptionKeys should be required (#260) feat: Add set & get deviceInfo & datatype (#250) expose member namespace in MapeoProject (#253) blobStore.createReadStream should not wait (#243) update MapeoManager to return and handle project public IDs (#247) return content hash from blobApi.create (#242)
This implements a solution to #124 so that we can know how much data there is to sync before starting replicating the hypercores.
The code here implements a new custom hypercore extension to send core bitfields over the creator core replication stream, enabling the other peer to build up remote bitfields for each core for the connected peer. This can then be compared with a device's own bitfields to know what data needs to be sent.