Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[POC] One-way block streamer #15

Merged
merged 18 commits into from
Jan 25, 2023
Merged

[POC] One-way block streamer #15

merged 18 commits into from
Jan 25, 2023

Conversation

plauche
Copy link
Collaborator

@plauche plauche commented Dec 16, 2022

This block-stream application provides the foundation for creating and exploring a new data transfer protocol designed for the IPFS-in-space scenario.

The current implementation is pretty simple, it reads in the contents of a file, breaks up the contents into blocks, and transmits those blocks in UDP packets to a receiver. The blocks are serialized into binary data using the Parity SCALE format. The receiver listens for the stream of blocks, attempts to find the root block, and then waits until all links in the root are satisfied before assembling the file. This simple and naive approach to IPFS data transfer is intended to lay a foundation of point-to-point block streaming to be iterated on in future project milestones.

Things which may still be included:

  • Unit testing
  • Verifying data & hash of individual blocks
  • Correctly handling all out-of-order messages. Currently "blocks" sent out of order are fine, but the ordering of chunks and other messages is not handled.

Things intentionally not included (in this POC):

  • Formal protocol messages and behavior
  • Block requests or any behavioral responses
  • Chunking the file into multiple levels to reduce total link count in each root block

@plauche plauche marked this pull request as ready for review January 10, 2023 18:11
@plauche
Copy link
Collaborator Author

plauche commented Jan 10, 2023

I briefly experimented with adjusting the degree on the FileBuilder to reduce the number of links in each root block, and found that, while it does decrease the root block size by decreasing the width of each level, it still is not enough to fit root blocks inside of the lab radio constraints, which maxes out at 60 bytes. With a degree of 2, each root block contains three CIDs, each of which is 36 bytes, with the whole block ending up at 113 bytes. May need to investigate alternative CID hashing settings to shorten them, or other CID optimizations.

I think I will move optimizations for the 60 byte packet limit into a follow-up PR focusing on that.

@plauche
Copy link
Collaborator Author

plauche commented Jan 13, 2023

I've ended up working in an idea to send blocks incrementally and that has allowed me to send all blocks (including the root) inside of the radio packet size limit!

@plauche plauche merged commit 6e6ace0 into main Jan 25, 2023
@plauche plauche deleted the poc/one-way-block-stream branch January 25, 2023 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant