-
Notifications
You must be signed in to change notification settings - Fork 1.2k
refactor: ⎆ Migration to pull-streams (from Node.js Streams) #403
Comments
Thank you for captaining this @dignifiedquire :) :) Let's make sure not to forget documentation :) |
Draft for the readme note.
|
before doing the work of chaning every module, i highly recommend you pick a small set and work with them-- see how they go. You wont understand how an abstraction fits a project until you play with it for a while. and def better to find that out with a small set than after changing everything. |
@jbenet we already did that as outlined at the beginning of this issue. I ported the minimal needed set to get secio in swarm working and see how things are working out, and they worked out great. Afterwards I talked with @diasdavid and we decided to move forward. The result of this is this endeveaour of moving all things into high quality ports. |
Sub. Let me know if I can help with READMEs. |
Thanks @RichardLitt ❤️ will ping you as soon as I have a consistent message ready. |
Missing from the list: |
added |
One more: libp2p/interface-transport#8 |
Status Update - The enchanted adventure of the missing backpressure mechanism.The migration to pull-streams is almost complete, we currently have PR's across every single module that required to be updated in order for the migration to be complete. These PR include the code, documentation is still lacking. There is one major blocker though, although we've been sucessful porting almost every single module to pull-streams, our stream-muxing stress tests are failing (fortunately we had them before this endeavour). We are still investigating the issue, having ruled out a bunch of the possible causes, we now have a strong conviction that the backpressure mechanism is failing when we convert back and forth pull and Node.js streams. Me and @dignifiedquire have been spending a considerable amount of time debugging, if you would like to help, I've created 3 PR with different experiments and have added a way to test it, so you can observe the failures by yourself. All of these are documented:
In addition to the fact that (apparently), backpressure stops working properly when too much wrapping happens, we also see the process running Out Of Memory when tested exhaustively. This can be because the backpressure mechanism doesn't work as it should (keeping a lot of memory floating around) or simply because pull-streams have a high memory fingerprint, making the memory run out just because of having too many instances (less probable). |
I finally had a breakthrough just now! I figured out why the pausing happens. The reason for it is that at some point the pull-stream side actually has too much data, and says "hey slow down" to the spdy-transport stream (the After this everything comes to a halt quickly due to the fact that the The only somewhat viable solution I found so far was to add add if (this.isPaused()) {
this.resume()
} to the I am not certain in any way that this is the correct fix, so I'm very happy to receive more input here. One important step I have not done yet is to create a test case using only node streams to recreate this problem with backpressure on the underlying socket with spdy-transport. |
I've created the above mentioned test and fix here: dignifiedquire/spdy-transport@0e9c847 |
@dignifiedquire where are the PR's for:
|
@dignifiedquire are there PR for the ipld-service? |
@diasdavid no there is not, I would suggest changing that when we actually integrate it. |
@dignifiedquire we've been directing some groups to check it out, we have to make sure it continues to work |
We did? Sure if people are actually using it we can update it now. I did not expect that |
@dignifiedquire yep :) I've made my CR to all the remaining repos, once all of those points are taken care (very little stuff), I'll do the merge and release \o/ :D |
Woooo! All the Pull Streams are in :D |
As noted in #362 I did the due diligence of confirming that using
pull-streams
all the way through solves our issues with secio. The result of this can be seen in libp2p/js-libp2p-switch#67. (Warning lots of module linking required to get this running)The next step is to move from the prototype into a fully tested version for all modules.
Below is a list of the modules that will need work.
ipfs
js-ipfs
- PR [WIP] Move to pull-streams #440interface-ipfs-core
- PR fix(files): don't expect a specific sorting of files ipfs-inactive/interface-js-ipfs-core#70js-ipfs-api
- delayed - Issue Migrate to pull-streams ipfs-inactive/js-ipfs-http-client#367js-ipfs-unixfs-engine
- PR [WIP] Pull streams ipfs-inactive/js-ipfs-unixfs-engine#64js-libp2p-ipfs
(Only testing) - PR pull ipfs-inactive/js-libp2p-ipfs-nodejs#17js-libp2p-ipfs-browser
(Only testing) - PR refactor: use pull-streams ipfs-inactive/js-libp2p-ipfs-browser#84js-ipfs-merkle-dag
- PR [WIP] refactor: move to pull-streams js-ipfs-merkle-dag#44js-ipfs-block-service
- PR refactor: Migrate to pull-streams js-ipfs-block-service#23js-ipfs-ipld
- PR refactor: migrate to pullstreams ipld/js-ipld#54js-ipfs-repo
- PR [WIP] Move to pull-streams js-ipfs-repo#76interface-pull-blob-store
(replacesabstract-blob-store
) https://github.com/ipfs/interface-pull-blob-storeidb-pull-blob-store
https://github.com/ipfs/js-idb-pull-blob-storefs-pull-blob-store
https://github.com/ipfs/js-fs-pull-blob-storejs-ipfs-bitswap
- PR [WIP] Pull streams js-ipfs-bitswap#25js-multistream
- PR [WIP] pull-streams multiformats/js-multistream-select#19libp2p
interface-connection
- PR Migrate to pull-streams libp2p/interface-connection#10js-libp2p-identify
- PR [WIP] move to pull-streams libp2p/js-libp2p-identify#2js-libp2p-secio
- PR [WIP] Refactor to use pull streams libp2p/js-libp2p-secio#8js-libp2p-spdy
- PR [WIP] Move to pull-streams libp2p/js-libp2p-spdy#16js-libp2p-websockets
- PR [WIP] Move to pull-streams libp2p/js-libp2p-websockets#18js-libp2p-tcp
- PR Migrate to pull-streams libp2p/js-libp2p-tcp#20js-libp2p-webrtc-star
PR Migrate to pull-streams libp2p/js-libp2p-webrtc-star#12js-libp2p-ping
- PR migrate to pull-streams libp2p/js-libp2p-ping#11js-libp2p-swarm
- PR [WIP] Add secio & move to pull-streams libp2p/js-libp2p-switch#67interface-stream-muxer
- PR [WIP] Pull-streams libp2p/interface-stream-muxer#7interface-transport
- PR https://github.com/libp2p/interface-transport#8Documentation
The text was updated successfully, but these errors were encountered: