rust-ipfs?

[adamski]

I'm looking around for an IPFS implementation I can use in a cross-platform desktop/mobile application. I've been playing with https://github.com/Agorise/c-ipfs, but it seems to be incomplete and unstable from my testing.

I don't necessary need a full IPFS implementation, but would ideally like to be able to interact with nodes powered by go-ipfs. My use case is "offline first" application user data storage, and I'd like to replace my current dependency on Couchbase server infastructure with IPFS.

In order to use this library I would need to create a C++ wrapper for it, but it seems that things are moving and changing fast here so I wonder if its better to wait until the API has settled?

[whyrusleeping]

@adamski With a working libp2p, you can simply connect to any existing ipfs node and start speaking the protocols you want. You would have to reimplement some of those, but the ipfs protocols are actually quite simple. Bitswap is just a single protobuf that you send to request blocks and receive (the same protobuf) to receive those blocks. You could probably implement cpp-ipfs using rust-libp2p pretty easily ;)

[adamski]

Great, thats very encouraging, thanks.

[adamski]

@whyrusleeping Coming back to this - are there any examples of using libp2p to speak to IPFS? Are there any docs that explain some of the terms here, e.g. "protobuf"?

[adamski]

@tomaka are you able to help with this at all? Any helpful links to similar JS code you can point me to?

[whyrusleeping]

are there any examples of using libp2p to speak to IPFS? Are there any docs that explain some of the terms here, e.g. "protobuf"?

Other than ipfs? (ipfs uses libp2p to talk to other ipfs nodes)

What bits do you want to do? If you just want to be able to fetch data from ipfs nodes, you'll want to implement bitswap. The go implementation of that is here: https://github.com/ipfs/go-bitswap

The raw protobuf message is defined here: https://github.com/ipfs/go-bitswap/blob/master/message/pb/message.proto

[adamski]

Yes, I just want to fetch and provide data to other IPFS nodes. Thanks for the links!

[adamski]

Does anyone here have a rough idea of days required to implement this? I'm in the process of applying for a research collaboration grant with a local university where implementing rust-ipfs will be the main focus. Thanks.

[tomaka]

Implementing the bitswap protocol (the name of the protocol used by IPFS) shouldn't be very difficult, as far as I know (I haven't looked into it). However you would also need to implement record stores in Kademlia (#146) and create the CLI interface.

[adamski]

Thanks. I assume that the Kademlia record store is for local storage of other nodes on the network? Would that be based on https://github.com/libp2p/interface-record-store, or more similar to https://github.com/libp2p/js-libp2p-kad-dht?

Is there any documentation that explains some of this stuff?

[tomaka]

Let's move the discussion to #146

[dvc94ch]

I got bitswap working. The weird thing is that I always get a negative response to find_provider. By hardcoding kademlia.find_node(GO_IPFS_PEER_ID.parse().unwrap()) when creating a new behaviour I'll receive the block if I restart the program a couple of times, I assume that is part of the bitswap strategy of the go ipfs node that decides based on previous transactions to send or not send the block.
Is this a symptom of #146 ?
Mmh, sometimes it works sometimes it doesn't. I can't look up the rust-libp2p node from the go ipfs impl.
Next thing to try is if I can get it to work without dht, using mdns and pubsub should work on the local network as an alternative I think.
Some mdns weirdness going on: mdns-scan detects the rust-libp2p and the google chromecast, but not the go-ipfs or js-ipfs daemons.

[tomaka]

The weird thing is that I always get a negative response to find_provider.

This might be caused by #694

[gnunicorn]

@dvc94ch do you have your code available online somewhere?

[dvc94ch]

There are still some issues with it. I can publish it this week, just haven't gotten around to it yet.

[dvc94ch]

@tomaka Does the chat example still work in master? I've been struggling with some kind of deadlock problem where the first message is received, but the reply never is, so I've tried tracing calls to NetworkBehaviour and UpgradeInbound/UpgradeOutbound and comparing it to the chat example, but I can't get the chat example working anymore.

[dvc94ch]

Never mind, it inexplicably works again after rebooting. weird

[tomaka]

#910 should have fixed the example sometimes deadlocking.

[dvc94ch]

I believe the problem is with the mplex muxer. I enabled the yamux muxer and it works. This diff should break the chat application.

git diff
diff --git a/src/lib.rs b/src/lib.rs
index c917d18..2cf1d6e 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -219,7 +219,7 @@ pub fn build_tcp_ws_secio_mplex_yamux(local_private_key: secio::SecioKeyPair)
         .and_then(move |out, endpoint| {
             let peer_id = out.remote_key.into_peer_id();
             let peer_id2 = peer_id.clone();
-            let upgrade = core::upgrade::SelectUpgrade::new(yamux::Config::default(), mplex::MplexConfig::new())
+            let upgrade = mplex::MplexConfig::new()
                 // TODO: use a single `.map` instead of two maps
                 .map_inbound(move |muxer| (peer_id, muxer))
                 .map_outbound(move |muxer| (peer_id2, muxer));

[dvc94ch]

@gnunicorn Looking up providers in the dht still doesn't work, but two rust-ipfs clients can exchange blocks if they're on the same LAN. The go-ipfs and js-ipfs clients arent discovered via mdns.

https://github.com/dvc94ch/rust-ipfs

[tomaka]

The go-ipfs and js-ipfs clients arent discovered via mdns.

Go-ipfs and js-ipfs implement mDNS differently than the libp2p specs. I went with following the specs, but the consequence is that we don't discover the Go and Js nodes.

[adamski]

@dvc94ch thanks for your efforts!

I think, to make a Rust IPFS implementation as useful as possible it needs to be able to talk to existing Go and JS nodes. But perhaps that is outside the scope of rust-libp2p.

[gnunicorn]

Go-ipfs and js-ipfs implement mDNS differently than the libp2p specs. I went with following the specs, but the consequence is that we don't discover the Go and Js nodes.

@tomaka , if you know the difference(s) or can point to what you know to be incompatible, maybe you could put that into a ticket and we can provide a legacy-mDNS-crate to allow to connect to them, too?

[tomaka]

@tomaka , if you know the difference(s) or can point to what you know to be incompatible, maybe you could put that into a ticket and we can provide a legacy-mDNS-crate to allow to connect to them, too?

I don't actually know what they are doing. I sniffed their network traffic and it was very different from what the specifications mention. For the details one would have to read their source code.

I remember talking to someone from Protocol Labs about this, who mentioned that they wanted to eventually switch to what the specifications were.

[dvc94ch]

if you know the difference(s) or can point to what you know to be incompatible

I hacked the js-libp2p-mdns implementation to work with rust-libp2p.
The differences are listed below:

• service name is _p2p._udp.local vs ipfs.local
• host name is base32 encoded vs base58
• txt record is under additionals vs answers
• txt record data is dnsaddr=/multiaddr/p2p/peer_id vs peer_id

In addition to that the js-libp2p-mdns implementation sends A, AAAA and SRV records

[tomaka]

In addition to that the js-libp2p-mdns implementation sends A, AAAA and SRV records

Yeah, #734

[adamski]

What's the likelihood of js-libp2p-mdns changing to adhere to the specs anytime soon? I imagine it might break things if they change it..

For my particular use case, I don't mind having only nodes I set up being discoverable (local / remote data replication), but eventually I want to be able to have those nodes discoverable by JS and Go nodes and vice versa.

[gnunicorn]

I hacked the js-libp2p-mdns implementation to work with rust-libp2p.
The differences are listed below:

* service name is `_p2p._udp.local` vs `ipfs.local` 
* host name is base32 encoded vs base58
* txt record is under `additionals` vs `answers`
* txt record data is `dnsaddr=/multiaddr/p2p/peer_id` vs `peer_id`

Curious. Running go-ipfs, reporting unknown messages in our mDNS, I can see the query name as _ipfs-discovery._udp.local and l08d3t9uspkq8q9q6vhmjprlh1rgcj8c0rrf951fvkmq5b3xn5fckc1._p2p._udp.local (the beginning being the peer id I assume). Are go-ipfs and the js-version incompatible to one another 🤔 ?

---

Legacy service query received: Packet { header: Header { id: 51103, query: true, opcode: StandardQuery, authoritative: false, truncated: false, recursion_desired: false, recursion_available: false, authenticated_data: false, checking_disabled: false, response_code: NoError, questions: 1, answers: 0, nameservers: 0, additional: 0 }, questions: [Question { qname: Name("_ipfs-discovery._udp.local"), prefer_unicast: false, qtype: PTR, qclass: IN }], answers: [], nameservers: [], additional: [], opt: None }
Unknown message received: Packet { header: Header { id: 23734, query: true, opcode: StandardQuery, authoritative: false, truncated: false, recursion_desired: false, recursion_available: false, authenticated_data: false, checking_disabled: false, response_code: NoError, questions: 1, answers: 0, nameservers: 0, additional: 0 }, questions: [Question { qname: Name("l08d3t9uspkq8q9q6vhmjprlh1rgcj8c0rrf951fvkmq5b3xn5fckc1._p2p._udp.local"), prefer_unicast: false, qtype: PTR, qclass: IN }], answers: [], nameservers: [], additional: [], opt: None }
Unknown message received: Packet { header: Header { id: 0, query: true, opcode: StandardQuery, authoritative: false, truncated: false, recursion_desired: false, recursion_available: false, authenticated_data: false, checking_disabled: false, response_code: NoError, questions: 1, answers: 0, nameservers: 0, additional: 0 }, questions: [Question { qname: Name("ipfs.local"), prefer_unicast: false, qtype: PTR, qclass: IN }], answers: [], nameservers: [], additional: [], opt: None }

confirmed jsipfs deamon and ipfs deamon send mDNS messages but don't find each other 🤦‍♀️

[dignifiedquire]

I think @richardschneider worked on this interop last, maybe they can clear something up.

[gnunicorn]

Indeed a known issue: libp2p/go-libp2p#176 (comment)

[gnunicorn]

#968 is now able to discover legacy IPFS clients in js and go. However, as both of them are technically specific to the ipfs-domain, I am not sure we want to have that code in the regular libp2p code, but also don't really see a way to make it more generic or nicer. I am thinking of future-gating it...

[dvc94ch]

@gnunicorn maybe the best solution is the one all large corporations use. Write a spec for the rust-libp2p-mdns implementation, get it accepted in libp2p/specs and let every one else fix their code.

[tomaka]

Write a spec for the rust-libp2p-mdns implementation, get it accepted in libp2p/specs and let every one else fix their code.

There's already a spec and the rust-libp2p-mdns implementation hopefully follows it. We're basically at the last step.

[dvc94ch]

Some things are specific to rust-libp2p no? Like the service tag must be _p2p._udp.local and that dnsaddr must be a valid {multiaddr}/p2p/{PeerId} and that implementations must provide a TXT record with a valid dnsaddr.

[tomaka]

That's exactly what is specified here: https://github.com/libp2p/specs/blob/2f3e55d391338a32a7cb67e4e4aa7b78d02e0d8c/discovery/mdns.md

[dvc94ch]

Hadn't seen it because it's not in master... Thanks! So @gnunicorn and I found an issue where the multiaddr returned by the legacy mdns was unsupported which is fixed now and that the jsipfs node fails an identity check if you attempt to connect before the peer was discovered. I'm not sure what the next steps are other than waiting.

[folex]

I hacked the js-libp2p-mdns implementation to work with rust-libp2p.

@dvc94ch Hi! Do you, by any chance, have this hack available somewhere? I'm trying to connect rust-libp2p with js-libp2p, and having all kind of troubles. Incompatibility between Rust & JS mDNS is one of them.

[dvc94ch]

https://github.com/dvc94ch/js-libp2p-mdns/commits/master

[tomaka]

cc ipfs/infra#496