Peer-to-peer sharing (offline)

[chrisekelley]

Bunsen needs better offline support. It can save dats for viewing offline, but what if you want to share a dat while offline? We've experimented with it in the past, but sharing between two Android Bunsen instances while not connected to the Internet has not worked. So far.

Here is a thread from our super-long dat-gateway thread that illustrates a potential solution, using discovery-swarm-stream-java. This solution uses TCP connections to the peers (no uTP) and DNS and mDNS lookup (no dht)

---

rjsteinert commented 20 days ago
If browsers just need a discovery gateway to get what I think of as "calling cards" to the sources of archives, then I must say that's pretty rad. @RangerMauve Is it possible to be a peer in that paradigm and share the archive back? Similar I suppose to webtorrent.

Make a single page application that intercepts all dat:// URLS client side and redirects them to something service workers can intercept.

This sounds really interesting, especially the "service workers can intercept" part. @chrisekelley I think that negates part of the need for "fs provided by node" because you can download somefiles and then dynamically serve files from service worker cache right? For example, if you generate some random string in the browser memory, don't have a file on the file system called random-string.txt but serve that random string at that url from service worker, you can actually do an http request in the browser for . random-string.txt and it won't know it's not served from an actual server. I think the other part of why "fs provided by node" might be need is the storage limitation. Luckily, that storage limitation is not as bad in practice as it sounds in what documentation I've been able to find on it https://twitter.com/rjsteinert/status/974626460676829185. Note, there is a limit on storage in nodejs as well, granted it's probably going to be more space unless you run a hard drive like me with 95% full all the time.

@RangerMauve

RangerMauve commented 20 days ago
Since you're in the context of a cordova App, I actually have amazing news for you: The discovery swarm proxy thing is already available on Android thanks to @fsteff.

https://github.com/fsteff/discovery-swarm-stream-java

https://github.com/fsteff/DatDiscoveryWebview

This means you don't even need to set up a node.js process in the background!

can we still take advantage of Intents to open dat url's automatically

If you go with the single page application approach, then you can have the intents open http://whatevertheSPA is served from/{datkey}/{path} to have it detect the key and load whatever

Regarding storage, I don't think it's necessary to go with nodejs in order to make it usable.

For now, I was thinking that dats which you create are stored in indexeddb, and dats you didn't create are stored in memory.

Since you're in Cordova, you could probably make use of a cordova plugin for file storage and wrap it with random-access-storage

@fsteff

fsteff commented 20 days ago •
Note that service workers have the restriction that they must be served over HTTPS(!)
I have found some posts in forums where people claim to be working to it, but have not found a solution.
The only "easy" solution I can think of is serving the app as a PWA with the option to install a local gateway(-service) and providing a gateway in the internet as a fallback service.

Depending discovery-swarm-stream-java:
I have plans to create a "drag an drop" service for android that developers only have to add to their app and start it via an intent.

@chrisekelley
Member
chrisekelley commented 19 days ago
@fsteff - discovery-swarm-stream-java as a service would be great; I can wrap it up into a Cordova plugin pretty easily then.

@fsteff

fsteff commented 19 days ago
@chrisekelley - I hope that I have time to do create a first prototype next week. Would love to see this being actually used. Still, as I mentioned above - the thing with the service workers might not be a working, and without it discovery-swarm-stream would not be of much use in a browser.

[fsteff]

Offline sharing should already work with the current solution. The java port has only a subset of the features of the nodejs impl.
The only problem that I experienced sometimes was that mDNS did not work very well.

[rjcorwin]

Offline sharing should already work with the current solution

@fsteff By current solution are you referring to your discovery-swarm-stream-java library?

I'm also wondering why offline sync between dat archive on Android doesn't work using the dat module for nodejs. dat-ecosystem/dat#829

[fsteff]

@rjsteinert I am referring to the default nodejs solution of dat.
(Did not know about datproject/dat#829)

The only problem that I experienced sometimes was that mDNS did not work very well.

I recently had problems connecting to beaker and node-cli using mDNS. Also, last week there was a discussion on the IRC #dat channel about the problem that it only uses one network interface, which might be the wrong one. I too had this problem with discovery-swarm-stream-java, which I solved by sending the query on all network interfaces.

[rjcorwin]

@fsteff Here's a related issue in the Beaker Browser issue queue

• Beaker offline not discovering peer on LAN? #614

I also discovered this bug in Beaker today...

• Beaker fails to start when offline #1098

[RangerMauve]

Here's the issue tracking the IP address problem in dns-discovery in node: mafintosh/dns-discovery#16

Also, just a heads up, but there's discussion about having some of the capabilities of discover-swarm-stream being in the hypercore-protocol instead of having it in a custom protocol over discovery-swarm. dat-ecosystem-archive/DEPs#38

Not sure if it'll happen, but it might be a breaking change in the future.