Browser support

[aulneau]

This is such a cool project! I started playing around with implementing this in a next.js react application, but found that certain dependencies don't really work in the context of a browser. Do you have any ideas on how to replicate or replace certain libs in this package to work universally?

Thanks!

[RangerMauve]

Yeah, the hard part is getting all the hyperswarm features into the browser. I've got hyperswarm-web working, mostly, but it's missing the connection deduplication code.

Try using webpack to compile your project, but replacing hyperswarm with hyperswarm-web. From there see if there are any methods missing in hyperswarm-web and send a PR for stubs on those methods (or actually implement the functionality if that seems doable).

[aulneau]

@RangerMauve thanks for the quick reply!

Yeah -- actually none of the hyperswarm specific stuff is an issue wrt to using in the browser. I had aliased your -web package and that worked great. The dependency on noise-peer is what is causing much of the issues. emilbayes/noise-peer#2, due to their underlying dependencies on sodium and it not having certain encryptions ported to javascript. https://github.com/sodium-friends/sodium-universal#compatibilty. I don't have enough context or domain knowledge to know how to improve or swap out these deps with more browser compatible ones.

[RangerMauve]

Ah, yes. I think this is the same issue we had in dat-sdk, try checking out the webpack config there which replaced sodium-native with a fork by Geut. That should hopefully work.

[aulneau]

That got it further along. I had to make some modifications to @geut/discovery-swarm-webrtc/lib/utils

const toHex = buff => {
  if (typeof buff === 'string') {
    return buff;
  }

// this is new, previous Buffer.isBuffer would fail with a `uint8array` 
// being passed here via noisePeer.keygen();

  if (buff && buff.buffer) { 
    const str = Buffer.from(buff.buffer).toString('hex');
    return str;
  }

  throw new Error('Cannot convert the buffer to hex: ', buff);
};

[RangerMauve]

I think @mafintosh or @martinheidegger recently mentioned that sodium-javascript got updated with the missing crypto functions. Maybe it'd make sense to try using that instead of the @geut fork?

[aulneau]

I think the @geut/sodium-javascript-plus fork is working correctly, the issue seems to be coming from within hyperswarm-web and its dependency on @geut/discovery-swarm-webrtc. I'll see if I can a minimal repo set up to show what's happening

[aulneau]

See here -> https://github.com/aulneau/p2plex-next.js

Had to change this: https://github.com/aulneau/p2plex-next.js/blob/master/vendor/discovery-swarm-webrtc/lib/utils.js#L6-L9 to accept the buffer correctly. The next error which I have a screenshot above is linked to this line of hyperswarm-proxy https://github.com/RangerMauve/hyperswarm-proxy/blob/master/messages.js#L80

I feel like it's related to buffer and not using the right one, or something along the lines of that.

[RangerMauve]

Hmm, is p2plex.publicKey a Buffer?

[aulneau]

Hmm, is p2plex.publicKey a Buffer?

It is a Uint8Array

[RangerMauve]

Cool, I think it's a Buffer when running in Node.js, so it'd be good to convert it to a buffer inside the key generation in p2plex. That's probably what's causing all these weird issues.

Try changing the get publicKey() function to look like this:

get publicKey () {
    return Buffer.from(this.keyPair.publicKey)
}

[aulneau]

That moved things forward!! This seems to be an issue now:

onstatickey: (remoteStaticKey, done) => {
        const publicKey = Buffer.from(remoteStaticKey);
        done();
        const dropped = info.deduplicate(this.publicKey, publicKey);
        if (dropped) return;

[RangerMauve]

I think for that we need to add some stubbed methods for the info object coming out of webrtc.

This info object is defined here: https://github.com/RangerMauve/hyperswarm-web/blob/master/index.js#L53

I think I've stubbed out the deduplicate method in hyperswarm-proxy already: https://github.com/RangerMauve/hyperswarm-proxy/blob/master/client.js#L191

[RangerMauve]

Actually, I'm not sure why that might be happening. Is there a stack trace you could post?

[RangerMauve]

It could be that we need to change this.keyPair to have buffers for secretKey and publicKey using something similar to the getter. Like this.keyPair.publicKey = Buffer.from(this.keyPair.publicKey) and similar for this.keyPair.secretKey or whatever it's called

[aulneau]

yep, this seems to be making it work.

  get keyPair() {
    const _keypair = this.opts.keyPair || noisePeer.keygen();
    return {
      publicKey: Buffer.from(_keypair.publicKey),
      secretKey: Buffer.from(_keypair.secretKey),
    };
  }

Running this

const plex1 = p2plex()
const plex2 = p2plex()

plex1.on('connection', (peer) => {
	peer.receiveStream('example').on('data', (data) => {
		console.log('Got data from', peer.publicKey, ':', data.toString('utf8'))
		plex1.destroy()
		plex2.destroy()
	})
})

plex2.findByPublicKey(plex1.publicKey).then((peer) => {
	peer.createStream('example').end('Hello World!')
})

Does not seem to cause any data console logs though. Additionally, the default wss server is down -> wss://hyperswarm.mauve.moe

[RangerMauve]

Dang. 🤔 Are WebRTC connections not going through, do you think?

[aulneau]

I'll have to play around with it more, seems like things are happening via the wss connection:

You've been so helpful, thank you.

I wonder if you'd be open to help out getting my mental model of how this could function within my app? It's a decentralized document editor built with blockstack auth and storage. I think I'll have to generate the keypairs that this uses for each user, and persist that, and then when they enter a document they would do something along the lines of peer.createStream(doc.id), at least that's how it's currently set up with the webrtc stuff I have now. Eventually once I understand this better, I'd love to write a provider to use p2plex for yjs.

[martinheidegger]

Since the last release sodium-javascript has crypto_aead_chacha20poly1305P support, but crypto_secretstream_xchacha20poly1305 is still missing, which is required for secretstream-stream, a dependency of noise-peer.

[RangerMauve]

@aulneau That sounds like a good plan TBH. 😁 p2plex will also be handy got multiplexing across documents and stuff.

By the way, would you be down to submit a PR with whatever changes you added to hyperswarm-web / p2plex that made things work for you?

[aulneau]

I have found another issue unfortunately, this time related to the use of noise-peer, which uses secretstream-stream. There is a reference to exports.HEADERBYTES = sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES which is undefined in browser usage. Do you have anymore leads related to how to switch out noise peer for something more web compat?

Best
t

[RangerMauve]

Could the upgrade of noise-peer have affected this? Try using the older version and see if that helps.

[okdistribute]

Could it be that upgrade is required for compatibility?

[aulneau]

Could the upgrade of noise-peer have affected this? Try using the older version and see if that helps.

Nah, it was the next issue regardless of version bump. Seems like noise-peer has used secretstream for a long time.

I don't really know the difference between crypto_secretstream_xchacha20poly1305 and crypto_aead_chacha20poly1305P is. Would there be a way to alias the one that is currently implemented in sodium-javascript?

[RangerMauve]

Jeeze, yeah I have no clue. 😅

I'm also down for switching the encryption layer with whatever works instead.

[okdistribute]

@aulneau worth opening an issue and pinging emilbayes or tinchoz

[aulneau]

Jeeze, yeah I have no clue. 😅

I'm also down for switching the encryption layer with whatever works instead.

@RangerMauve would this possibly work? https://github.com/paulmillr/noble-secp256k1

[RangerMauve]

@aulneau It doesn't look like that library supports streaming encryption, but otherwise it looks good enough. 🤷

[siman]

It would be cool to have this working in a browser 🚀