feature request: Support fetch over libp2p

[MarcoPolo]

I would like this to work:

fetch(https://QmPeerID.libp2p/some-http-api)

(Placeholder format, not sure what the url should look like. Suggestions welcome).

We could have a service worker intercept the fetch call, and make the HTTP request over a libp2p stream (like WebTransport, WebSockets, and, I think with the main page’s cooperation, WebRTC.)

This would allow a browser to use the exact same code to make requests to a normal https endpoint and a libp2p peer. No code changes required on the browser side. It effectively enables HTTP to be used in p2p environment.

For example, a browser user may see that my laptop has the content it wants, but it doesn’t have a domain name it could reach. If my laptop exposes the HTTP gateway api, the browser user could simply fetch the content from my browser and the service worker would handle the request over libp2p.

---

Alternative/Complementary ideas:

1. This could move even more logic to the service worker, such that the service worker can verify CIDs. Then browsers can simply fetch some cid and trust that it’s correct. (open question: can the service worker silently fail here?)
   a. The service worker could also get more freedom to use bitswap to fulfill the request.
2. This could be a library that integrates with existing service workers in case a user already has an existing service worker. Or stand on its own in case a user doesn’t have one already.

Related:

• Leverage Service Worker and WebTransport on Public Gateways ipfs/in-web-browsers#207 (comment)

---

FAQ

Q: Can’t we already do this with just a libp2p stream? Why do we need HTTP?
A: Yes, it is possible to use libp2p streams directly for communication between peers. However, integrating HTTP with libp2p in a browser context offers several advantages:

1. Compatibility with existing web infrastructure: By utilizing HTTP, you can leverage the existing web technologies, APIs, and tools that are built around it. This makes it easier for developers to adopt and integrate with their current applications, as they don't have to learn a new protocol or change their existing codebase significantly.
2. Developer familiarity: Most web developers are already familiar with HTTP and how to work with it in the context of web browsers. Using HTTP over libp2p streams enables developers to use their existing knowledge and skills, lowering the barrier to entry for decentralized web development.
3. Reuse of HTTP semantics: HTTP is a well-defined and widely used protocol that provides features such as caching, content negotiation, and authentication. By using HTTP over libp2p, you can take advantage of these features and build on top of an established, reliable protocol.
4. Browser support: Browsers have built-in support for HTTP, which means that using HTTP over libp2p allows for more seamless integration with browser APIs like fetch. This makes it easier to build applications that can be easily accessed and used by users without the need for additional plugins or extensions.

-FAQ answer by ChatGPT (proofread by me)

[achingbrain]

I would like this to work:

fetch(https://QmPeerID.libp2p/some-http-api)

Can you define some inputs and some outputs please?

-ChatGPT (proofread by me)

😩

[MarcoPolo]

Can you define some inputs and some outputs please?

// Input example 1. Classic HTTPS
const resp = await fetch("https://ipfs.io/ipfs/QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u?format=car")
const carBuffer =  await resp.arrayBuffer()

// output
String.fromCharCode.apply(null, new Uint8Array(buf)); // "<car framing>... Hello World..."

// input example 2. HTTP over libp2p
const resp = await fetch("https://QmSomePeerID.libp2p/ipfs/QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u?format=car")
const carBuffer =  await resp.arrayBuffer()

// output
String.fromCharCode.apply(null, new Uint8Array(buf)); // "<car framing>... Hello World..."

The client doesn't have to do anything different for this to work.

If you want some more inputs and outputs, I'm happy to ask ChatGPT to generate them ;)

[achingbrain]

This seems quite reminiscent of https://github.com/SgtPooki/helia-service-worker-gateway - it's an application that uses libp2p and bitswap (e.g. IPFS) in a service worker to intercept HTTP requests and serve the content from IPFS - have you tried it out?

If you want some more inputs and outputs, I'm happy to ask ChatGPT to generate them ;)

No, thank you. If I want my time wasted by a machine I'll play on my Switch.

[MarcoPolo]

Yes, this is related to that. This is more about doing the HTTP part over libp2p. https://github.com/SgtPooki/helia-service-worker-gateway would use this to be able to fetch content via bitswap or an HTTP API (either public https server or over libp2p).

Before even talking about service workers we could have a libp2p-fetch helper that takes a Request and returns a Response. Service workers can then use this if they want to extend native fetch or if they want to do this themselves.

---

Also to be clear the ChatGPT was just for the answer in the FAQ, not the rest of the issue. In retrospect that wasn't obvious, so I apologize.

[achingbrain]

Aha, awesome stuff. There's one fairly serious caveat - fetch in the browser cannot do full-duplex streaming, so the only protocols usable over something like this will be simple request/response protocols, or protocols that use two streams - one for input and one for output.

For the dual-stream approach you'd then need some extra accountancy at either end to tie them together/make sure they are closed at the same time, etc.

This doesn't sound like something that would be part of js-libp2p itself - you wouldn't use this in node.js, for example. More so it would be an app that used js-libp2p?

[achingbrain]

It might not be an app at all, it might even be a special transport that creates the service worker and uses fetch to do the stream creation and muxing transparently in the background?

[MarcoPolo]

fetch in the browser cannot do full-duplex streaming, so the only protocols usable over something like this will be simple request/response protocols

Yes. That's fine. Most HTTP apis don't use full duplex streaming (afaik). This isn't trying to run existing libp2p protocols over HTTP, this is trying to run HTTP over libp2p. The normal HTTP caveats will apply, and that's okay.

This doesn't sound like something that would be part of js-libp2p itself - you wouldn't use this in node.js, for example. More so it would be an app that used js-libp2p?

I'll probably make another repo to add to our collection (js-libp2p-fetch?). You might use this in node.js. You probably first want the server side of this which given an HTTP server handler, you make it accessible via libp2p (like what this PR does in go-libp2p).

I imagine many future datatransfer protocols will want to build on top of HTTP since there are many nice properties that come along with that when you have a publicly accessible server. Supporting HTTP on libp2p is a way to extend that datatransfer protocol so that anyone can run it, not only public servers with a domain name and certificate.

[aschmahmann]

https://QmSomePeerID.libp2p

Please don't use uppercase characters in anything that looks like a subdomain. It'll just cause headaches down the road that you don't need.

You're likely better off just doing what is generally done for IPNS keys and using the base36 CID encoded form of peerIDs https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md#string-representation. Note, don't use base32 since it's 2 characters over the 63 character limit for subdomain components with ED25519 keys (ipfs/kubo#7318).

---

🚲🏠 what should the URL look like?

Mostly flagging that it's probably something that should be figured out (in the specs repo?) and there might be a variety of opinions with inputs like "follows standards correctly", "is aesthetically pleasing", and "is closer to be usable in browsers given the current state of things".

You can see some of the info about how this works with IPFS here https://docs.ipfs.tech/how-to/address-ipfs-on-web/ and following the historical links. TLDR: there's a lot of context here and you're probably better off:

1. Just choosing one to demo with
2. Starting a discussion in the specs repo so relevant parties can discuss. We could do it here too, but I suspect the specs repo is more correct for this kind of thing.

[MarcoPolo]

🚲🏠 what should the URL look like?

Important to note that this part is all on the client side so it should be very easy to change. We don't have to get it perfect on the first try.

[MarcoPolo]

Initial pass: https://github.com/MarcoPolo/js-libp2p-fetch. This is just the logic for doing http over a duplex. The service worker would be a thin wrapper around this

[SgtPooki]

JS Colo 2024 discussion

basically done, in https://github.com/MarcoPolo/js-libp2p-fetch.

open an issue in that repo to move it to libp2p org.

Actually, it was already moved to https://github.com/libp2p/js-libp2p-http-fetch

Action items

1. close this issue. https://github.com/libp2p/js-libp2p-http-fetch exists