Manifest files for unixfs loaded via HTTP Gateway

[lidel]

This is a placeholder issue for creating a spec for "content root's manifest files" which would be part of the data published on IPFS (eg. /ipfs/CID/_something or /ipfs/CID/.well-known/ipfs/foo) that provides hints to HTTP Gateway around things like redirects, content types and custom HTTP headers.

Related reading:

• Proposal: manifest file for the IPFS gateway
• Proposal: native redirects config
• Content Type set by HTTP Gateway

Possible reuse of:

• _redirects (prior art in netlify docs
  • DONE
    • IPIP: https://specs.ipfs.tech/ipips/ipip-0002/
    • Specification: https://specs.ipfs.tech/http-gateways/web-redirects-file/
    • Docs: https://docs.ipfs.tech/how-to/websites-on-ipfs/redirects-and-custom-404s/
• _headers file (prior art in netlify docs and cloudflare docs)
  • TBD

Feel free to discuss/post ideas below, or open PRs with IPIP drafts that could be reviewed independently.

[BigLep]

Related are notes from the 2021-11-16 IPFS operators meeting: https://hackmd.io/MiYYUCR-RWqhAKfJ_B15wA

[justindotpub]

Hi @lidel. I started working at Fission this year and have recently been tasked with implementing redirects support in go-ipfs. @cbrake started working on this in this PR 🙏 and I plan on continuing the work in this PR.

If you don't mind, I'd like to make sure I understand the process here as far as specs and issue management go so I don't cause any annoying friction.

From the issues I've read through it appears that it was decided to split out redirects support from manifests support when Cloudflare announced Cloudflare Pages with _redirects support, similar to Netlify's _redirects support. This issue I'm commenting on appears to be specific to spec'ing out manifest support. Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?

Regardless of whether or not manifests and redirects are separate specs, my assumption is that the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec.

Assuming I'm on the right track, I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.

Thank you in advance for any feedback and guidance. If you have any other tips on the code base or any plans you had for how the code should be reorged (I see you've been doing some refactoring recently) I'd love to hear them.

Thank you! 🙏

[BigLep]

Hi @justincjohnson. Thanks for reaching out here. @lidel is great at seeing GitHub notifications and processing them. I don't know how many others are though. If you don't get a response in the next day or so, feel free to start a thread in #ipfs-operators in FIL Slack or #ipfs-dev in IPFS Discord.

[lidel]

Hi @justincjohnson, really appreciate reaching out!

I believe tldr from the prior discussions is that we translated requirements into two sibling features that will improve the devexp around unixfs website hosting using IPFS gateway:

• _redirects (prior art in netlify docs, also discussed in Proposal: native redirects config kubo#7392 (comment))
  • does not seem to be controversial, ready to be worked on
  • mvp should support:
    • relative and absolute redirects
    • "catch-all" redirect for PWAs
• _headers file (prior art in netlify docs)
  • bit more tricky, as we need to be very careful which headers are allowed where
  • comes with some open questions around which headers could be specified without X-foo prefix.

Feel free to work on both, but focusing on _redirects first is a good idea. I am happy to guide / help with reviews. 👍

the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec. [..]
I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.

Yes, this plan sounds good.

There are caveats around both headers and redirects that we will fully understand while writing sharness tests for them, so specs may change multiple times before we finish the mvp implementation. It is fine to keep discussion around go-ipfs PR and wait with specs until the dust settles.

Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?

We have various gateway improvements in flight, so it is better to have separate specs for now.
When you feel like writing the spec, just open a PR against ipfs/specs that adds it as a standalone file under gateways/REDIRECTS_FILE.md – this way we can get it done without being blocked on anything.

ps. I left some quick comments in ipfs/kubo#8816

[justindotpub]

Fabulous! Thanks for the guidance @lidel. I'll get started.

[justindotpub]

For the benefit of anyone coming here looking for information on redirects support, see ipfs/kubo#8890.

[eth-limo]

@lidel a _headers file would be fantastic. At eth.limo, we've had multiple users request the addition/removal/modification of the following (user defined) headers:

User defined headers

Header	Rationale
content-security-policy	useful not only for iframing protection but also for better dependency/resource security and XSS defense. Ideally dApps/dWebsites should have whitelisted domains and in-line script hashes within the html bundle. Encouraging better version pinning could help defend against things like the Ledger JS library compromise from last year.
cross-origin-resource-policy	best practices for cross-origin isolation. Additionally, advanced features such as SharedArrayBuffer require one or more cross-origin-* headers to be properly configured. See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer
cross-origin-opener-policy	^^^
cross-origin-embedder-policy	^^^
permissions-policy	Should be customizable by the content publisher, depending on what features are required.
x-frame-options	Configurable depending upon the need for iframing. However, this can also be addressed via the content-security-policy frame-ancestor directive.
x-xss-protection	legacy, but can provide a "belt and suspenders" approach for XSS protections in conjunction with a well defined content-security-policy. At a minimum, only the 1; mode=block directive should be configurable due to potential vulnerabilities arising from botched filtering: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection#vulnerabilities_caused_by_xss_filtering
referrer-policy	useful for dApps/dWebsites interested in some privacy mitigations for public gateways.

This would greatly improve the functionality of static dWebsites delivered locally, or through a gateway.

Gateway operator defined headers

There are many more, but this is a good starting point. These should not be configurable by the end user as they can globally impact the operation and quality of the service.

Header	Rationale
strict-transport-security	as mentioned elsewhere, this is really designed for the gateway operator to configure. It has global ramifications and should not be configurable by content owners.
clear-site-data	another header that can potentially alter top-level origin settings. Ideally the gateway operator would configure this to automatically wipe cookies issued by other subdomains in the absence of inclusion on the Public Suffix List (PSL)
cache-control	likely something the gateway operator should control since caching is directly related to resource usage (bandwidth, etc...)

[lidel]

Thank you, useful feedback.
Agree, allowing arbitrary headers is risky, especially in the future when new things are added to the web platform.

To get this moving froward, we could start with a safelist of allowed headers.
Ones that allow website owners to adjust security for their Origin sgtm.

For next steps here, interested parties could

• open an IPIP similar to one that added _redirects:
  • https://specs.ipfs.tech/ipips/ipip-0002/
  • https://specs.ipfs.tech/http-gateways/web-redirects-file/
  • It would add _headers to web gateways with Origin isolation (subdomains, DNSLink gws), and ip[fn]s:// handlers), similar to one that added _redirects, and have it a list of headers that are allowed to be customized.
• write reference implementation for boxo/gateway library (which ships in IPFS Desktop/Kubo/rainbow/Brave)

If anyone is interested in kicking this off and providing implementation and specs, I am willing to lend necessary review time. 👍
Sponsoring this work via https://ipshipyard.com/contact-us is also an option.