Missing features for subdomain based gateway

[lucidNTR]

The subdomain approach solves many issues for ipfs hosted webapps but i encountered the following missing features/ new issues:

given a web application deployed to ipfs and the root folder published to ipns
at http://bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy.ipns.localhost/

• it is impossible to get a json of the folder listing even when the gateway is configured to allow
  /api/v0/ls
  a workaround could be to manually create a manifest file, but this could not contain the root hash of the folder itself, as that would change the hash again. There is also no other easy way to get the current root hash because the index.html is served from the root, so a head request to http://bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy.ipns.localhost/ will have the etag of the index.html in the root, not the hash of the root itself

• it is impossible to get the folder contents by hash instead of by folder path, this is a huge issue, because it makes infinite caching impossible, which is one of the big selling pints of ipfs.
  without the subdomain gateway i would serve the webapp at:
  someogateway.com/ipns/
  and then reference the resources inside the webbapp with
  someogateway.com/ipfs/<ipfs hash of image 1 etc.>

Proposed solution:
i propose 2 changes to subdomain gateway behaviour

1. requests to a "folder", especially the root of a folder published to ipns such as http://bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy.ipns.localhost/
   is served the same response as https://ipfs.io/api/v0/ls?arg=bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy
   IF the accept content type header is set to "application/json" that way there is no interference with existing gateway behaviour and with serving the index.html file

2. ipfs gateways expose the /ipfs/ prefix that is available in non subdomain based setups also at the subdomain so it is possible to request resources by hash like this:

http://bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy.ipns.localhost/ipfs/<ipfs hash of image 1 etc.>
the ONLY downside is that it is not possible to have folders named ipfs or ipns in the first folder level of a folder published to ipfs AND reference that folder or subresorces by path. But as the existing behaviour basically breaks caching completely compared to the pre subdomain gateway i think this would be acceptable

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[lucidNTR]

security notice: the proposed change introduces a cors attack vector, if an attacker manages to make you visit your_ipns_application_root.ipfs.gateway.com/ipns or ipfs/attackers_evil_application/

so i would say exposing /ipns/ would be too risky, and the hashes available in the /ipfs/ endpoint should be whitelisted by default to only hashes inside the current sub tree .

[Stebalien]

Can you not remove the CID.ipfs. part of the domain and make the request against the root domain of the gateway?

WRT the second issue, files will still be cached via the ETAG and will also be cached on the gateway itself. But it will cost you a round trip.

[lidel]

it is impossible to get a json of the folder listing even when the gateway is configured to allow /api/v0/ls

As @Stebalien suggested, subdomain gateway supports /api on the root hostname: https://dweb.link/api/v0/ls?arg=QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm

expose the /ipfs/ prefix that is available in non subdomain based setups also at the subdomain [..]

No, that would defeat the purpose of Origin isolation and add unnecessary complexity.

Arbitrary CIDs can be loaded from the parent hostname like this: https://dweb.link/ipfs/QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm (returns redirect to a subdomain)

@lucidNTR is that not enough for your use case? what is missing?

Requests to a "folder", especially the root of a folder published to ipns such as http://bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy.ipns.localhost
is served the same response as ipfs.io/api/v0/ls?arg=bafzbeibxww7y6v5metkgmxeqkvls74uqvptbhommrb5b7xianigcicdvuy
If the Accept content type header is set to "application/json" that way there is no interference with existing gateway behavior and with serving the index.html file

I like this idea, it was already suggested in #3543 (comment), but there is one caveat: doing so would leak the current schema of /api/v0/ls into Gateway. What happens when we change the API and introduce /api/v1/ls? I think we need to include API version somehow (eg. Accept: application/json;version=0) so we are able to return error when client asks for API responce in a version that is no longer supported.

[lucidNTR]

Can you not remove the CID.ipfs. part of the domain and make the request against the root domain of the gateway?

-> unfortunately that is not possible as CORS specifies a root of a subdomain as a completely seperate domain. or do ipfs gateways automatically configure some CORS policies for root domains from sub domains, in that case this issue would be solved and acceptable but i could not see documentation for this behaviour...

WRT the second issue, files will still be cached via the ETAG and will also be cached on the gateway itself. But it will cost you a round trip.

-> the problem is not that caching will not happen, but that infinite caching setups, that rely on immutable paths, and that are crucial for IPFS hosting are not possible. Basically i think content adressable storage is at the very heart of ipfs, not enabling ipfs webapps to address their own resources as content hash would be like apple not allowing mouse support in a mac app. Its important for so many reasons, but the impossibility for this caching configuration is the best example why this is a real problem. again, if ipfs gateways would have a publicly documented automatic CORS setup of allowing sub-origin requests from ipns/ ipfs subdomains, this issue would also be solved.

[lucidNTR]

I like this idea, it was already suggested in #3543 (comment), but there is one caveat: doing so would leak the current schema of /api/v0/ls into Gateway. What happens when we change the API and introduce /api/v1/ls? I think we need to include API version somehow (eg. Accept: application/json;version=0) so we are able to return error when client asks for API responce in a version that is no longer supported.

i see the versioning issue, but maybe its better to either add the version as a hheader, or add the version into the json response, as in a path with version suffix, there would be conflicts with folder names. But this sounds very doable to me... Would really love this feature, as it would make deployments of my ipfs apps simpler and less error prone

[Stebalien]

or do ipfs gateways automatically configure some CORS policies for root domains from sub domains, in that case this issue would be solved and acceptable but i could not see documentation for this behaviour...

The gateway allows cross-site (read-only) requests to a subset of the API (includes get, ls, etc.).

-> the problem is not that caching will not happen, but that infinite caching setups, that rely on immutable paths, and that are crucial for IPFS hosting are not possible.

Public gateways are designed as a gateways into IPFS, not a replacement for running an IPFS daemon locally. You're always going to miss out on many of the benefits of IPFS if you're using someone else's gateway. When running a local IPFS daemon and using a local gateway, an extra round-trip to the go-ipfs daemon isn't really going to matter.

Directly pointing to content by hash in HTML documents is something you should avoid, actually. IPFS doesn't understand HTML documents so ipfs pin add /ipns/my.website.com won't follow these links.

---

Please feel free to continue asking questions and discussing, but I'm going to close this as "won't fix" for now as requesting resources cross-origin is still possible.

[lidel]

@lucidNTR do you mind opening a separate issue to track proposition (1)?
(support the accept content type header set to "application/json")

Creating one issue per topic is the only way to ensure ideas dont get missed in discussions like this one. Thank you!

[lucidNTR]

@Stebalien After thinking about the issue my main takeaway is this: I was thinking of a future where normal users would pick a ipfs gateway for them (eg. cloudflare, google, amazon) and browsers are configured to use them as currently firefox does with "dns via http", but your stance is that every user should have his own gateway with his own configuration. In my situation the standard behaviour of ipfs gateways such as what api is exposed and what cors policies are used is crucial for ipfs apps to run everywhere, in your future this is a non issue because every user can just adjust his gateway configuration according to his needs.