Making ipfs.io gateway writable

[jbenet]

0.3.9 was just released, and it carried @mildred + @cryptix's patch to make the gateway writable.

I would like to make ipfs.io/api offer a writable portion, so that applications can be built against that as they would any other web service. This is mostly a crutch until the protocol is deployed in end user browsers + devices. But it likely will be useful for a long time as that way systems with HTTP support can easily add things to IPFS.

Concerns:

• how long to cache things (ideally should cache >10m, may be up to luck).
• should rate limit uploads (per IP addr) to prevent DOSing the system. (prob something like 100MB?)
• should have denylists in place

Once this is in place, things like IPFSBin and ipfs.pics could be just frontend webapps (+ pinning)

[victorb]

Question regarding this. If the content is only stored for 10m (or really, less than permanent), I'm not sure people are gonna be using this exclusively because they want to be able to make sure that the content stays on the site.

If I'm gonna be able to remove any IPFS daemon or backend from IPFSBin, I need to store content permanently, and it doesn't seem like this solves that really. Maybe I'm misunderstanding.

Pinging @cloutier (ipfs.pics) to have another opinion about this.

[davidar]

@victorbjelkholm I may be wrong, but I suspect that the idea is that the gateway will only be used for getting content into the IPFS network, and then it's up to some other node to make sure it's stored permanently.

[jbenet]

@victorbjelkholm

1. write through ipfs.io
2. pin with a pinning service (like openipfs, other ipfs-cluster things, etc)

[jbenet]

need a way to force --pin=false for all ops on the gateway. another option is to blow out all pins periodically that are not from a special separate list.

[cryptix]

@jbenet: need a way to force --pin=false for all ops on the gateway.

How would you like to set this? Config option oder flag on 'ipfs daemon' ?

(edit: maybe come over to ipfs/kubo#1917 for the specifics)

[cloutier]

@didiercf has made a working prototype of our backup system that pins every picture from our server to our local computers. Right now it requests hashes to our database, but it could be made to work directly with the ipfs api. It wasn't super high in our priority list, but if people are interested, we could release its source code sooner.

@victorbjelkholm I think it would be a good idea to keep our nodes running anyway. It would be more resilient to keep them and use ipfs.io only as a failover.

@jbenet are you talking about an IP denylist or a content deny list? We have a list of hashes we banned. We could release that, maybe not as is (it would be a bit counterproductive) but maybe as a variant of the bloom filter. If multiple lists agree on a hash, ipfs could automaticaly refuse to store it. This could be abused a lot though.

[ion1]

@cloutier:

A bloom filter-only solution sounds scary. They should only be used as an optimization step before checking against actual hashes (which are expected to identify data uniquely, not just probabilistically).

An alternative to publishing a list of hashes for all censored content is to publish the hashes of the hashes. You are not pointing to the content directly but checking a known hash against the list is easy.

[jbenet]

I would like to put https://github.com/jbenet/http2ipfs-web on ipfs.io -- what do we need to finish this issue? just enabling it?

[ghost]

I'd like to get ipfs/kubo#1917 and pin=false before making ipfs.io writable

[jbenet]

@lgierth 👍