Clarify maximum block size

[robcat]

The current implementation specifies a fixed maximum block size of 1MiB.

This is a pretty fundamental limitation, it should be explained better why this is the case (IPLD does not seem to be limited in the same way).

Motivation

When ipfs/specs#130 will land, it will be straightforward to automatically convert many hashes already used for integrity checks.

For example, Canonical could construct the "ubuntu releases" object (it would contain links to the historical live cd ISOs) and publish it in IPNS. It will just need to convert these hashes in the CIDv1 format, using the "raw data" codec.

This object would have a perfectly legitimate purpose (i.e. proving the integrity of the live cds), but ipfs will not be able to handle its leaves.

[jbenet]

Reasons for limitation:

• security - easy to DOS nodes without forcing small chunks
• deduplication - small chunks can dedup. big ones effectively dont.
• latency - can externalize small pieces already (think a stream)
• bandwidth - optimize the use of bandwidth across many peers
• performance - much better perf to hold small pieces in memory. hash along the dag to verify the whole thing.

the big DOS problem with huge leaves is that malicious nodes can serve bogus stuff for a long time before a node can detect the problem (imagine having to download 4GB before you can check whether any of it is valid). this was super harmful for bittorrent (when people started choosing huge piece sizes), attackers would routinely do this, very cheaply-- just serve bogus random data. smaller chunks are very important here.

the way i would approach what you describe (which is pretty cool) is that canonical should both:

• provide the leaf hash with CIDv1. ipfs could address it but not easily transfer it (there may be exceptions here, where it could be a tunable parameter).
• provide a new graph (chunked, dedup, etc), whose output matches the other leaf.
• that way ipfs could pull the chunked one, and verify the whole one afterwards.

you raise a good point though that we need to address how to look at objects and know whether to pull them when they may be too big.

[whyrusleeping]

Could this issue be moved to ipfs/notes please?

[robcat]

Just one tangential issue: when trying to add a big block, ipfs stops in the middle of the "add", like this:

$ ipfs add -s size-1048577 /path/to/big/file
779.62 KB / 2.42 MB [=========>-----------------] 31.49 % 0
19:03:08.355 ERROR commands/h: object size limit exceeded client.go:247

Maybe ipfs should refuse altogether to use a chunking size greater than BlockSizeLimit and/or give a better error message?

[ghost]

Maybe ipfs should refuse altogether to use a chunking size greater than BlockSizeLimit and/or give a better error message?

Yes agreed

[ethernomad]

Is the blocksize limit enforced in the network layer, or just in the chunker?

[mateon1]

@ethernomad Both. As seen here, the chunker limits blocks to 1MB, but libp2p limits blocks to 2MB. The second limit would be hit for directories with large amounts of files prior to directory sharding being implemented.

[alphaCTzo7G]

I am guessing as network bandwidth, processor speeds, RAM sizes increases, the optimum block size should also scale as it becomes easier to do all the things that are mentioned in #3104 (comment). Further, the volume of data/size of individual files that people want to share with ipfs will also scale.

Is there a way to increase/tune the blocksize for future? If not, wont the minimum block size become a bottleneck?

[Kubuxu]

No, maximum blocksize won't become bottleneck as IPFS is not a PoW blockchain, there is no limitation on how many blocks can be sent per second.

As @jbenet said: smaller blocksize allows for more deduplication and security but we are working on rising it either way to to characteristics of few version addressed ecosystems (git as one of the example) but it has to be done without any security trade off.

[Stebalien]

@alphaCTzo7G if you want to better understand this problem, take a look at this discussion on the IPFS discourse.

[whyrusleeping]

Closing this issue for now, please move further discussion to a new ipfs/notes issue, or discourse.