This repository has been archived by the owner on Mar 28, 2023. It is now read-only.
Large directories will break at the IPFS level #639
Assignees
Labels
feature
Feature or enhancement to openbazaar-go
Comments
|
@JustinDrake do you have more information on this? Like a reproducible example of this having problems? What happens when it gets to this size currently? Also are there any IPFS open issues on this you can reference? |
|
IPFS has a block size limit of 1MB (and libp2p has a block size limit of 2MB), so blocks need to be kept small. Directories right now are a single block, so as they grow the block size will go over the limit, and they will not be fetch-able on the network. The PR that implemented experimental sharding is ipfs/kubo#3042, and there's discussion for example here ipfs/specs#32 |
|
Ok @cpacia can we do some testing with the flag enabled? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
As currently setup, large directories will not work. The reason is that sharding is not enabled, and the directory indexes (without sharding) are stored in a single block, which is capped at 1MB. So stores with large
listings,ratings, orimagesdirectories will break.It's possible to enable directory sharding but that still hidden behind an experimental flag. In any case, large directories need testing.
The text was updated successfully, but these errors were encountered: