Indexes are slurpy and allocates lots of memory

[Jorropo]

Indexes datastructure are fully copied into memory when parsing an index.

There is no limit on how big thoses copies were (because practical indexes used by peoples are bigger than what would be a safe limit).
So if you parse an index from untrusted user input, they can send you a really big index and memory hog or OOM you.

Would be nice if we have a way to parse untrusted indexes without opening yourself to OOMs.

This was first attempted in #312 (see discussion in https://github.com/ipld/go-car-priv/pull/2):

• 3923d31

But was reverted due to API breaking considerations (see discussion in the channel and https://github.com/ipld/go-car-priv/pull/18)):

• 4bc6774

See GHSA-9x4h-8wgm-8xfg

[email protected] also includes additional documentation regarding the dangers of consuming CARv2 index data from untrusted sources and a recommendation to regenerate indexes of CAR data from such sources where an index is required.

[masih]

Once streaming index read is implemented, re-add the tests that are removed here

[masih]

The index.ReadFrom tests are moved to run explicitly against that function instead of running as part of inspection. Once the streaming index read is implemented and used as part of inspection, those tests should also run as part of inspection as it was before.

[Jorropo]

@rvagg we do not use indexes in Kubo, and I have already too much things to do.
I cannot take on that work sorry.

I opened it because as far as I understood in the discussion we had indexes that allocates might be an issue, however boost was not able to do a refactor use io.ReaderAt instead (or / and was concerned of copying into bytes.Buffer due to performance issues I belive).

If consumers of indexes (boost) don't think this is an issue or don't care about it, then we can close this.