fix: clean up tree definitions

[vmx]

There were quite a few public type definitions, that were mostly replaced by the SectorShape* types. This commit cleans them up and moves them around if appropriate.

This should make the code easier to follow and the public API surface smaller.

BREAKING CHANGE: BinaryLCMerkleTree, BinaryMerkleTree, BinarySubMerkleTree, LCMerkleTree, LCStore, MerkleStore, MerkleTree, OctLCMerkleTree, OctLCSubMerkleTree, OctLCTopMerkleTree, OctMerkleTree, OctSubMerkleTree, OctTopMerkleTree, QuadLCMerkleTree and QuadMerkleTree are removed from the public interface.

---

This PR was triggered by a question about optimizing the GPU usage. I then dug deeper into the code to find out which tree sizes we are actually using and then found out that there are lots of types we don't actually use.

[storojs72]

I support this change, as it removes multiple confusing types of trees, since we have 3 various types of memory layout (Memory, Disk, LevelCacheStore) and 3 levels of tree's arity (Base, Sub, Top).

A small remark - I think that comment about arities is partially correct. Up to you, but I would rewrite it approximately as following:

The base tree arity defines structure of the tree (binary / oct tree) instantiated directly over the data represented as a set of leaves. E.g. DiskTree<_, _, _, U2> means definition of binary disk-backed tree, where each upper-layer node is actually a hash of two underlying nodes / leaves.

The sub-tree arity defines structure of so called "composed" tree - a first layer of base trees composition. E,g, DiskTree<_, _, U8, U2> means that we combine 8 binary trees into a single composed tree under root computed via hashing 8 binary trees' roots.

And, finally, the top-tree arity defines structure of so called "composed-composed" tree - a second layer of base trees composition. E,g, DiskTree<_, U4, U8, U2> means that we combine 4 composed trees (each combines 8 binary trees) into a single composed-composed tree under root computed via hashing 4 composed trees' roots.

[vmx]

Thanks @storojs72 for the clarification. Let me try to repeat it, to make sure I understand it correctly.

The base arity is used for all levels from the leaves up to a certain point (I originally misunderstood it as the base arity only be used for a single level).

The sub-artity and the top-arity are single levels only.

[vmx]

I had a another look at it. What happens is: You hash the tree with the base arity. We have trees where that would lead to multiple roots (64GiB sectors are an example). In order to get a single root, a sub-=/top-arity is specified. It will then hash the remaining nodes into a single root node.

Though now I have a question: what is the difference between those two:

pub type SectorShapeSub2 = LCTree<DefaultTreeHasher, U8, U2, U0>;
pub type SectorShapeTop2 = LCTree<DefaultTreeHasher, U8, U8, U2>;

[storojs72]

The base arity is used for all levels from the leaves up to a certain point (I originally misunderstood it as the base arity only be used for a single level).

The sub-artity and the top-arity are single levels only.

I think its correct. Depending on size of the data - consider it n , base binary tree can have up to log2[n] intermediate levels. When we group some base trees into a single composed tree, we add one additional level of composition. If we further group some composed trees into a single composed-composed tree, we add second additional level of composition. TBH, I don't know where did this requirement come from and why we have 2 levels of compositions (not 3 or 10) used for representing Filecoin sectors... The only additional detail I can mention is that Poseidon hashing works optimally exactly with 8-0-0, 8-8-0, 8-8-2 (oct) trees

[storojs72]

pub type SectorShapeSub2 = LCTree<DefaultTreeHasher, U8, U2, U0>; - it means we work with LevelCacheStore-backed compound tree (as top-tree arity is U0 - so there is no highest level of composition) that consists from 2 (sub-tree arity is U2) oct trees (base arity is U8),

pub type SectorShapeTop2 = LCTree<DefaultTreeHasher, U8, U8, U2>; - it means we work with LevelCacheStore-backed compound-compound tree (as top-tree arity is not U0), consisted from 2 (top tree arity is U2) compound trees, where each compound tree consists from 8 (sub tree arity is U8) oct trees (base arity is U8).

[vmx]

I wonder what practical differences those two trees have (perhaps the caching?), as they are both oct-trees where the top level has arity2. I guess I need to check the source.

[storojs72]

I think that sector size is the key. For 32gib sector we use 8-8-0 trees (compound), while for 64gib - 8-8-2 (compound-compound). Using "compound-compound" trees opens possibility to parallelise computation of inclusion proofs, I guess

[storojs72]

LGTM