backport IAVL to v3.1.0 and v4.2.0

[p0mvn]

Background

We would like to backport the fast storage IAVL changes to v3.x and v4.x

Being able to do this is important safety-wise and for future features. We shouldn’t be only relying on a centralized server to download archive nodes all the time.

Basically, these IAVL changes backported would allow us to sync from gen way faster from the first block. It is also important if we want to transition to a different database since we would have to replay the entire state.

Notes

• we skip v5.x because it is state-compatible with v6.x.
• we skip v2.x because there was an SDK issue
• we skip v1.x because it is not state-compatible with itself

Acceptance Criteria

• backport fast iavl to v3.x  #3518
• backport fast iavl to v4.x (create a separate issue) #3519

[faddat]

Hi, I'm on this, currently working on updating the cosmos sdk stuff needed for it.

PS: in the future it may make the most sense to refer to the state breaking versions as v3.x and v4.x --it is those branches that I target PR's to.

When this is complete I will release an archive image for rocksdb.

How high priority do we consider this?

It seems like it requires a backport of state listening to cosmos-sdk v0.42.10 and this gets pretty complicated. Work is started here:

https://github.com/osmosis-labs/cosmos-sdk/tree/v0.42.10-osmofast

A potentially more palatable option may be to backport iavl changes to v0.16.0

I just had a look at that and it doesn't seem real great either. wdyt?

Additionally, it is notable that a full sync is much, much faster with rocksdb.

[p0mvn]

Hi @faddat

Thanks so much for taking this.

in the future it may make the most sense to refer to the state breaking versions as v3.x and v4.x

Sounds good, changed the issue description to reflect this.

How high priority do we consider this?

This is not urgent but useful to have for full sync e.g. for rocks db

It seems like it requires a backport of state listening to cosmos-sdk v0.42.10 and this gets pretty complicated.

Could you help me understand why that is needed?

Also, I noticed that the pruning manager change was ported here. I think we can get by without it, for now, to limit the scope and make the backport easier. The pruning manager change is mainly needed to make snapshots work with rigorous pruning.

For now, we can require one of the following options to do a full sync:

• snapshots are to be disabled
• large pruning-keep-recent is set (>5K)

Let me know what you think

[faddat]

I am midway through making a rocks archive node automatcially. Right. now the main holdup is that I haven't automated it yet, pretty sure that with automation it is less than 24h total sync time.

[ValarDragon]

Lets close, and target this via IAVL v1