Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

export: don't export branch node's key in snapshot #688

Closed
yihuang opened this issue Feb 22, 2023 · 1 comment
Closed

export: don't export branch node's key in snapshot #688

yihuang opened this issue Feb 22, 2023 · 1 comment

Comments

@yihuang
Copy link
Collaborator

yihuang commented Feb 22, 2023

in IAVL, a branch node's key is always the smallest leaf key in it's right branch, the key can be infered during the import process, so no need to include it in the snapshot at all.

Migration process

  • Step 1, change the import logic to derive the branch node's key on the fly, ignore the one included in the snapshot, this is not a consensus breaking change.
  • Step 2, change the export logic to skip branch node's key, this should be considered as a breaking change and need a coordinated upgrade, so the snapshots created after the upgrade are updated consistently across the network.

Do we need to update the snapshot format version for this?

Consequences

Positives

  • Reduce snapshot size, say L is the number of leave nodes, previously we export 2 * L - 1 keys and L values, now we export L keys and L values, if key and value are the same length on average, it reduces 30% size of snapshot.
  • In the future, can apply delta encoding to the leaf keys to further reduce snapshot size, since the leaf keys are exported in order, apply delta encoding to it should produce very good compression rate.
yihuang added a commit to yihuang/iavl that referenced this issue Feb 22, 2023
Closes: cosmos#688

since it can be derived on the fly
@yihuang yihuang changed the title don't export branch node's key don't export branch node's key in snapshot Feb 22, 2023
@yihuang yihuang changed the title don't export branch node's key in snapshot export: don't export branch node's key in snapshot Feb 23, 2023
@yihuang
Copy link
Collaborator Author

yihuang commented Apr 12, 2023

fixed by #703

@yihuang yihuang closed this as completed Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant