-
Notifications
You must be signed in to change notification settings - Fork 129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
trie: implement lazy loading of trie nodes from disk #2838
Comments
I found that Ethereum is also using a lazy loading strategy with the |
In substrate, they are using this trie crate for all trie operations, and they create a TrieDB structure backed by a DB with a cache, and they are implementing the lazy loading there |
This change will help us to improve our memory usage and our node startup time when we are using an snapshot, also improving our debugging time when we are trying to sync to the tip of the chain and we need to use snapshots for that. |
This issue will be Done when #3805 is completed |
Design Design considered complete |
Our memory usage is high since we keep all trie nodes in memory.
Instead we should (like Substrate) store all
node hash -> node encoding
on disk, and lazy load them as we need to access the trie (both read and write). Only the root hash should be kept in memory, and every node should be retrieved from disk as we go deeper in the trie (to insert or retrieve).We should also implement pruning for nodes inserted/modified in uncle blocks (and make sure the finalized block doesn't contain them).
We can also implement a cache to keep the most frequently used nodes in memory and prevent decoding them again
Related with #2835 and #2836
The text was updated successfully, but these errors were encountered: