Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Select eviction policy for DA compression temporal registry #2231

Open
Dentosal opened this issue Sep 20, 2024 · 0 comments
Open

Select eviction policy for DA compression temporal registry #2231

Dentosal opened this issue Sep 20, 2024 · 0 comments
Labels
tech-debt The issue is to improve the current code and make it more clear/generic/reusable/pretty/avoidable.

Comments

@Dentosal
Copy link
Member

Introduced in #1609.

DA compression uses "temporal registry", a key-value database where each compressed block header lists new writes to the registry. Since the keyspace is limited to 24bits per type, keys have to be removed according to some logic. The scheme is designed so that changing the way the evicted keys are picked can be freely changed without affecting backwards compatibility.

Currently, the code simply removes the lowest key that's not used by the current block. This suboptimal. Ideally, the values are least likely to be reused in the future will be dropped first (and values never written to are always dropped first). Something like LRU might be a good candidate for this. To get best results, we should observe real data from the system to see what the reuse rates are.

// Pick first key not in the set
// TODO: this can be optimized by keeping a counter of the last key used
// TODO: use a proper algo, maybe LRU?
let mut key = RegistryKey::ZERO;
while self.keep_keys[keyspace].contains(&key) {
key = key.next();
assert_ne!(key, RegistryKey::ZERO, "Ran out of keys");
}
self.keep_keys[keyspace].insert(key);
Ok(key)

@Dentosal Dentosal added the tech-debt The issue is to improve the current code and make it more clear/generic/reusable/pretty/avoidable. label Sep 20, 2024
@Dentosal Dentosal mentioned this issue Sep 20, 2024
6 tasks
xgreenx added a commit that referenced this issue Oct 5, 2024
Related #1605. VM PR
FuelLabs/fuel-vm#670.

This PR adds DA compression crate for Fuel blocks, performed upon block
creation. The compressed blocks are stored into the offchain database
and can be fetched using the GraphQL API.

## Note for reviewers

To keep this reasonably compact, decompression support is not included
in this PR, and will be done as a follow-up. As a result, the full data
roundtrip testing is not part of this PR. There's no proof here that
compression of full blocks is reversible.

## TODO
#### Features
- [x] Temporal registry db support
- [x] Optimize temporal registry eviction implementation
- [x] Implement TxId ↔ TxPointer lookups
- [x] Integrate with the block committer (GraphQL interface, probably)
#### Tests
- [x] compressed blocks are available from non-block-producer nodes
- [ ] e2e test for the full decompression cycle (moved to a follow-up)


## Follow-up issues
- Sync the node from L1:
#2208
- Decompression roudntrip tests
#2238
- Figure out which cache eviction algorithm/behavior is wanted:
#2231
- Figure out if we need to remove the compressed blocks from the db
after a while
- Merkle roots for fraud proofs
#2232

---------

Co-authored-by: Aaryamann Challani <[email protected]>
Co-authored-by: Green Baneling <[email protected]>
Co-authored-by: Rafał Chabowski <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tech-debt The issue is to improve the current code and make it more clear/generic/reusable/pretty/avoidable.
Projects
None yet
Development

No branches or pull requests

1 participant