You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DA compression uses "temporal registry", a key-value database where each compressed block header lists new writes to the registry. Since the keyspace is limited to 24bits per type, keys have to be removed according to some logic. The scheme is designed so that changing the way the evicted keys are picked can be freely changed without affecting backwards compatibility.
Currently, the code simply removes the lowest key that's not used by the current block. This suboptimal. Ideally, the values are least likely to be reused in the future will be dropped first (and values never written to are always dropped first). Something like LRU might be a good candidate for this. To get best results, we should observe real data from the system to see what the reuse rates are.
Related #1605. VM PR
FuelLabs/fuel-vm#670.
This PR adds DA compression crate for Fuel blocks, performed upon block
creation. The compressed blocks are stored into the offchain database
and can be fetched using the GraphQL API.
## Note for reviewers
To keep this reasonably compact, decompression support is not included
in this PR, and will be done as a follow-up. As a result, the full data
roundtrip testing is not part of this PR. There's no proof here that
compression of full blocks is reversible.
## TODO
#### Features
- [x] Temporal registry db support
- [x] Optimize temporal registry eviction implementation
- [x] Implement TxId ↔ TxPointer lookups
- [x] Integrate with the block committer (GraphQL interface, probably)
#### Tests
- [x] compressed blocks are available from non-block-producer nodes
- [ ] e2e test for the full decompression cycle (moved to a follow-up)
## Follow-up issues
- Sync the node from L1:
#2208
- Decompression roudntrip tests
#2238
- Figure out which cache eviction algorithm/behavior is wanted:
#2231
- Figure out if we need to remove the compressed blocks from the db
after a while
- Merkle roots for fraud proofs
#2232
---------
Co-authored-by: Aaryamann Challani <[email protected]>
Co-authored-by: Green Baneling <[email protected]>
Co-authored-by: Rafał Chabowski <[email protected]>
Introduced in #1609.
DA compression uses "temporal registry", a key-value database where each compressed block header lists new writes to the registry. Since the keyspace is limited to 24bits per type, keys have to be removed according to some logic. The scheme is designed so that changing the way the evicted keys are picked can be freely changed without affecting backwards compatibility.
Currently, the code simply removes the lowest key that's not used by the current block. This suboptimal. Ideally, the values are least likely to be reused in the future will be dropped first (and values never written to are always dropped first). Something like LRU might be a good candidate for this. To get best results, we should observe real data from the system to see what the reuse rates are.
fuel-core/crates/compression/src/eviction_policy.rs
Lines 21 to 31 in 8529440
The text was updated successfully, but these errors were encountered: