DA compression for fuel-tx types #670
Conversation
There was a problem hiding this comment.
Thank you nice PR!=) It is cool that Compact can be derived almost for all types.
I feel like we put too many details about the implementation of the algorithm and how keys are created, and it seems we can simplify that part a lot to be agnostic. But the main with compaction looks good; I will review it one more time, more precisely when PR is in final shape.
BTW, compilation fails for me:
Co-authored-by: Brandon Vrooman <[email protected]>
Co-authored-by: Brandon Vrooman <[email protected]>
| pub use fuel_derive::{ | ||
| Compress, | ||
| Decompress, | ||
| }; |
There was a problem hiding this comment.
Also, Compress implements both Compressible and CompressibleBy, and the define macros having the same names as the trait obfuscates this fact even more.
Yeah, it is true. But Compressible is a super trait of CompressibleBy, so it is not so bad=D But I agree that it is hidden property of the macro that can be discovered only from the comment.
But in the case of DecompressibleBy everything is clear=) The macro implements the same trait.
But it is up to you, I will not push it futher=)
…el-vm into dento/da-compression
Co-authored-by: Rafał Chabowski <[email protected]>
Co-authored-by: Rafał Chabowski <[email protected]>
Related #1605. VM PR FuelLabs/fuel-vm#670. This PR adds DA compression crate for Fuel blocks, performed upon block creation. The compressed blocks are stored into the offchain database and can be fetched using the GraphQL API. ## Note for reviewers To keep this reasonably compact, decompression support is not included in this PR, and will be done as a follow-up. As a result, the full data roundtrip testing is not part of this PR. There's no proof here that compression of full blocks is reversible. ## TODO #### Features - [x] Temporal registry db support - [x] Optimize temporal registry eviction implementation - [x] Implement TxId ↔ TxPointer lookups - [x] Integrate with the block committer (GraphQL interface, probably) #### Tests - [x] compressed blocks are available from non-block-producer nodes - [ ] e2e test for the full decompression cycle (moved to a follow-up) ## Follow-up issues - Sync the node from L1: #2208 - Decompression roudntrip tests #2238 - Figure out which cache eviction algorithm/behavior is wanted: #2231 - Figure out if we need to remove the compressed blocks from the db after a while - Merkle roots for fraud proofs #2232 --------- Co-authored-by: Aaryamann Challani <[email protected]> Co-authored-by: Green Baneling <[email protected]> Co-authored-by: Rafał Chabowski <[email protected]>
Part of FuelLabs/fuel-core#1605. See the corresponding fuel-core PR as well: FuelLabs/fuel-core#1609
This PR adds
Compactderive macro andfuel-compressionmachinery, that are used to convertTransactionand contained types intoCompressedTransactionandCompressed*. The compact types are equivalent to the original types, except that malleable fields are removed, and often-repeated types likeAssetIdandAddressare converted to three-byte keys to a separate database, called temporal storage.UtxoIds (which are used only once) are converted to useTxPointerinstead.