Keep record of events from L1 in Block Header #1769

MitchTurner · 2024-03-20T21:18:33Z

This PR introduces a new field on the block header generated section called event_inbox_root. This is calculated after execution by adding all the hashes of the events from the L1 relayer to a merkle tree and generating the root.

Also renamed the message receipt root to message_outbox_root to keep up the theme.

Dentosal

LGTM; nice and small changeset

crates/types/src/blockchain/header.rs

crates/services/executor/src/executor.rs

crates/fuel-core/src/schema/block.rs

crates/types/src/blockchain/block.rs

Voxelot · 2024-03-21T02:51:54Z

we should do something like a sed -i -- 's/message_receipt_root/message_outbox_root/g' * to cleanup all the old references to the previous name of the outbox field.

crates/client/src/client/types/block.rs

xgreenx · 2024-03-21T23:00:20Z

crates/services/executor/src/executor.rs

        for da_height in next_unprocessed_da_height..=header.da_height.0 {
            let da_height = da_height.into();
            let events = self
                .relayer
                .get_events(&da_height)
                .map_err(|err| ExecutorError::RelayerError(err.into()))?;
            for event in events {
+                root_calculator.push(event.hash().as_ref());


Hmm, I'm curious: does Ethereal RPC guarantee that the order of events is the same as they were emitted? @MitchTurner , Could you double-check that?=)

Looking at ethers, there is a field for indexing fields within each block. We could sort all the logs by this in the relayer to play it safe.

We could also consider lexicographically sorting all the event hashes before merklizing

The order we receive the events shouldn't be dependent on the Ethereal RPC specifically, rather our implementation of Relayer as a whole.

IF the order isn't deterministic, then I agree with @Voxelot that we could just do some deterministic sorting in our Relayer adapter.

Do we want to do that as part of this PR?

Or we could just order them after we get them from the Relayer.

The best ordering Imo is to use log_index, as it ensures fair ordering of event processing. I'd prefer to avoid muddling our events in the executor with ethereum specific fields, so I'd be fine with making a separate PR to ensure ordering on the relayer side.

The alternative is to add some kind of ordering field to events, and making the executor sort them every time. However, this would also incur additional consensus overhead which I think we could avoid.

@Dentosal

## [Version 0.24.0] ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 **Full Changelog**: v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](#1786): Regenesis now includes off-chain tables. - [#1716](#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](#1770): Add the new L1 event type for forced transactions. - [#1767](#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](#1747): The DA block height is now included in the genesis state. - [#1740](#1740): Remove optional fields from genesis configs - [#1737](#1737): Remove temporary tables for calculating roots during genesis. - [#1731](#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](#1779): Modify Relayer service to order Events from L1 by block index - [#1783](#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after #1664. - [#1693](#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in #1731 * Regenesis support by @MujkicA in #1693 * Remove genesis temp tables by @MujkicA in #1737 * Remove optional fields from configs by @MujkicA in #1740 * Weekly `cargo update` by @github-actions in #1745 * Regenesis should also store da block height by @xgreenx in #1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in #1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in #1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in #1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in #1757 * Weekly `cargo update` by @github-actions in #1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in #1760 * Adjustments to the upcoming testnet configs by @xgreenx in #1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in #1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in #1768 * Keep record of events from L1 in Block Header by @MitchTurner in #1769 * Feature/new fti event by @Voxelot in #1770 * Forkless state transition with upgradable WASM executor by @xgreenx in #1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in #1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in #1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in #1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in #1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in #1786 * Release v0.24.0 by @xgreenx in #1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in #1792 **Full Changelog**: v0.23.0...v0.24.1

@Dentosal

## [Version 0.24.0] ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.0

@Dentosal

## Version v0.24.0 ### Added - [#1786](FuelLabs/fuel-core#1786): Regenesis now includes off-chain tables. - [#1716](FuelLabs/fuel-core#1716): Added support of WASM state transition along with upgradable execution that works with native(std) and WASM(non-std) executors. The `fuel-core` now requires a `wasm32-unknown-unknown` target to build. - [#1770](FuelLabs/fuel-core#1770): Add the new L1 event type for forced transactions. - [#1767](FuelLabs/fuel-core#1767): Added consensus parameters version and state transition version to the `ApplicationHeader` to describe what was used to produce this block. - [#1760](FuelLabs/fuel-core#1760): Added tests to verify that the network operates with a custom chain id and base asset id. - [#1752](FuelLabs/fuel-core#1752): Add `ProducerGasPrice` trait that the `Producer` depends on to get the gas price for the block. - [#1747](FuelLabs/fuel-core#1747): The DA block height is now included in the genesis state. - [#1740](FuelLabs/fuel-core#1740): Remove optional fields from genesis configs - [#1737](FuelLabs/fuel-core#1737): Remove temporary tables for calculating roots during genesis. - [#1731](FuelLabs/fuel-core#1731): Expose `schema.sdl` from `fuel-core-client`. ### Changed #### Breaking - [#1771](FuelLabs/fuel-core#1771): Contract 'states' and 'balances' brought back into `ContractConfig`. Parquet now writes a file per table. - [1779](FuelLabs/fuel-core#1779): Modify Relayer service to order Events from L1 by block index - [#1783](FuelLabs/fuel-core#1783): The PR upgrade `fuel-vm` to `0.48.0` release. Because of some breaking changes, we also adapted our codebase to follow them: - Implementation of `Default` for configs was moved under the `test-helpers` feature. The `fuel-core` binary uses testnet configuration instead of `Default::default`(for cases when `ChainConfig` was not provided by the user). - All parameter types are enums now and require corresponding modifications across the codebase(we need to use getters and setters). The GraphQL API remains the same for simplicity, but each parameter now has one more field - `version`, that can be used to decide how to deserialize. - The `UtxoId` type now is 34 bytes instead of 33. It affects hex representation and requires adding `00`. - The `block_gas_limit` was moved to `ConsensusParameters` from `ChainConfig`. It means the block producer doesn't specify the block gas limit anymore, and we don't need to propagate this information. - The `bytecodeLength` field is removed from the `Create` transaction. - Removed `ConsensusParameters` from executor config because `ConsensusParameters::default` is not available anymore. Instead, executors fetch `ConsensusParameters` from the database. - [#1769](FuelLabs/fuel-core#1769): Include new field on header for the merkle root of imported events. Rename other message root field. - [#1768](FuelLabs/fuel-core#1768): Moved `ContractsInfo` table to the off-chain database. Removed `salt` field from the `ContractConfig`. - [#1761](FuelLabs/fuel-core#1761): Adjustments to the upcoming testnet configs: - Decreased the max size of the contract/predicate/script to be 100KB. - Decreased the max size of the transaction to be 110KB. - Decreased the max number of storage slots to be 1760(110KB / 64). - Removed fake coins from the genesis state. - Renamed folders to be "testnet" and "dev-testnet". - The name of the networks are "Upgradable Testnet" and "Upgradable Dev Testnet". - [#1694](FuelLabs/fuel-core#1694): The change moves the database transaction logic from the `fuel-core` to the `fuel-core-storage` level. The corresponding [issue](FuelLabs/fuel-core#1589) described the reason behind it. ## Technical details of implementation - The change splits the `KeyValueStore` into `KeyValueInspect` and `KeyValueMutate`, as well the `Blueprint` into `BlueprintInspect` and `BlueprintMutate`. It allows requiring less restricted constraints for any read-related operations. - One of the main ideas of the change is to allow for the actual storage only to implement `KeyValueInspect` and `Modifiable` without the `KeyValueMutate`. It simplifies work with the databases and provides a safe way of interacting with them (Modification into the database can only go through the `Modifiable::commit_changes`). This feature is used to [track the height](https://github.com/FuelLabs/fuel-core/pull/1694/files#diff-c95a3d57a39feac7c8c2f3b193a24eec39e794413adc741df36450f9a4539898) of each database during commits and even limit how commits are done, providing additional safety. This part of the change was done as a [separate commit](FuelLabs/fuel-core@7b1141a). - The `StorageTransaction` is a `StructuredStorage` that uses `InMemoryTransaction` inside to accumulate modifications. Only `InMemoryTransaction` has a real implementation of the `KeyValueMutate`(Other types only implement it in tests). - The implementation of the `Modifiable` for the `Database` contains a business logic that provides additional safety but limits the usage of the database. The `Database` now tracks its height and is responsible for its updates. In the `commit_changes` function, it analyzes the changes that were done and tries to find a new height(For example, in the case of the `OnChain` database, we are looking for a new `Block` in the `FuelBlocks` table). - As was planned in the issue, now the executor has full control over how commits to the storage are done. - All mutation methods now require `&mut self` - exclusive ownership over the object to be able to write into it. It almost negates the chance of concurrent modification of the storage, but it is still possible since the `Database` implements the `Clone` trait. To be sure that we don't corrupt the state of the database, the `commit_changes` function implements additional safety checks to be sure that we commit updates per each height only once time. - Side changes: - The `drop` function was moved from `Database` to `RocksDB` as a preparation for the state rewind since the read view should also keep the drop function until it is destroyed. - The `StatisticTable` table lives in the off-chain worker. - Removed duplication of the `Database` from the `dap::ConcreteStorage` since it is already available from the VM. - The executor return only produced `Changes` instead of the storage transaction, which simplifies the interaction between modules and port definition. - The logic related to the iteration over the storage is moved to the `fuel-core-storage` crate and is now reusable. It provides an `interator` method that duplicates the logic from `MemoryStore` on iterating over the `BTreeMap` and methods like `iter_all`, `iter_all_by_prefix`, etc. It was done in a separate revivable [commit](FuelLabs/fuel-core@5b9bd78). - The `MemoryTransactionView` is fully replaced by the `StorageTransactionInner`. - Removed `flush` method from the `Database` since it is not needed after FuelLabs/fuel-core#1664. - [#1693](FuelLabs/fuel-core#1693): The change separates the initial chain state from the chain config and stores them in separate files when generating a snapshot. The state snapshot can be generated in a new format where parquet is used for compression and indexing while postcard is used for encoding. This enables importing in a stream like fashion which reduces memory requirements. Json encoding is still supported to enable easy manual setup. However, parquet is prefered for large state files. ### Snapshot command The CLI was expanded to allow customizing the used encoding. Snapshots are now generated along with a metadata file describing the encoding used. The metadata file contains encoding details as well as the location of additional files inside the snapshot directory containing the actual data. The chain config is always generated in the JSON format. The snapshot command now has the '--output-directory' for specifying where to save the snapshot. ### Run command The run command now includes the 'db_prune' flag which when provided will prune the existing db and start genesis from the provided snapshot metadata file or the local testnet configuration. The snapshot metadata file contains paths to the chain config file and files containing chain state items (coins, messages, contracts, contract states, and balances), which are loaded via streaming. Each item group in the genesis process is handled by a separate worker, allowing for parallel loading. Workers stream file contents in batches. A database transaction is committed every time an item group is succesfully loaded. Resumability is achieved by recording the last loaded group index within the same db tx. If loading is aborted, the remaining workers are shutdown. Upon restart, workers resume from the last processed group. ### Contract States and Balances Using uniform-sized batches may result in batches containing items from multiple contracts. Optimal performance can presumably be achieved by selecting a batch size that typically encompasses an entire contract's state or balance, allowing for immediate initialization of relevant Merkle trees. ### Removed - [#1757](FuelLabs/fuel-core#1757): Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf`. ## What's Changed * Expose `schema.sdl` add some helper types and traits by @Dentosal in FuelLabs/fuel-core#1731 * Regenesis support by @MujkicA in FuelLabs/fuel-core#1693 * Remove genesis temp tables by @MujkicA in FuelLabs/fuel-core#1737 * Remove optional fields from configs by @MujkicA in FuelLabs/fuel-core#1740 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1745 * Regenesis should also store da block height by @xgreenx in FuelLabs/fuel-core#1747 * Duplicating blacklisting feature for TxPool from `0.22.4` by @xgreenx in FuelLabs/fuel-core#1748 * Moved `StorageTransaction` to the `fuel-core-storage` crate by @xgreenx in FuelLabs/fuel-core#1694 * Prepare the codebase to use base gas price during block production #1642 by @MitchTurner in FuelLabs/fuel-core#1752 * Removed `protobuf` from everywhere since `libp2p` uses `quick-protobuf` by @xgreenx in FuelLabs/fuel-core#1757 * Weekly `cargo update` by @github-actions in FuelLabs/fuel-core#1758 * Added tests to verify that the network operates with a custom chain id and base asset id by @xgreenx in FuelLabs/fuel-core#1760 * Adjustments to the upcoming testnet configs by @xgreenx in FuelLabs/fuel-core#1761 * Added consensus parameters version and state transition version to the `ApplicationHeader` by @xgreenx in FuelLabs/fuel-core#1767 * Moved `ContractsInfo` table to the off-chain database by @xgreenx in FuelLabs/fuel-core#1768 * Keep record of events from L1 in Block Header by @MitchTurner in FuelLabs/fuel-core#1769 * Feature/new fti event by @Voxelot in FuelLabs/fuel-core#1770 * Forkless state transition with upgradable WASM executor by @xgreenx in FuelLabs/fuel-core#1716 * Removed the usage of the `lazy_static` from teh codebase by @xgreenx in FuelLabs/fuel-core#1781 * Patch to use `fuel-vm 0.48.0` by @xgreenx in FuelLabs/fuel-core#1783 * Modify Relayer service to order Events from L1 by block index by @MitchTurner in FuelLabs/fuel-core#1779 * refactor: Prepare (re)genesis for off chain tables by @segfault-magnet in FuelLabs/fuel-core#1771 * feat: Add some off chain tables to regenesis by @segfault-magnet in FuelLabs/fuel-core#1786 * Release v0.24.0 by @xgreenx in FuelLabs/fuel-core#1791 * Moved chain specification into `fuel-core-bin` crate by @xgreenx in FuelLabs/fuel-core#1792 **Full Changelog**: FuelLabs/fuel-core@v0.23.0...v0.24.1

MitchTurner self-assigned this Mar 20, 2024

Add event root to block header

36fc1ec

MitchTurner force-pushed the include-event-record-in-header branch from bb02a07 to 36fc1ec Compare March 20, 2024 21:24

MitchTurner added 3 commits March 20, 2024 15:19

Add test for event inbox root, might not be passing

a94e8bb

Fix test and other compilation stuff

dd6b0ca

Update CHANGELOG

9402921

MitchTurner requested a review from a team March 20, 2024 23:48

MitchTurner marked this pull request as ready for review March 20, 2024 23:48

Move stuff in the test around

8eb2714

Dentosal previously approved these changes Mar 21, 2024

View reviewed changes

Voxelot requested changes Mar 21, 2024

View reviewed changes

crates/types/src/blockchain/header.rs Outdated Show resolved Hide resolved

Use correct Event type and record it inside process_da method

f6e6e22

MitchTurner dismissed Dentosal’s stale review via f6e6e22 March 21, 2024 02:21

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/services/executor/src/executor.rs Outdated Show resolved Hide resolved

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/fuel-core/src/schema/block.rs Show resolved Hide resolved

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/fuel-core/src/schema/block.rs Outdated Show resolved Hide resolved

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/types/src/blockchain/block.rs Outdated Show resolved Hide resolved

MitchTurner added 3 commits March 21, 2024 13:29

Fix names, add getter, remove comments

95663d4

Update schema

c069557

Fix snapshots

99cddf3

MitchTurner requested a review from a team March 21, 2024 21:53

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/client/src/client/types/block.rs Show resolved Hide resolved

Include field in client header type

646cfbf

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/client/src/client/types/block.rs Show resolved Hide resolved

Voxelot reviewed Mar 21, 2024

View reviewed changes

crates/client/src/client/types/block.rs Show resolved Hide resolved

Include field in From constructor

54de938

xgreenx approved these changes Mar 21, 2024

View reviewed changes

Voxelot approved these changes Mar 21, 2024

View reviewed changes

MitchTurner mentioned this pull request Mar 22, 2024

Ensure that L1 Events received from the relayer are in a deterministic order #1772

Closed

MitchTurner merged commit c855fda into master Mar 22, 2024
31 checks passed

MitchTurner deleted the include-event-record-in-header branch March 22, 2024 02:23

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.0 #1791

Merged

xgreenx mentioned this pull request Mar 30, 2024

Release v0.24.1 #1793

Merged

MitchTurner mentioned this pull request Apr 10, 2024

Add specification for how L1 events are ordered and the merkle root is calculated FuelLabs/fuel-specs#565

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keep record of events from L1 in Block Header #1769

Keep record of events from L1 in Block Header #1769

MitchTurner commented Mar 20, 2024 •

edited by Voxelot

Loading

Dentosal left a comment

Voxelot commented Mar 21, 2024 •

edited

Loading

xgreenx Mar 21, 2024

Voxelot Mar 21, 2024

Voxelot Mar 21, 2024

MitchTurner Mar 21, 2024

MitchTurner Mar 22, 2024

Voxelot Mar 22, 2024

Keep record of events from L1 in Block Header #1769

Keep record of events from L1 in Block Header #1769

Conversation

MitchTurner commented Mar 20, 2024 • edited by Voxelot Loading

Dentosal left a comment

Choose a reason for hiding this comment

Voxelot commented Mar 21, 2024 • edited Loading

xgreenx Mar 21, 2024

Choose a reason for hiding this comment

Voxelot Mar 21, 2024

Choose a reason for hiding this comment

Voxelot Mar 21, 2024

Choose a reason for hiding this comment

MitchTurner Mar 21, 2024

Choose a reason for hiding this comment

MitchTurner Mar 22, 2024

Choose a reason for hiding this comment

Voxelot Mar 22, 2024

Choose a reason for hiding this comment

MitchTurner commented Mar 20, 2024 •

edited by Voxelot

Loading

Voxelot commented Mar 21, 2024 •

edited

Loading