Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store API-related tables in a separate folder #1568

Closed
xgreenx opened this issue Dec 21, 2023 · 0 comments
Closed

Store API-related tables in a separate folder #1568

xgreenx opened this issue Dec 21, 2023 · 0 comments
Assignees
@xgreenx

Comments

@xgreenx
Copy link
Collaborator

xgreenx commented Dec 21, 2023

No description provided.

@xgreenx xgreenx changed the title Extract API related tables into a separate service Store API-related in a separate folder Dec 21, 2023
@xgreenx xgreenx changed the title Store API-related in a separate folder Store API-related tables in a separate folder Jan 8, 2024
@xgreenx xgreenx self-assigned this Jan 11, 2024
This was referenced Jan 24, 2024
Merged
Merged
xgreenx added a commit that referenced this issue Jan 30, 2024
@xgreenx
Making relayer independent from the executor (#1625)
3c55250 
It is preparation for the
#1568.

The changes make relayer storage independent from the executor storage.
Before, the relayer and executor shared write ownership to the
`Messages` table. The relayer was inserting new messages, and the
executor was removing them.

With this change, only the executor modifies the `Messages`
table(inserts and removes messages). The relayer has its own new
`History` table, that stores all events from the DA layer per each
height.

This change also makes the insertion of upcoming events from DA as part
of the state transition, allowing in the future handle [force
transaction
inclusion](#1626).

The change:
- Adds blanket implementation for the `VmStorageRequirements` since the
executor requires access to the `FuelBlocks` table, and we can inherit
this implementation.
- Adds new tests for the executor that verifies fetching data from the
relayer.
- Introduces a new general `Event` type for messages and forced
transactions (in the future).
xgreenx added a commit that referenced this issue Feb 2, 2024
@xgreenx
Use a separate database for each data domain (#1629)
114d450 
Closes #1568

The change splits the `Database` into 3 databases: 
- `Database<OnChain>` - Stores only data required for normal work of the
blockchain.
- `Database<OffChain>` - Stores only data used by the off-chain services
like GraphQL.
- `Database<Relayer>` - Stores relayer-related data like events(messages
or transactions) from L1.

The `Database<Description>` type has a generic `Description` that
implements the `DatabaseDescription` trait:

```rust
/// The description of the database that makes it unique.
pub trait DatabaseDescription: 'static + Clone + Debug + Send + Sync {
    /// The type of the column used by the database.
    type Column: StorageColumn + strum::EnumCount + enum_iterator::Sequence;
    /// The type of the height of the database used to track commits.
    type Height: Copy;

    /// Returns the expected version of the database.
    fn version() -> u32;

    /// Returns the name of the database.
    fn name() -> &'static str;

    /// Returns the column used to store the metadata.
    fn metadata_column() -> Self::Column;

    /// Returns the prefix for the column.
    fn prefix(column: &Self::Column) -> Option<usize>;
}
```

Each database has its folder, defined by the
`DatabaseDescription::name`, where actual data is stored.

<img width="353" alt="image"
src="https://github.com/FuelLabs/fuel-core/assets/18346821/8b642384-0dd4-4668-a415-0748be3e88f0">


Each database has its own `Column` type that describes all columns,
avoiding overlaps with other tables. The change updates a little bit
`StrucutredStorage` implementation and `TableWithBlueprint` to be more
flexible and use the `Column` defined by the table, instead of hardcoded
`fuel_core_storage::column::Column`.


Other small changes:
- Unified the logic of storing the database's metadata. It will be
useful for #1589 to
implement a unified `commit_chagnes` function.
- The `latest_height` function now uses the height from the metadata
table.
- Removed relayers-related tables and columns from the
`fuel-core-storage` crate.
- Removed part of GraphQL tables and columns from the
`fuel-core-storage`. The last part will be removed during
#1583.
- Moved `tx_count` metrics from `BlockImporter` to GraphQL off-chain
worker. Any statistic that requires a persistent state in the database
may be done outside of the blockchain.
- Remove `chain_name` from the database. The `ConsensusParameters`
already contains this information.
- Removed the `checkpoint` function from the `RocksDB` since it is not
used. Later it will be added again back but with another implementation
during #1589.
- Removed `Column::ForeignColumn`, since each database has its own
`Column` type. Removed the macro rules added to handle `ForeignColumn`.
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
@crypto523
Making relayer independent from the executor (#1625)
aabfd6d 
It is preparation for the
FuelLabs/fuel-core#1568.

The changes make relayer storage independent from the executor storage.
Before, the relayer and executor shared write ownership to the
`Messages` table. The relayer was inserting new messages, and the
executor was removing them.

With this change, only the executor modifies the `Messages`
table(inserts and removes messages). The relayer has its own new
`History` table, that stores all events from the DA layer per each
height.

This change also makes the insertion of upcoming events from DA as part
of the state transition, allowing in the future handle [force
transaction
inclusion](FuelLabs/fuel-core#1626).

The change:
- Adds blanket implementation for the `VmStorageRequirements` since the
executor requires access to the `FuelBlocks` table, and we can inherit
this implementation.
- Adds new tests for the executor that verifies fetching data from the
relayer.
- Introduces a new general `Event` type for messages and forced
transactions (in the future).
crypto523 added a commit to crypto523/fuel-core that referenced this issue Oct 7, 2024
@crypto523
Use a separate database for each data domain (#1629)
4f71ff9 
Closes FuelLabs/fuel-core#1568

The change splits the `Database` into 3 databases: 
- `Database<OnChain>` - Stores only data required for normal work of the
blockchain.
- `Database<OffChain>` - Stores only data used by the off-chain services
like GraphQL.
- `Database<Relayer>` - Stores relayer-related data like events(messages
or transactions) from L1.

The `Database<Description>` type has a generic `Description` that
implements the `DatabaseDescription` trait:

```rust
/// The description of the database that makes it unique.
pub trait DatabaseDescription: 'static + Clone + Debug + Send + Sync {
    /// The type of the column used by the database.
    type Column: StorageColumn + strum::EnumCount + enum_iterator::Sequence;
    /// The type of the height of the database used to track commits.
    type Height: Copy;

    /// Returns the expected version of the database.
    fn version() -> u32;

    /// Returns the name of the database.
    fn name() -> &'static str;

    /// Returns the column used to store the metadata.
    fn metadata_column() -> Self::Column;

    /// Returns the prefix for the column.
    fn prefix(column: &Self::Column) -> Option<usize>;
}
```

Each database has its folder, defined by the
`DatabaseDescription::name`, where actual data is stored.

<img width="353" alt="image"
src="https://github.com/FuelLabs/fuel-core/assets/18346821/8b642384-0dd4-4668-a415-0748be3e88f0">


Each database has its own `Column` type that describes all columns,
avoiding overlaps with other tables. The change updates a little bit
`StrucutredStorage` implementation and `TableWithBlueprint` to be more
flexible and use the `Column` defined by the table, instead of hardcoded
`fuel_core_storage::column::Column`.


Other small changes:
- Unified the logic of storing the database's metadata. It will be
useful for FuelLabs/fuel-core#1589 to
implement a unified `commit_chagnes` function.
- The `latest_height` function now uses the height from the metadata
table.
- Removed relayers-related tables and columns from the
`fuel-core-storage` crate.
- Removed part of GraphQL tables and columns from the
`fuel-core-storage`. The last part will be removed during
FuelLabs/fuel-core#1583.
- Moved `tx_count` metrics from `BlockImporter` to GraphQL off-chain
worker. Any statistic that requires a persistent state in the database
may be done outside of the blockchain.
- Remove `chain_name` from the database. The `ConsensusParameters`
already contains this information.
- Removed the `checkpoint` function from the `RocksDB` since it is not
used. Later it will be added again back but with another implementation
during FuelLabs/fuel-core#1589.
- Removed `Column::ForeignColumn`, since each database has its own
`Column` type. Removed the macro rules added to handle `ForeignColumn`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xgreenx xgreenx

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@xgreenx