Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add skip db write flags for tables in parquet migration tranche 2 and 3 #429

Merged
merged 4 commits into from
Jun 28, 2024

Conversation

yuunlimm
Copy link
Contributor

@yuunlimm yuunlimm commented Jun 26, 2024

Summary

We have categorized our tables into four tranches to manage database better writes and prioritize our parquet processing efforts. The goal is to enable skipping database writes for tables in Tranche 2 and Tranche 3 by adding appropriate flags in each processor responsible for these models.

Here is the list of tables we have: https://docs.google.com/spreadsheets/d/1gSuE27GsGKLCBLOxhlzonvHh_bzF5i1Uzx27Mf61uh0/edit?gid=0#gid=0

Tranche Categories:

Tranche 1: tables that are huge and high priority
Tranche 2: tables that are huge and low usage
Tranche 3: v1 tables to replace with v2 tables.
Tranche 4: tables to revisit.

This enables skipping db writes for tables included in Tranche 2 and 3 by adding a flag in each processor responsible for each model.

We can add table names in the deprecated_tables field in the config as needed

Revision 2

  • excluded the flags for transaction_metadata_processor as clients can stop running this processor completely, and these were for analytics purposes only.

test plan

Screenshot 2024-06-26 at 1 09 21 PM
Screenshot 2024-06-26 at 1 09 17 PM

tested when CURRENT_COIN_BALANCES was added

@yuunlimm yuunlimm requested a review from a team June 26, 2024 20:15
rust/processor/src/processors/coin_processor.rs Outdated Show resolved Hide resolved
rust/processor/src/processors/coin_processor.rs Outdated Show resolved Hide resolved
.deprecated_tables
.contains(TableFlags::CURRENT_FUNGIBLE_ASSET_BALANCES)
{
current_fungible_asset_balances.clear();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note: this table is on a different deprecation timeline than the other ones for the coin -> FA migration. It's fine to add the flag, but we don't want to turn it off just yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to re-eval this table current_fungible_asset_balances with @ying-w

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need this in parquet, we can delete this table. according to Ying, we can replace with fungible_asset_balances

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will start a separate migration plan for this table with the different timeline.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait fungible_asset_balances is going to Parquet, but we need to keep current_fungible_asset_balances for now for the coin -> FA migration, which I'll deal with .

@yuunlimm yuunlimm requested a review from rtso June 27, 2024 18:32
@yuunlimm
Copy link
Contributor Author

Migration view pr for current_collection_datas and migrating current_token_pending_claims logic to v2 will be followed up in separate pr

@yuunlimm yuunlimm force-pushed the yuunlimm/tranche-2-and-3-db-skip branch from ff853e2 to 4ee5f8c Compare June 28, 2024 17:33
@yuunlimm yuunlimm enabled auto-merge (squash) June 28, 2024 18:08
@yuunlimm yuunlimm force-pushed the yuunlimm/tranche-2-and-3-db-skip branch from 4ee5f8c to d8abd65 Compare June 28, 2024 20:22
@yuunlimm yuunlimm merged commit e351895 into main Jun 28, 2024
7 checks passed
@yuunlimm yuunlimm deleted the yuunlimm/tranche-2-and-3-db-skip branch June 28, 2024 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants