-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add skip db write flags for tables in parquet migration tranche 2 and 3 #429
Conversation
.deprecated_tables | ||
.contains(TableFlags::CURRENT_FUNGIBLE_ASSET_BALANCES) | ||
{ | ||
current_fungible_asset_balances.clear(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a note: this table is on a different deprecation timeline than the other ones for the coin -> FA migration. It's fine to add the flag, but we don't want to turn it off just yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we need to re-eval this table current_fungible_asset_balances
with @ying-w
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we don't need this in parquet, we can delete this table. according to Ying, we can replace with fungible_asset_balances
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will start a separate migration plan for this table with the different timeline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait fungible_asset_balances
is going to Parquet, but we need to keep current_fungible_asset_balances
for now for the coin -> FA migration, which I'll deal with .
Migration view pr for |
ff853e2
to
4ee5f8c
Compare
4ee5f8c
to
d8abd65
Compare
Summary
We have categorized our tables into four tranches to manage database better writes and prioritize our parquet processing efforts. The goal is to enable skipping database writes for tables in Tranche 2 and Tranche 3 by adding appropriate flags in each processor responsible for these models.
Here is the list of tables we have: https://docs.google.com/spreadsheets/d/1gSuE27GsGKLCBLOxhlzonvHh_bzF5i1Uzx27Mf61uh0/edit?gid=0#gid=0
Tranche Categories:
Tranche 1: tables that are huge and high priority
Tranche 2: tables that are huge and low usage
Tranche 3: v1 tables to replace with v2 tables.
Tranche 4: tables to revisit.
This enables skipping db writes for tables included in Tranche 2 and 3 by adding a flag in each processor responsible for each model.
We can add table names in the
deprecated_tables
field in the config as neededRevision 2
test plan
tested when CURRENT_COIN_BALANCES was added