Bids 2379/new bigtable cl schema #2490
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
manuelsc
reviewed
Sep 11, 2023
manuelsc
reviewed
Sep 11, 2023
D13ce
reviewed
Sep 11, 2023
D13ce
reviewed
Sep 11, 2023
D13ce
reviewed
Sep 11, 2023
D13ce
reviewed
Sep 11, 2023
D13ce
reviewed
Sep 11, 2023
…ce data from the node instead of bigtable
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements a new bigtable cl schema that eliminates performance issues with the current schema.
Currenlty we store that data in one large table with the epoch and block number as row key and all validator specific data in columns.
This causes the column size to grow together with the validator set size. As for certain networks the validator set size is quite large we will ultimately run into performance issues.
New schema:
-) Create a new bigtable table that contains the validator history (proposals, balances, attestations, sync_committees, reward_details)
-) Row key is in the format <chain_id><reversed_validator_index><history_type><reversed_epoch/slot> where we use epoch if its just one entry per epoch and slot if there are multiple entries per epoch. reversed_validator_index is the reverse notation of the validator index so 1234 becomes 4321. reversed_epoch remains as max_uint64 - epoch
The new schema will be slower on writes than the current one but significant faster on reads if they only concern a few validators (validator view & dashboard view). It will be slower for reads over the whole validator set which are needed for the statistics process. If the slowness in regards to writes & statistics is tolerable needs to be tried using real data. Nevertheless bigtable scaling should be able to accommodate with the new schema as the reversed validator index will allow a very efficient table partitioning.
Prior to merging & deploying the PR a full data migration from the old to the new schema is required.
🤖 Generated by Copilot at f0364c5
This pull request optimizes various database queries and functions that fetch validator data from Bigtable, by passing a list of validator indices instead of querying all validators. It also adds a new function to reverse strings and fixes a bug in the burn page data query. The affected files are
db/statistics.go,handlers/validator.go,services/charts_updater.go,db/db.go,services/notifications.go,services/services.go, andutils/utils.go.