Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: Validator projection catching up speed is slow #593

Merged

Conversation

ysong42
Copy link
Contributor

@ysong42 ysong42 commented Nov 8, 2021

Solution: fixed #586

The main bottleneck is in handling BlockCreated event, we have some code that is updating all active validators' uptime. Previously, we are doing it in a loop manner, each validator will do one update SQL.

In my improved version, there are mainly two changes:

  • update validators up time in batch mode, merge many update SQL into one
  • Only updated uptime related columns, not updating all columns of validator

Update multiple rows reference: https://stackoverflow.com/a/26715934

Benchmark

the improvement is at BlockCreated-updateValidator column

Table columns:
image

Before optimization:
image

After optimization:
image

Benchmark data files (local indexer running against Crypto.org mainnet):
before-optimize-validator.csv
after-optimize-validator.csv

@ysong42
Copy link
Contributor Author

ysong42 commented Nov 8, 2021

I am not using squirrel library to generate the SQL. As update with from statement is not supported by the library yet...

Masterminds/squirrel#82
Masterminds/squirrel#291

@ysong42 ysong42 force-pushed the performance/586-validator-project-slow-catch-up-speed branch from 2d352aa to 47ec132 Compare November 8, 2021 06:34
@davcrypto
Copy link
Contributor

LGTM

@davcrypto davcrypto merged commit bcae397 into master Nov 9, 2021
@ysong42 ysong42 deleted the performance/586-validator-project-slow-catch-up-speed branch November 9, 2021 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Problem: Validator projection catching up speed is slow
2 participants