Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use COPY to speed up all ChangeProcessors #5098

Closed
6 tasks done
tamirms opened this issue Nov 1, 2023 · 1 comment
Closed
6 tasks done

Use COPY to speed up all ChangeProcessors #5098

tamirms opened this issue Nov 1, 2023 · 1 comment
Assignees
Labels
horizon performance issues aimed at improving performance

Comments

@tamirms
Copy link
Contributor

tamirms commented Nov 1, 2023

In #5086 we improved performance of the ClaimableBalancesChangeProcessor by using COPY to bulk insert claimable balance rows. We should use COPY to improve the performance of all the remaining change processors.

Most of the change processors combine new ledger entries and updates of existing ledger entries into a single list which is then upserted into the DB. We should change this workflow so that we handle new ledger entries separately from updating existing ledger entries because we can bulk insert new ledger entries efficiently with COPY.

The claimable balance change processor has already been taken care of in #5086 and the asset stats processor is still in flux because of #5033 . So we should implement the COPY optimization on the following remaining change processors:

I anticipate that once COPY is used in all the change processors, ingesting from the history archives will be significantly faster, which means that state rebuilds will much faster than before.

@tamirms tamirms added horizon performance issues aimed at improving performance labels Nov 1, 2023
@tamirms tamirms changed the title Use COPY to insert into speed up all ChangeProcessors Use COPY to speed up all ChangeProcessors Nov 1, 2023
@mollykarcher mollykarcher moved this from Backlog to Next Sprint Proposal in Platform Scrum Nov 1, 2023
@urvisavla urvisavla self-assigned this Nov 2, 2023
@urvisavla urvisavla moved this from Next Sprint Proposal to Current Sprint in Platform Scrum Nov 3, 2023
@urvisavla urvisavla moved this from Current Sprint to In Progress in Platform Scrum Nov 8, 2023
@urvisavla urvisavla moved this from In Progress to Done in Platform Scrum Nov 21, 2023
@hethm999
Copy link

hethm999 commented Dec 6, 2023

ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
horizon performance issues aimed at improving performance
Projects
Status: Done
Development

No branches or pull requests

4 participants