Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services/horizon: Add a brin index on history_trades_60000(timestamp) #3947

Merged
merged 3 commits into from
Sep 22, 2021

Conversation

paulbellamy
Copy link
Contributor

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with name of package that is most changed in the PR, ex.
    services/friendbot, or all or doc if the changes are broad or impact many
    packages.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated any docs (developer docs, .md
    files, etc... affected by this change). Take a look in the docs folder for a given service,
    like this one.

Release planning

  • I've updated the relevant CHANGELOG (here for Horizon) if
    needed with deprecations, added features, breaking changes, and DB schema changes.
  • I've decided if this PR requires a new major/minor version according to
    semver, or if it's mainly a patch change. The PR is targeted at the next
    release branch if it's not a patch change.

What

Add a BRIN index onto history_trades_60000, on the timestamp column.

Why

Fixes #3939

I ran the queries that a RebuildTradeAggregationBuckets does on staging with timings, and found:

finding ledger start timestamp: 0.5ms
finding ledger end timestamp: 0.5ms
deleting 40 old records: 360ms
inserting 40 new records: 3ms

After some explain analyze on the delete query, I found that the time was being taken in finding which aggregates to delete. The primary key btree index it was using to filter timestamps is not very effective. BRINs are small, and great for time-series data, so I added one. Now delete takes ~1ms.

Known limitations

What do we want to do with the migration here? AMM also has a migration number 49. Should we rename this one? or the other?

@paulbellamy paulbellamy requested a review from a team September 21, 2021 18:11
@tamirms
Copy link
Contributor

tamirms commented Sep 21, 2021

What do we want to do with the migration here? AMM also has a migration number 49. Should we rename this one? or the other?

I think we should rename the AMM migration because this branch is closer to getting merged into master than the amm branch

@paulbellamy paulbellamy merged commit d32d747 into stellar:master Sep 22, 2021
@paulbellamy paulbellamy deleted the paulb/trade_agg_brin_index branch September 22, 2021 10:30
bartekn pushed a commit that referenced this pull request Sep 22, 2021
…#3947)

* Add a brin index on history_trades_60000(timestamp)

* Update changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Debug ingestion speed post RebuildTradeAggregationBuckets
3 participants