services/horizon/db2/history: Improve /trades DB queries performance #2869
Conversation
| return q | ||
| } | ||
|
|
||
| q.rawSQL = fmt.Sprintf("(%s) UNION (%s) ", firstSQL, secondSQL) |
There was a problem hiding this comment.
take a look at TransactionByHash() to see an alternative way to construct the UNION query using squirrel
There was a problem hiding this comment.
This was my first approach but it doesn't work when you want to add ORDER BY and/or LIMIT to UNION (requires brackets).
| q.sql = q.appendOrdering(q.sql, op, idx, page.Order) | ||
| } | ||
|
|
||
| q.sql = q.sql.Limit(page.Limit) |
There was a problem hiding this comment.
should this be moved to the else clause?
There was a problem hiding this comment.
Good catch, fixed in 013fdcb.
|
This is cool! What's the overhead for migration to add the index look like for full pubnet history? |
|
@ire-and-curses I included this in the PR summary:
The new index is around ~1GB as far as I can remember. Can confirm tomorrow. |
|
Super. |
PR Checklist
PR Structure
otherwise).
services/friendbot, orallordocif the changes are broad or impact manypackages.
Thoroughness
.mdfiles, etc... affected by this change). Take a look in the
docsfolder for a given service,like this one.
Release planning
needed with deprecations, added features, breaking changes, and DB schema changes.
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
This commit improves performance of queries generated by
TradesQwhich is used in/tradesendpoint.First, a quick fix. A new multicolumn index was added on
base_asset_id, counter_asset_id, history_operation_id, order. It supportsTradesForAssetPairqueries and improved the speed of such queries from ~5s on average in SDF cluster to a few ms. Adding the index took around 2 minutes in SDF cluster (with full history).Second, I updated queries that contain
base_* = X OR counter_* = Xconditions toUNIONs. Specifically,ForAccountandForOffer. Previous query was using a wrong index (htrd_pidwhich required a lot of filtering) and adding new indices (base_+history_operation_id+order) didn't really help (I believe because ofORstatement). New query decreased execution times down to ms.Known limitations
github.com/Masterminds/squirreldoes not supportUNIONqueries so it's built manually.sccscenarios. A larger refactor is required to support mocking.