services/horizon/ingest: Batch Offers insert/updates #3917
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tamirms
reviewed
Sep 14, 2021
tamirms
approved these changes
Sep 14, 2021
bartekn
added a commit
that referenced
this pull request
Sep 14, 2021
This commit adds a new `UpsertOffers` method (and generic `upsertRows`) and modifies `OffersProcessor` to use it. The performance of `OffersProcessor` degraded recently due to number of offers in the network and batching updates should improve the performance of the processor and ingestion overall.
5 tasks
2opremio
reviewed
Sep 22, 2021
| @@ -14,8 +14,7 @@ type QOffers interface { | |||
| GetOffersByIDs(ctx context.Context, ids []int64) ([]Offer, error) | |||
| CountOffers(ctx context.Context) (int, error) | |||
| GetUpdatedOffers(ctx context.Context, newerThanSequence uint32) ([]Offer, error) | |||
| NewOffersBatchInsertBuilder(maxBatchSize int) OffersBatchInsertBuilder | |||
| UpdateOffer(ctx context.Context, offer Offer) (int64, error) | |||
| UpsertOffers(ctx context.Context, offers []Offer) error | |||
| RemoveOffers(ctx context.Context, offerIDs []int64, lastModifiedLedger uint32) (int64, error) | |||
There was a problem hiding this comment.
RemoveOffers can be replaced by UpsertOffers (since a removal is implemented by a tombstone which be set with UpserOffers).
This would save us (up to) a query in each offers processor iteration.
CC @bartekn
There was a problem hiding this comment.
I'd like to keep it separated for the reasons explained in #3944 (comment).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
PR Structure
otherwise).
services/friendbot, orallordocif the changes are broad or impact manypackages.
Thoroughness
.mdfiles, etc... affected by this change). Take a look in the
docsfolder for a given service,like this one.
Release planning
needed with deprecations, added features, breaking changes, and DB schema changes.
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
This commit adds a new
UpsertOffersmethod (and genericupsertRows) and modifiesOffersProcessorto use it.Close #3913.
Why
The performance of
OffersProcessordegraded recently due to number of offers in the network and batching updates should improve the performance of the processor and ingestion overall.Known limitations
The other processors should probably switch to
upsertRowsmethods. This can be done in a separate PR.