services/horizon/internal/expingest: Remove orderbook graph from ingestion system #2639
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tamirms
force-pushed
the
orderbook-stream-followup
branch
2 times, most recently
from
490e9bc
to
f037b2e
Compare
tamirms
force-pushed
the
orderbook-stream-followup
branch
from
f037b2e
to
fbd36c2
Compare
bartekn
approved these changes
Jun 1, 2020
| log.WithField("ingestLedger", ingestLedger). | ||
| WithField("lastIngestedLedger", lastIngestedLedger). | ||
| Info("bumping ingest ledger to next ledger after ingested ledger in db") | ||
| ingestLedger = lastIngestedLedger + 1 |
There was a problem hiding this comment.
Maybe we should transition state?
Suggested change
| ingestLedger = lastIngestedLedger + 1 | |
| return resume(lastIngestedLedger), nil |
There was a problem hiding this comment.
I think it's better as is because at this moment in the code we have the select for update lock. if we transition to the state we need to release the select for update and the next node would need to acquire the select for update lock. I think by bumping the ingestLedger we reduce latency in the ingestion process
bartekn
approved these changes
Jun 1, 2020
| const verificationFrequency = time.Hour | ||
| const ( | ||
| verificationFrequency = time.Hour | ||
| updateFrequency = 30 * time.Second |
There was a problem hiding this comment.
Can we change it to 5 sec. so it updates with every ledger (on average)?
Suggested change
| updateFrequency = 30 * time.Second | |
| updateFrequency = 5 * time.Second |
Co-authored-by: Bartek Nowotarski <[email protected]>
7 tasks
bartekn
pushed a commit
that referenced
this pull request
Jun 2, 2020
…ple ledgers (#2649) Before #2639 we were updating the order book graph via ingestion on every single ledger. The order book graph was implemented with the expectation that you must apply changes for every single ledger and no ledgers should be skipped. However, we can no longer maintain that invariant because now we are applying updates to the order book graph via the OrderBookStream. The OrderBookStream periodically polls for new / updated offers from the Horizon DB. It is possible that ingestion has advanced by more than one ledger since the last time OrderBookStream updated the order book graph. Therefore we must relax the restrictions for how updates are applied to an order book graph. We still require ledgers to be strictly increasing when applying updates to an order book graph. However, now we allow updates to span multiple ledgers. We also allow you to remove an offer which is not present in the order book. Consider the case where you apply updates from ledger x. Let's say the next time we update the order book graph ingestion has already advanced to x+2. It is possible that an offer was created in ledger x+1 and in ledger x+2 we are removing the offer. However, our graph was last updated at ledger x, so when we try to remove the offer which was created in ledger x+1 we will panic unless we deliberately skip over offers which are not present in the order book.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Checklist
PR Structure
otherwise).
services/friendbot, orallordocif the changes are broad or impact manypackages.
Thoroughness
.mdfiles, etc... affected by this change). Take a look in the
docsfolder for a given service,like this one.
Release planning
needed with deprecations, added features, breaking changes, and DB schema changes.
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
Follow-up to #2630 which removes orderbook graph ingestion.
Why
After #2630 , the horizon ingesting nodes no longer need to maintain an in memory order book. Instead, the request serving horizon instances will populate their order books by polling offers from the horizon db. This polling process is downstream of ingestion.
Known limitations
[ N/A]