Migrate Horizon to new ingestion

[bartekn]

Remaining ingestion issues

• Make sure fee and ledger upgrade changes are processed #1470 Make sure fee and ledger upgrade changes are processed - when a protocol upgrade happens it's possible that the state is updated too. New ingestion must apply these changes.
• Ingestion: verify updating data against stellar-core #1550 Ingestion: verify updating data against stellar-core
• Configure ingestion pipeline from a configuration file #1393 Configure ingestion pipeline from a configuration file - would be great to have new Horizon to allow filtering data and turning on/off features. It's not blocking and requires more design process so the other issues are more important.
• Ingestion system upgrades (not creating an issue yet). In case the ingestion system is upgraded we currently clear the state and build it from a latest checkpoint. When we'll start adding historical data (txs, ops, effects) this needs to be updated as we shouldn't be removing historical data. Also think about reingestion of historical data.
• Ensure history archive and ledger backend are for the same network.

Endpoints

In general I think we should prepend all new endpoints with /exp prefix (including /accounts and /paths unfortunately). The system is still experimental and when we're sure that everything's working correctly we can easily replace old endpoints. List of endpoints with a short summary of actions required below. Separate issues to be created.

---

No changes planned to the following endpoints:

• /
• /metrics
• POST /transactions

For the following endpoints we need a new processor saving ledgers to a DB (tracked in #1709):

• /ledgers
• /ledgers/{ledger_id}
• /fee_stats - generation maybe in post-processing hook?

For the following endpoints we need a new processor (#1711) saving transactions to a DB (tracked in #1712 ):

• /transactions
• /transactions/{tx_id}
• /ledgers/transactions
• /accounts/{account_id}/transactions

For the following endpoints we need a new processor saving operations to a DB:

• /operations
• /operations/{id}
• /ledgers/{id}/operations
• /ledgers/{id}/payments
• /accounts/{account_id}/operations
• /accounts/{account_id}/payments
• /transactions/{tx_id}/operations
• /transactions/{tx_id}/payments
• /payments

We already have a new offers table (and processor updating offers in a database) so should be easy to implement endpoints below (tracked in #1597):

• /offers
• /offers/{id}
• /accounts/{account_id}/offers
• /order_book - I think order book graph can be used, it already has sorted order books inside.
• /paths (already implemented)

Add more filters ex. by trustline to /accounts. Also we need to add accounts table that will keep all accounts data (ex. signers, trustlines should be in a separate table) (tracked in #1766).

• /accounts/ (partially)
• /accounts/{account_id}/
• /assets (should use trustlines table, if there's a need: aggregating data every X ledgers for performance) (Implement /assets endpoint using experimental ingestion #1765)

All endpoints below are responsible for displaying effects and we need a set of new processors to extract them (see "Effects processors" notes):

• /effects
• /ledgers/effects
• /accounts/{account_id}/effects
• /accounts/{account_id}/trades
• /accounts/{account_id}/data/{key}
• /operations/{id}/effects
• /transactions/{tx_id}/effects
• /trades
• /offers/{id}/trades
• /trade_aggregations

Effects processors notes

• Should be easy to do with LedgerTransaction.GetChanges.
• The old ingestion system has this huge switch checking operation type and creating effects based on this. Would be great to use pipeline to pass only transactions with operations a given processor is interested. So something like:

-> FilterTransactionsProcessor{OperationType: CreateAccount}
  -> CreateAccountEffectsProcessor{}
-> FilterTransactionsProcessor{OperationType: Payment}
  -> PaymentEffectsProcessor{}

OR, filter operations needed by a given effects group:

-> FilterTransactionsProcessor{OperationType: []{CreateAccount, Payment, PathPayment, ...}
  -> AccountEffectsProcessor{}
-> FilterTransactionsProcessor{OperationType: []{SetOptions}
  -> SignerEffectsProcessor{}

both ways will make the code easier to understand and test.

• Not in a scope of the this issue but in a future pipeline will also update trade aggregations buckets (Optimize trade aggregations query #632).

[ire-and-curses]

🎉 🎉 🎉 🎉 🎉