You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Configure ingestion pipeline from a configuration file #1393 Configure ingestion pipeline from a configuration file - would be great to have new Horizon to allow filtering data and turning on/off features. It's not blocking and requires more design process so the other issues are more important.
Ingestion system upgrades (not creating an issue yet). In case the ingestion system is upgraded we currently clear the state and build it from a latest checkpoint. When we'll start adding historical data (txs, ops, effects) this needs to be updated as we shouldn't be removing historical data. Also think about reingestion of historical data.
Ensure history archive and ledger backend are for the same network.
Endpoints
In general I think we should prepend all new endpoints with /exp prefix (including /accounts and /paths unfortunately). The system is still experimental and when we're sure that everything's working correctly we can easily replace old endpoints. List of endpoints with a short summary of actions required below. Separate issues to be created.
No changes planned to the following endpoints:
/
/metrics
POST /transactions
For the following endpoints we need a new processor saving ledgers to a DB (tracked in #1709):
/ledgers
/ledgers/{ledger_id}
/fee_stats - generation maybe in post-processing hook?
For the following endpoints we need a new processor (#1711) saving transactions to a DB (tracked in #1712 ):
/transactions
/transactions/{tx_id}
/ledgers/transactions
/accounts/{account_id}/transactions
For the following endpoints we need a new processor saving operations to a DB:
/operations
/operations/{id}
/ledgers/{id}/operations
/ledgers/{id}/payments
/accounts/{account_id}/operations
/accounts/{account_id}/payments
/transactions/{tx_id}/operations
/transactions/{tx_id}/payments
/payments
We already have a new offers table (and processor updating offers in a database) so should be easy to implement endpoints below (tracked in #1597):
/offers
/offers/{id}
/accounts/{account_id}/offers
/order_book - I think order book graph can be used, it already has sorted order books inside.
/paths (already implemented)
Add more filters ex. by trustline to /accounts. Also we need to add accounts table that will keep all accounts data (ex. signers, trustlines should be in a separate table) (tracked in #1766).
The old ingestion system has this huge switch checking operation type and creating effects based on this. Would be great to use pipeline to pass only transactions with operations a given processor is interested. So something like:
Remaining ingestion issues
Ingestion: verify updating data against stellar-core #1550 Ingestion: verify updating data against stellar-coreEndpoints
In general I think we should prepend all new endpoints with
/exp
prefix (including/accounts
and/paths
unfortunately). The system is still experimental and when we're sure that everything's working correctly we can easily replace old endpoints. List of endpoints with a short summary of actions required below. Separate issues to be created.No changes planned to the following endpoints:
/
/metrics
POST /transactions
For the following endpoints we need a new processor saving ledgers to a DB (tracked in #1709):
/ledgers
/ledgers/{ledger_id}
/fee_stats
- generation maybe in post-processing hook?For the following endpoints we need a new processor (#1711) saving transactions to a DB (tracked in #1712 ):
/transactions
/transactions/{tx_id}
/ledgers/transactions
/accounts/{account_id}/transactions
For the following endpoints we need a new processor saving operations to a DB:
/operations
/operations/{id}
/ledgers/{id}/operations
/ledgers/{id}/payments
/accounts/{account_id}/operations
/accounts/{account_id}/payments
/transactions/{tx_id}/operations
/transactions/{tx_id}/payments
/payments
We already have a new
offers
table (and processor updating offers in a database) so should be easy to implement endpoints below (tracked in #1597):/offers
/offers/{id}
/accounts/{account_id}/offers
/order_book
- I think order book graph can be used, it already has sorted order books inside./paths
(already implemented)Add more filters ex. by trustline to
/accounts
. Also we need to addaccounts
table that will keep all accounts data (ex. signers, trustlines should be in a separate table) (tracked in #1766)./accounts/
(partially)/accounts/{account_id}/
/assets
(should usetrustlines
table, if there's a need: aggregating data every X ledgers for performance) (Implement /assets endpoint using experimental ingestion #1765)All endpoints below are responsible for displaying effects and we need a set of new processors to extract them (see "Effects processors" notes):
/effects
/ledgers/effects
/accounts/{account_id}/effects
/accounts/{account_id}/trades
/accounts/{account_id}/data/{key}
/operations/{id}/effects
/transactions/{tx_id}/effects
/trades
/offers/{id}/trades
/trade_aggregations
Effects processors notes
LedgerTransaction.GetChanges
.switch
checking operation type and creating effects based on this. Would be great to use pipeline to pass only transactions with operations a given processor is interested. So something like:OR, filter operations needed by a given effects group:
both ways will make the code easier to understand and test.
The text was updated successfully, but these errors were encountered: