Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services/horizon: Add the ability to toggle ingestion of Soroban elements #5175

Closed
sreuland opened this issue Jan 19, 2024 · 1 comment
Closed

Comments

@sreuland
Copy link
Contributor

sreuland commented Jan 19, 2024

What problem does your feature solve?

Some Change and Transaction processors in the ledger ingestion pipeline are parsing Soroban transaction metadata and are extracting data. These include, at minimum:

  • the AssetStatsProcessor which extensively tracks changes to wrapped Stellar assets (see ingestAssetContractMetadata) - we discovered fundamental conflict here due to asset balances are based on trustlines/cb/lp entries which can also be changed by contracts and state verifier needs the contract tx-meta to re-assemble the full asset state value to compare against archive ledger entries of which trustlines/cb/lp include contract changes.
  • the EffectProcessor which produces (account|contract)_(credited|debited) effects (see ingestEffects)
  • history_transactions table should have serialized base64 for an empty xdr.TransactionMetaV3{} intx_meta, and for the API resource model emitted for Transaction for this row needs to show the same value for
    result_meta_xdr
  • 'asset_balance_changesindetailsof API Operation resource model, will be empty when Op is of type InvokeHostFunctionOp`, i.e. it wont parse contract events from tx meta to derive this in operations processor.

(We should take care to ensure the list above is complete, it may be non-exhaustive.)

These may incur significant increases in the compute time needed to finish ingesting a ledger when the Soroban transaction volume increases, and we need a way to disable this at runtime as a safety valve.

What would you like to see?

A new Horizon configuration setting, DISABLE_SOROBAN_INGEST_PROCESSORS, which defaults to FALSE. When it's TRUE, it will trigger specific routines in the aforementioned processors to skip Soroban-related parsing/extraction.

Note: This should target the v2.28.0 release first, not master.

What alternatives are there?

Wait for Horizon's ingestion time to blow up as Soroban resource limits increase.

@sreuland sreuland moved this from Backlog to Current Sprint in Platform Scrum Jan 19, 2024
@Shaptic Shaptic changed the title services/horizon: config setting to disable soroban aspects in change/tx ingest processing services/horizon: Add the ability to toggle ingestion of Soroban elements Jan 20, 2024
@Shaptic Shaptic added this to the Soroban Pubnet Release milestone Jan 20, 2024
@sreuland sreuland moved this from Current Sprint to In Progress in Platform Scrum Jan 24, 2024
sreuland added a commit to urvisavla/go that referenced this issue Jan 24, 2024
sreuland added a commit to urvisavla/go that referenced this issue Jan 24, 2024
sreuland added a commit to urvisavla/go that referenced this issue Jan 25, 2024
@sreuland
Copy link
Contributor Author

#5174 merged

@github-project-automation github-project-automation bot moved this from In Progress to Done in Platform Scrum Jan 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

3 participants