Design backend for serving TxMeta

[tamirms]

The results from https://docs.google.com/document/d/1YETNALx5EzqZDNSVWzTfaK5Ogw84PsBlrt64nOr-Njg/edit?usp=sharing show that precomputing TxMeta is very effective in speeding up ingestion. We need to design a solution for distributing TxMeta so Horizon operators (along with Hubble) can ingest precomputed TxMeta instead of relying on captive core.

Using a blobstore like S3 or GCS (google cloud storage) is appealing because then we don't need to build any infrastructure to serve requests for TxMeta. However, there are concerns about download latency being too slow. Perhaps these concerns can be mitigated by using cloudflare for caching and batching several ledgers together in one file.

To complete this issue we need a design document which proposes a solution for distributing TxMeta and analyzes the cost and performance of the solution.

[sreuland]

I'd throw kafka broker in for consideration as 'blobstore' candidate and alternative to proprietary cloud options. Available on most cloud infra providers as 'managed' deployment or can deploy internal(but, requires substantial ops support). Kafka has been used in terabyte situations and has a client-broker transport for throughput, h/a, message delivery. ideas on model would be like message_id: <ledger_id> and message_payload: <base64_ledger_txmeta> additional kafka message headers could be added for letting clients do pro-active filtering/routing based on other attributes of the ledger.

the notion of consumer offest in the protocol could be interesting way to enable random access to ledgers by sequence number, and thereby allowing clients to consume historical ledgers in a custom replay ranged use case, synonymous with reingest range <from> <to>.

using the kafka message offset for random access to ledgers would entail using a single partition topic strategy and initially publishing messages to the topic starting with genesis ledger as offset=0, that way the insertion order of messages is preserved and the offset mirrors the ledger sequence number such as ledgerN will be at offsetN-1.

[sreuland]

may want to include as part of HLD(high level design) if/how the existing mono-repo ingestion sdk will enable access to new TxMeta source, new ledgerbackend? identify which programming languages are highly desired for ingestion sdk with this new TxMeta capability but not present yet, is the mono-go the only one currently, are any other languages minimally required as part of new TxMeta solution?

also, have the ingestion sdk provide outbound path from the ledgerbackend interface, so have the ability for apps to build publishers to remote tx meta sources as well, same sdk can be pub and sub sides.

[mollykarcher]

may want to include as part of HLD(high level design) if/how the existing mono-repo ingestion sdk will enable access to new TxMeta source, new ledgerbackend? identify which programming languages are highly desired for ingestion sdk with this new TxMeta capability but not present yet, is the mono-go the only one currently, are any other languages minimally required as part of new TxMeta solution?

also, have the ingestion sdk provide outbound path from the ledgerbackend interface, so have the ability for apps to build publishers to remote tx meta sources as well, same sdk can be pub and sub sides.

+1 to this. I think it makes sense to productize/productionalize both the publish and consumption side. That is, there is an SDK/package that produces this new TxMeta ledger backend, and there is also one that consumes from it. I'm not particularly opinionated on whether those should be the same package or different packages, but I think keeping them both in the ingest package could definitely make sense.

[sreuland]

@chowbao , just wanted to reference this design ticket earmarked for remote tx meta storage, as you're design proposal is overlapping, we'll want to incorporate your summary here, thanks!