Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: gRPC based indexer #83

Open
linuskendall opened this issue Jun 21, 2023 · 0 comments · May be fixed by #183
Open

Proposal: gRPC based indexer #83

linuskendall opened this issue Jun 21, 2023 · 0 comments · May be fixed by #183

Comments

@linuskendall
Copy link
Collaborator

Currently the DAS API stack relies on having a geyser plugin indexer. One proposal would be to add/transition to a geyser grpc based indexer (https://github.com/rpcpool/yellowstone-grpc). This would allow:

  1. Multiple parallel indexers connected to different grpc endpoints (even different providers)
  2. Better failover
  3. Lower cost since Yellowstone gRPC can be delivered as a shared service.

For account updates and transaction updates this should be pretty straight forward, you can subscribe to confirmed or finalized commitment level via gRPC, convert to Plerkle format and push to redis.

You could also implement a block check mode which checks that you have indexed all transactions from a given block and if NOT, fetch that block via gRPC to fill in missing TXs.

There is experimental support in yellowstone-grpc for fetching program account state which would allow snapshots of the Tokenkeg account to be taken for indexing of traditional NFTs. Snapshots could also be handled through one time ingest through a tool like https://github.com/terorie/solana-snapshot-etl which could even run the plerkle plugin to update all accounts at once. With snapshot-etl the order would be:

  1. Start up the geyser grpc ingestion
  2. Run snapshot-etl to fill in all the missing assets while grpc subscriptions are running.
muhitrhn pushed a commit to muhitrhn/digital-asset-rpc-infrastructure that referenced this issue Oct 6, 2023
* feat(das): add task queue depth metric

* count pending tasks only

Co-authored-by: Nikhil Acharya <[email protected]>

---------

Co-authored-by: Nikhil Acharya <[email protected]>
muhitrhn pushed a commit to muhitrhn/digital-asset-rpc-infrastructure that referenced this issue Oct 6, 2023
@kespinola kespinola linked a pull request Apr 10, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant