Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save blockchain data to elasticsearch database #6270

Closed
7 tasks done
oxarbitrage opened this issue Mar 6, 2023 · 0 comments · Fixed by #6274 or #6295
Closed
7 tasks done

Save blockchain data to elasticsearch database #6270

oxarbitrage opened this issue Mar 6, 2023 · 0 comments · Fixed by #6274 or #6295
Assignees
Labels
A-diagnostics Area: Diagnosing issues or monitoring performance

Comments

@oxarbitrage
Copy link
Contributor

oxarbitrage commented Mar 6, 2023

We want to export block data (the whole block history) into an elasticsearch database so we can use kibana for data exploration.

TODO

  • Use Rust elasticsearch crate in order to save blocks into elasticsearch.
    • Use a bulk request when wee are far away from the tip.
    • Reduce the bulk size when we get closer to the tip so we can have real time data in kibana as the chain is in sync.
  • Hide all elasticsearch code behind a rust feature.
  • Add a config section for elastic host, username and password.
  • Manual test the full thing in Testnet.
  • Create an entry in the book on how to setup and run.
@oxarbitrage oxarbitrage self-assigned this Mar 6, 2023
@mpguerra mpguerra added this to Zebra Mar 6, 2023
@github-project-automation github-project-automation bot moved this to 🆕 New in Zebra Mar 6, 2023
@oxarbitrage oxarbitrage added P-Low ❄️ A-diagnostics Area: Diagnosing issues or monitoring performance labels Mar 6, 2023
@mpguerra mpguerra linked a pull request Mar 9, 2023 that will close this issue
6 tasks
@mergify mergify bot closed this as completed in #6274 Mar 13, 2023
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Zebra Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-diagnostics Area: Diagnosing issues or monitoring performance
Projects
Archived in project
1 participant