Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade the SQL tables to refactor the code in a more optimised way #52

Merged
merged 0 commits into from
Mar 6, 2023

Conversation

cortze
Copy link
Collaborator

@cortze cortze commented Jan 23, 2023

Motivation

The current crawler looks like a quick patch in our attempt to shift from BoltDB to Postgresql. The logic is crappy as we tend to fetch PeerInfo one by one from the SQL database (which is awful and a waste of CPU cycles).
This PR aims to solve those problems by refactoring the crawler to use better the power of SQL (as we want to keep it because it is pretty nice to post-process the data).

Tasks

  1. Divide the different aspects of the SQL database into different modules/tables
    - [ x] PeerInfo
    - [ x] IpInfo
    - [ x] ConnectionEvents
    - [ x] Network-related data (Ethereum, IPFS, etc)
  2. Apply new logic to the crawler
    - [ ] Optimise the logic of the crawler to get the advantage of the SQL queries
    - [ ] peering Strategy
    - [ ] Make new Prometheus metrics exporters based on the SQL database
  3. Add unitary tests in the individual modules to make maintenance faster

Copy link
Collaborator

@tdahar tdahar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is a big change but it makes it easier to understand the code, which also looks more organized.
Looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants