Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(scanner): Start scanning where it left off last time #8022

Closed
3 of 7 tasks
Tracked by #7728
oxarbitrage opened this issue Nov 28, 2023 · 3 comments · Fixed by #8080
Closed
3 of 7 tasks
Tracked by #7728

feat(scanner): Start scanning where it left off last time #8022

oxarbitrage opened this issue Nov 28, 2023 · 3 comments · Fixed by #8080
Assignees
Labels
A-blockchain-scanner Area: Blockchain scanner of shielded transactions I-slow Problems with performance or responsiveness

Comments

@oxarbitrage
Copy link
Contributor

oxarbitrage commented Nov 28, 2023

Motivation

We discussed in the Zebra team for the need of restarting a scan that was interrupted, left behind or if new keys are added to the config after a restart. Ideally the scan for a key should start around where it was left when applicable.

Suggested Design

A possible design might be:

  • Insert new keys into the database with the height before their birthday height (done in PR change(scan): Use the on-disk database for keys and results #8036)
  • Add a database method to get the last scanned height for each key in the database, by getting the last value in each key's range (based on the existing code for getting the first value in each key's range)
  • When scanning blocks:
    • Start from the lowest height for any key.
    • Only scan with the keys that haven't scanned that height already.

Testing

  • Make sure cached blocks at or below the last scanned height aren't scanned
  • Make sure cached blocks above the last scanned height are scanned
  • Verify that all incoming blocks within zebrad are scanned
  • Test with different heights for different keys
@oxarbitrage oxarbitrage added S-blocked Status: Blocked on other tasks P-Medium ⚡ A-blockchain-scanner Area: Blockchain scanner of shielded transactions labels Nov 28, 2023
@mpguerra mpguerra added this to Zebra Nov 28, 2023
@github-project-automation github-project-automation bot moved this to 🆕 New in Zebra Nov 28, 2023
@teor2345
Copy link
Contributor

I updated this design after the changes in PR #8036

@mpguerra
Copy link
Contributor

mpguerra commented Dec 1, 2023

Hey team! Please add your planning poker estimate with Zenhub @arya2 @oxarbitrage @teor2345 @upbqdn

@teor2345
Copy link
Contributor

teor2345 commented Dec 4, 2023

I updated the tests based on what we skipped for other tickets. Feel free to open more tickets for those tests.

@oxarbitrage oxarbitrage self-assigned this Dec 4, 2023
@oxarbitrage oxarbitrage removed the S-blocked Status: Blocked on other tasks label Dec 5, 2023
@teor2345 teor2345 added the I-slow Problems with performance or responsiveness label Dec 6, 2023
@mpguerra mpguerra linked a pull request Dec 11, 2023 that will close this issue
7 tasks
@mergify mergify bot closed this as completed in #8080 Dec 13, 2023
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Zebra Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-blockchain-scanner Area: Blockchain scanner of shielded transactions I-slow Problems with performance or responsiveness
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants