Rescan tokens to process missing transactions #202
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…o avoid reprocessing transactions
|
Uplevel your code reviews with CodeRabbit ProCodeRabbit ProIf you like this project, please support us by purchasing the Pro version. The Pro version has advanced context, superior noise reduction and several proprietary improvements compared to the open source version. Moreover, CodeRabbit Pro is free for open source projects. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Files selected (25)
- .golangci.yml (1)
- api/api.go (4)
- api/errors.go (1)
- api/tokens.go (3)
- api/types.go (1)
- cmd/census3/main.go (7)
- db/queries/tokens.sql (1)
- db/treedb/treedb.go (1)
- helpers/web3/web3_client.go (2)
- scanner/const.go (1)
- scanner/helpers.go (1)
- scanner/providers/farcaster/provider.go (2)
- scanner/providers/gitcoin/gitcoin_provider.go (3)
- scanner/providers/gitcoin/gitcoin_provider_test.go (3)
- scanner/providers/holders_provider.go (2)
- scanner/providers/poap/poap_provider.go (2)
- scanner/providers/poap/poap_provider_test.go (1)
- scanner/providers/web3/erc20_provider.go (8)
- scanner/providers/web3/erc721_provider.go (8)
- scanner/providers/web3/erc777_provider.go (8)
- scanner/providers/web3/errors.go (1)
- scanner/providers/web3/web3_provider.go (1)
- scanner/scanner.go (10)
- scanner/updater.go (1)
- scripts/block_logs/main.go (1)
Files not summarized due to errors (25)
- .golangci.yml (nothing obtained from openai)
- api/api.go (nothing obtained from openai)
- api/types.go (nothing obtained from openai)
- api/tokens.go (nothing obtained from openai)
- cmd/census3/main.go (nothing obtained from openai)
- api/errors.go (nothing obtained from openai)
- helpers/web3/web3_client.go (nothing obtained from openai)
- scanner/const.go (nothing obtained from openai)
- db/treedb/treedb.go (nothing obtained from openai)
- db/queries/tokens.sql (nothing obtained from openai)
- scanner/providers/farcaster/provider.go (nothing obtained from openai)
- scanner/helpers.go (nothing obtained from openai)
- scanner/providers/poap/poap_provider.go (nothing obtained from openai)
- scanner/providers/holders_provider.go (nothing obtained from openai)
- scanner/providers/gitcoin/gitcoin_provider_test.go (nothing obtained from openai)
- scanner/providers/web3/erc20_provider.go (nothing obtained from openai)
- scanner/providers/gitcoin/gitcoin_provider.go (nothing obtained from openai)
- scanner/scanner.go (diff tokens exceeds limit)
- scanner/updater.go (diff tokens exceeds limit)
- scanner/providers/poap/poap_provider_test.go (nothing obtained from openai)
- scanner/providers/web3/erc721_provider.go (nothing obtained from openai)
- scanner/providers/web3/web3_provider.go (nothing obtained from openai)
- scanner/providers/web3/erc777_provider.go (nothing obtained from openai)
- scanner/providers/web3/errors.go (nothing obtained from openai)
- scripts/block_logs/main.go (nothing obtained from openai)
Files not reviewed due to errors (25)
- api/types.go (no response)
- api/errors.go (no response)
- cmd/census3/main.go (no response)
- api/tokens.go (no response)
- .golangci.yml (no response)
- api/api.go (no response)
- db/treedb/treedb.go (no response)
- scanner/const.go (no response)
- db/queries/tokens.sql (no response)
- scanner/helpers.go (no response)
- scanner/providers/farcaster/provider.go (no response)
- helpers/web3/web3_client.go (no response)
- scanner/providers/poap/poap_provider_test.go (no response)
- scanner/providers/gitcoin/gitcoin_provider.go (no response)
- scanner/providers/poap/poap_provider.go (no response)
- scanner/providers/holders_provider.go (no response)
- scanner/providers/gitcoin/gitcoin_provider_test.go (no response)
- scanner/providers/web3/erc20_provider.go (no response)
- scanner/providers/web3/errors.go (no response)
- scanner/providers/web3/web3_provider.go (no response)
- scanner/providers/web3/erc721_provider.go (no response)
- scanner/providers/web3/erc777_provider.go (no response)
- scanner/scanner.go (no response)
- scanner/updater.go (no response)
- scripts/block_logs/main.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with 
CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
… iteration an support sorted scan
There was a problem hiding this comment.
Files selected (6)
- cmd/census3/main.go (7)
- scanner/providers/web3/erc20_provider.go (8)
- scanner/providers/web3/erc721_provider.go (8)
- scanner/providers/web3/erc777_provider.go (8)
- scanner/scanner.go (10)
- scanner/updater.go (1)
Files not summarized due to errors (6)
- scanner/scanner.go (diff tokens exceeds limit)
- scanner/updater.go (diff tokens exceeds limit)
- cmd/census3/main.go (nothing obtained from openai)
- scanner/providers/web3/erc20_provider.go (nothing obtained from openai)
- scanner/providers/web3/erc777_provider.go (nothing obtained from openai)
- scanner/providers/web3/erc721_provider.go (nothing obtained from openai)
Files not reviewed due to errors (6)
- scanner/providers/web3/erc721_provider.go (no response)
- scanner/providers/web3/erc777_provider.go (no response)
- scanner/providers/web3/erc20_provider.go (no response)
- cmd/census3/main.go (no response)
- scanner/updater.go (no response)
- scanner/scanner.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
There was a problem hiding this comment.
Files selected (9)
- db/treedb/treedb.go (1)
- scanner/providers/holders_provider.go (2)
- scanner/providers/web3/erc20_provider.go (8)
- scanner/providers/web3/erc721_provider.go (8)
- scanner/providers/web3/erc777_provider.go (8)
- scanner/providers/web3/errors.go (1)
- scanner/providers/web3/web3_provider.go (2)
- scanner/scanner.go (10)
- scanner/updater.go (1)
Files not summarized due to errors (9)
- scanner/providers/web3/erc20_provider.go (diff tokens exceeds limit)
- scanner/providers/web3/erc777_provider.go (nothing obtained from openai)
- scanner/scanner.go (diff tokens exceeds limit)
- scanner/updater.go (diff tokens exceeds limit)
- db/treedb/treedb.go (nothing obtained from openai)
- scanner/providers/web3/web3_provider.go (nothing obtained from openai)
- scanner/providers/web3/erc721_provider.go (nothing obtained from openai)
- scanner/providers/web3/errors.go (nothing obtained from openai)
- scanner/providers/holders_provider.go (nothing obtained from openai)
Files not reviewed due to errors (9)
- scanner/providers/web3/errors.go (no response)
- scanner/providers/holders_provider.go (no response)
- db/treedb/treedb.go (no response)
- scanner/providers/web3/erc721_provider.go (no response)
- scanner/providers/web3/erc777_provider.go (no response)
- scanner/providers/web3/erc20_provider.go (no response)
- scanner/providers/web3/web3_provider.go (no response)
- scanner/updater.go (no response)
- scanner/scanner.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
There was a problem hiding this comment.
Files selected (1)
- scanner/providers/gitcoin/gitcoin_provider_test.go (3)
Files not summarized due to errors (1)
- scanner/providers/gitcoin/gitcoin_provider_test.go (nothing obtained from openai)
Files not reviewed due to errors (1)
- scanner/providers/gitcoin/gitcoin_provider_test.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
There was a problem hiding this comment.
Files selected (3)
- api/tokens.go (3)
- scanner/scanner.go (10)
- scanner/updater.go (1)
Files not summarized due to errors (3)
- scanner/scanner.go (diff tokens exceeds limit)
- scanner/updater.go (diff tokens exceeds limit)
- api/tokens.go (nothing obtained from openai)
Files not reviewed due to errors (3)
- api/tokens.go (no response)
- scanner/scanner.go (no response)
- scanner/updater.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
There was a problem hiding this comment.
Files selected (1)
- api/types.go (1)
Files not summarized due to errors (1)
- api/types.go (nothing obtained from openai)
Files not reviewed due to errors (1)
- api/types.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
p4u
requested changes
Jun 26, 2024
There was a problem hiding this comment.
Files selected (9)
- db/treedb/treedb.go (1)
- scanner/const.go (1)
- scanner/providers/holders_provider.go (2)
- scanner/providers/web3/erc20_provider.go (8)
- scanner/providers/web3/erc721_provider.go (8)
- scanner/providers/web3/erc777_provider.go (8)
- scanner/providers/web3/web3_provider.go (2)
- scanner/scanner.go (10)
- scanner/updater.go (1)
Files not summarized due to errors (9)
- scanner/providers/web3/erc20_provider.go (diff tokens exceeds limit)
- db/treedb/treedb.go (nothing obtained from openai)
- scanner/scanner.go (diff tokens exceeds limit)
- scanner/updater.go (diff tokens exceeds limit)
- scanner/providers/web3/erc721_provider.go (nothing obtained from openai)
- scanner/const.go (nothing obtained from openai)
- scanner/providers/web3/erc777_provider.go (nothing obtained from openai)
- scanner/providers/web3/web3_provider.go (nothing obtained from openai)
- scanner/providers/holders_provider.go (nothing obtained from openai)
Files not reviewed due to errors (9)
- scanner/providers/holders_provider.go (no response)
- scanner/providers/web3/erc20_provider.go (no response)
- scanner/const.go (no response)
- db/treedb/treedb.go (no response)
- scanner/providers/web3/erc721_provider.go (no response)
- scanner/providers/web3/erc777_provider.go (no response)
- scanner/providers/web3/web3_provider.go (no response)
- scanner/updater.go (no response)
- scanner/scanner.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
…ers to add them to the token filter after save holders
lucasmenendez
force-pushed
the
f/rescan_tokens
branch
from
c6cf141
to
cb943ab
Compare
There was a problem hiding this comment.
Files selected (9)
- db/treedb/treedb.go (1)
- scanner/const.go (1)
- scanner/providers/holders_provider.go (2)
- scanner/providers/web3/erc20_provider.go (8)
- scanner/providers/web3/erc721_provider.go (8)
- scanner/providers/web3/erc777_provider.go (8)
- scanner/providers/web3/web3_provider.go (2)
- scanner/scanner.go (10)
- scanner/updater.go (1)
Files not summarized due to errors (9)
- scanner/providers/web3/erc20_provider.go (diff tokens exceeds limit)
- scanner/providers/web3/web3_provider.go (nothing obtained from openai)
- scanner/scanner.go (diff tokens exceeds limit)
- scanner/updater.go (diff tokens exceeds limit)
- scanner/providers/web3/erc777_provider.go (nothing obtained from openai)
- scanner/const.go (nothing obtained from openai)
- db/treedb/treedb.go (nothing obtained from openai)
- scanner/providers/web3/erc721_provider.go (nothing obtained from openai)
- scanner/providers/holders_provider.go (nothing obtained from openai)
Files not reviewed due to errors (9)
- scanner/const.go (no response)
- scanner/providers/holders_provider.go (no response)
- scanner/providers/web3/erc777_provider.go (no response)
- db/treedb/treedb.go (no response)
- scanner/providers/web3/erc721_provider.go (no response)
- scanner/providers/web3/erc20_provider.go (no response)
- scanner/providers/web3/web3_provider.go (no response)
- scanner/scanner.go (no response)
- scanner/updater.go (no response)
Review comments generated (0)
- Review: 0
- LGTM: 0
Tips
Chat with CodeRabbit Bot (@coderabbitai)
- Reply on review comments left by this bot to ask follow-up questions. A review comment is a comment on a diff or a file.
- Invite the bot into a review comment chain by tagging
@coderabbitaiin a reply.
Code suggestions
- The bot may make code suggestions, but please review them carefully before committing since the line number ranges may be misaligned.
- You can edit the comment made by the bot and manually tweak the suggestion if it is slightly off.
Pausing incremental reviews
- Add
@coderabbitai: ignoreanywhere in the PR description to pause further reviews from the bot.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Includes new endpoints to run a token rescan storing just the no processed logs. It includes the use of arbo merkle tree as key-value database to store the processed logs by token.
It also includes a scanner refactor splitting it in two background tasks, the scanner, which now just short the tokens to scan to include them into the updater, and the updater, which manage a queue of tokens to scan, associated to a start and end block. This refactor allows to rescan a token from the API, including it in the updater.