Add new lotus-shed command for backfillling actor events #11088
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fridrik01
force-pushed
the
backfill-events-cli
branch
2 times, most recently
from
6fe2258 to
ed3a508
Compare
fridrik01
changed the title
Stebalien
approved these changes
Jul 20, 2023
Comment on lines
+70
to
+81
| currTs, err := api.ChainHead(ctx) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if cctx.IsSet("from") { | ||
| // we need to fetch the tipset after the epoch being specified since we will need to advance currTs | ||
| currTs, err = api.ChainGetTipSetByHeight(ctx, abi.ChainEpoch(cctx.Int("from")+1), currTs.Key()) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| } |
There was a problem hiding this comment.
nit: no need to call ChainHead if we're just going to call ChainGetTipSetByHeight.
|
|
||
| // select the highest event id that exists in database, or null if none exists | ||
| var entryID sql.NullInt64 | ||
| err = stmtSelectEvent.QueryRow( |
There was a problem hiding this comment.
Consider "insert where not exists" instead of checking then inserting.
fridrik01
force-pushed
the
backfill-events-cli
branch
from
ed3a508 to
dafa837
Compare
fridrik01
force-pushed
the
backfill-events-cli
branch
from
dafa837 to
7192151
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related issue: #10807
Replaces previous PR: #10939
Context
When lotus is configured to enable indexing of actor events (when
EnableEthRPC=trueandDisableHistoricFilterAPI=false) then Lotus will observe for new tipset and store all the actor events in a sqlite table stored atsqlite/events.db.We however, don't have any way to backfill these actor events. This PR therefore implements this backfilling using a new lotus-shed` command.
Note that this implementation does not compute state (it only looks up the receipts so they need to exist on chain).
Test plan
Backfill active node that is already storing events
As expected, backfilling should not have any side effects on a node that is already saving actor events.
Backfill a node that had never stored any events
Verify backfill results
I compared the
events.dbthat was generated from scratch via backfill to anotherevents.dbwhich was generated from by an active lotus node. I attached the second database asevents2and used the following sql to verify there are no missing events: