cmd: command to dump preimages enumerated in snapshot order, in a flat file #27819
Conversation
Signed-off-by: Ignacio Hagopian <[email protected]>
| } | ||
| exportOverlayPreimagesCommand = &cli.Command{ | ||
| Action: exportOverlayPreimages, | ||
| Name: "export-overlay-preimages", |
There was a problem hiding this comment.
| Name: "export-overlay-preimages", | |
| Name: "export-preimages", |
I mean, it's just exporting preimages, right?
| @@ -374,6 +374,73 @@ func ExportPreimages(db ethdb.Database, fn string) error { | |||
| return nil | |||
| } | |||
|
|
|||
| // ExportOverlayPreimages exports all known hash preimages into the specified file, | |||
| // in the same order as expected by the overlay tree migration. | |||
There was a problem hiding this comment.
Please specify what that "same order" entails
| stIt.Release() | ||
| } | ||
| count++ | ||
| if count%100000 == 0 { |
There was a problem hiding this comment.
Please instead add a log output every 8 seconds or so, Exporting preimages with some stats about how many, how much remaining, how much time elapsed etc.
| fh, err := os.OpenFile(fn, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, os.ModePerm) | ||
| if err != nil { | ||
| return err | ||
| } |
There was a problem hiding this comment.
truncate? Wouldn't it be nicer if we could abort/resume, somehow?
| for accIt.Next() { | ||
| acc, err := types.FullAccount(accIt.Account()) | ||
| if err != nil { | ||
| return fmt.Errorf("invalid account encountered during traversal: %s", err) | ||
| } | ||
| addr := rawdb.ReadPreimage(statedb.Database().DiskDB(), accIt.Hash()) |
There was a problem hiding this comment.
Hm, interesting. So, your way of doing this is to
- Iterate the account-snapshot, (ordered by hashed address)
- For each
h(a), read preimage.
An alternative way to do it would be to iterate the premages: in a first phase, out to an external file. In a second phase, that file could be (iteratively) sorted by post-image instead of pre-image.
It would be interesting to see the differences in speed between the two approaches.
A benefit with the second approach is that it wouldn't be sensitive to new data -- if you get an additional chunk of preimages (either from a few more blocks, or some external source), you could just append them to the "unsorted" file, and then re-sort it.
There was a problem hiding this comment.
Ah, I see now that the order you want is not just "ordered by post-hash", the ordering is "ordered by post-hash account, then the preimages for that account's storage trie preimages". So the final ordering would be
acc1
acc2
storagekey1-acc2
storagekey2-acc2
acc3
Seems like a pretty strange ordering -- and also highly sensitive to changes in state. If you advance one block, you need to redo everything, because the ordering is state-dependent and cannot be performed given only the data itself.
| if acc.Root == types.EmptyRootHash { | ||
| stIt, err := chain.Snapshots().StorageIterator(mptRoot, accIt.Hash(), common.Hash{}) |
There was a problem hiding this comment.
This seems bass-ackwards. No need to create an iterator if the root is the empty root hash. Tho it's faster for sure :)
|
Superseded by #28256 |
This is a rebase of gballet#246 on top of
master.