cmd/{geth,utils}: add cmd to export preimages in snap enumeration order

[gballet]

This add a subcommand to only export the preimages of every hash found during a snapshot enumeration. This tool is needed for the verkle transition, in order to distribute the preimages needed for the conversion.

Only the preimages that are necessary for the conversion are dumped, which shrinks the file from 70GB to ~4GB.

The order is that of the snapshot enumeration, which gives a speedup since database thrashing is causing a major slowdown during the conversion.

[MariusVanDerWijden]

Can you do something like
count % 1000 && time.Since(logged) > 8*time.Second here?
That would be in line with the other export commands which try to log every 8 seconds

[MariusVanDerWijden]

You could do something like " [] here, (similar to the the exportCommand), so we don't need to define another flag for the tree root

[holiman]

feed the stIt.Hash() into a channel instead, and anohter goroutine does the rawdb.ReadPreimage on the other end of that channel

[holiman]

You should probably have a buffer here, so the source doesn't have to wait for the sink as much ?

[holiman]

When we're missing a preimage, now it would just die with `invalid preimage size'. Wouldn't it be nicer to spit out at least the missing preimage (and maybe even continue: printing out all missing preimages, so the user can look them up and then redo the conversion?)

[holiman]

Suggested change

	func ExportSnapshotPreimages(chain *core.BlockChain, fn string, root common.Hash) error {
	func ExportSnapshotPreimages(chain *core.BlockChain, fn string, root common.Hash) error {

Perhaps ExportActivePreimages to signify that it is the preimages that are all active or relevant for the given state

[gballet]

TODO after review: move to snapshot.go, rename to something that is clearer and remove (export|import)-preimages commands

[holiman]

• geth export-preimages: exports hash preimages to an RLP encoded stream. It does so by db-iterating thus: db.NewIterator([]byte("secure-key-"), and just rlp-encodes blobs back to back.
• geth db export preimage file. Also uses the same iterator db.NewIterator(rawdb.PreimagePrefix, nil), but the output format is [ byte op, []byte key, []byte value], so I guess the output size is roughly 2x. The output from a db export can be imported via geth db import (generic importer).

geth export-preimages has been deprecated for some time, I'm fine with removing that. As for this one, it might make sense to keep it separate, in order to keep the output smaller (not use the generic format with op|key|value).

As opposed to the other commands, which are only dependant on db, this is tied to state -- we need to actually iterate the state leafs. So IMO this should not reside under geth db ( Low level database operations), so I guess geth snapshot is the most suitable location for it.

[rjl493456442]

I am fine with removing exportPreimagesCommand, but let's keep the importPreimageComand for a while, otherwise we can't import them back.

[holiman]

triage discussion: make the output format rlp, compatible with the old geth export-preimages command, and keep the geth import-preimages so we can import them.

[holiman]

Here's something I've been pondering.. So, right now, we have

• Generic format rlp-stream ( [ { op, key, value } ... ] ) -- can be imported by generic importer, usable by any data, not just preimages. Geth-specific, not convertible to other clients.
• RLP-export forrmat rlp-stream ( preimage, preimage...) in a file. The 'preimage's are just byte-blobs, can be 32-byte, 20-byte (maybe also shorter ones?)

This PR would spit non-deduplicated preimages in "total-order".

• "total order" : Ordering is account-trie, but whenever an account has storage, the order is broken up since a range of storage preimages follows.
• Non-deduplicated: identical storage preimages will be present multiple times.

Now, maybe we should use a third format?

How about:

• Split up account preimages and storage preimages. One file containing 20-byte preimages, one file containing 32 byte preimages.
• The iteration sequence of accounts is still non-thrashing.
• There would be no duplication of storage preimages
• Locating a specific preimage (whether for accounts or storage slots) would be pretty easy, since they are ordered by post-image. The post-image would give a fairly good first estimate of it's location in the file, and then it would be easy to scan to or binary search the remaining space.

from 70GB to ~4GB.

I wonder how large the accounts-file and storage-file would be, respectively, under this schema. If they are small enough, I don't think thrashing on the storage preimages would be much of an issue.

[gballet]

that third scheme would save some space if indeed there is no index and there is a binary search to find the corresponding value. But reading means you need to binary search the slot file to find your hash, and hash it every time, so it's O(n log n) of a rather expensive operation (the linear factor is because this has to be done for every account storage).

It could be parallelized, but this is expected to work in the background during the transition, so there is already massive parallelization going on at the same time; and we would be saving 2GB at most, so I don't think it's worth it. I can give it a try in Istanbul.

[rjl493456442]

lgtm, but I will try to export and import by myself just to ensure.

[rjl493456442]

ubuntu@mm:~/gary/preimage$ ./snapshot-preimage-export snapshot export-preimages --datadir . preimage.gz
INFO [11-10|20:53:13.969] Maximum peer count                       ETH=50 LES=0 total=50
INFO [11-10|20:53:13.971] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [11-10|20:53:13.974] Using pebble as the backing database
INFO [11-10|20:53:13.974] Allocated cache and file handles         database=/home/ubuntu/gary/preimage/geth/chaindata cache=512.00MiB handles=524,288
INFO [11-10|20:53:14.115] Opened ancient database                  database=/home/ubuntu/gary/preimage/geth/chaindata/ancient/chain readonly=true
INFO [11-10|20:53:14.115] State scheme set to already existing     scheme=hash
INFO [11-10|20:53:14.117] Set global gas cap                       cap=50,000,000
INFO [11-10|20:53:14.118] Initializing the KZG library             backend=gokzg
INFO [11-10|20:53:14.139] Using pebble as the backing database
INFO [11-10|20:53:14.139] Allocated cache and file handles         database=/home/ubuntu/gary/preimage/geth/chaindata cache=512.00MiB handles=524,288
INFO [11-10|20:53:14.322] Opened ancient database                  database=/home/ubuntu/gary/preimage/geth/chaindata/ancient/chain readonly=true
INFO [11-10|20:53:14.322] State scheme set to already existing     scheme=hash
INFO [11-10|20:53:14.334] Exporting preimages                      file=preimage.gz
INFO [11-10|20:53:14.335] Exported preimages                       count=16 elapsed="619.407µs" file=preimage.gz


ubuntu@mm:~/gary/preimage$ ./snapshot-preimage-export import-preimages  --datadir ./ ./preimage.gz
INFO [11-10|20:56:18.733] Maximum peer count                       ETH=50 LES=0 total=50
INFO [11-10|20:56:18.736] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [11-10|20:56:18.740] Using pebble as the backing database
INFO [11-10|20:56:18.740] Allocated cache and file handles         database=/home/ubuntu/gary/preimage/geth/chaindata cache=512.00MiB handles=524,288
INFO [11-10|20:56:18.930] Opened ancient database                  database=/home/ubuntu/gary/preimage/geth/chaindata/ancient/chain readonly=true
INFO [11-10|20:56:18.930] State scheme set to already existing     scheme=hash
INFO [11-10|20:56:18.933] Set global gas cap                       cap=50,000,000
INFO [11-10|20:56:18.934] Initializing the KZG library             backend=gokzg
INFO [11-10|20:56:18.963] Using pebble as the backing database
INFO [11-10|20:56:18.963] Allocated cache and file handles         database=/home/ubuntu/gary/preimage/geth/chaindata cache=512.00MiB handles=524,288
INFO [11-10|20:56:19.235] Opened ancient database                  database=/home/ubuntu/gary/preimage/geth/chaindata/ancient/chain readonly=false
INFO [11-10|20:56:19.236] Importing preimages                      file=./preimage.gz
Import done in 16.327101ms

Should work.