Archive storage

[Scooletz]

An archive node is an instance of an Ethereum client configured to build an archive of all historical states. There are two queries that it needs to answer:

1. the account information at the given block
2. the storage information at the given block for the given account.

This means that both queries could be transformed into a query that:

1. starts with the account = @account equality,
2. then is followed by storage = @storage (for storage)
3. then is followed by block <= @block as the change might have been applied in the past

It should be possible then to use the same db structure where the key is encoded as a NibblePath by concatenating account + storage + block and mapping some payload to it. This concatenation would allow Paprika's prefixing behavior to potentially make the dataset smaller. Potentially a separate page type with a different flushing behavior and a different fan out could be applied.

From the consumption point of view, archive processing could be a sidecar that processes storage & state slots that are flushed by the Blockchain.FlusherTask method. Then, each block that is applied could be inspected and applied to a separate archive root. This could be done by extending RootPage by only one address that would point to the separate Archive tree.

Examples

1. eth_getBalance 0x1234567890ABCDEF 234234
   1. find first key lower than 0x1234... + 0 + `234234
   2. 0 stands for balance
2. eth_getNonce 0x1234567890ABCDEF 234234
   1. find first key lower than 0x1234... + 1 + `234234
   2. 1 stands for nonce
3. eth_gstorage 0x1234567890ABCDEF 0x1 234234
   1. find id of the account, first lower than 0x1234... + 234234, save to id
   2. find key lower than id + keccak(0x1) + 234234

Current state

Archive node sizes:

1. Nethermind: 15TB
2. Erigon/Reth have a few TB archive, not sure the correct number, but I guess 2-5 TB

Remarks

1. The application of the data could be done in a relaxed manner, where the flush to disk happens only now and then. There are options for this already.
2. Storing or calculating hashes, could be made optional. We could skip memoization of Merkle so that Branch is max 2 bytes instead of 512 bytes.
3. Split Account into Account.Balance, Account.CodeHash, Account.StorageRoot, Account.Nonce so that less frequently changed data are not overwritten over and over again.
4. Querying the state should take into consideration the transient in memory state and the archive.
5. The other option is to have the Archive totally separated. That could be considered, but there's something into having it embedded into the block processing, where the data are fresh, in-memory and ready to be stored by this archiving sidecar.
6. Archive should probably separate the archive tree so that the Merkle and the current state is kept where it is, where the archive is kept separately.
7. Archive SHOULD not use mmap for its storage. It's much too big. This means that manual buffer management will be needed. From writing point of view not that much will change as Paprika uses file write + file cache coherency. From the reading point of view and ownership this will change a lot requiring ownership and copying buffers.

[LukaszRozmej]

There should be a (slow) way to support deep reorganistations with archive storage.
I expect archive storage to be indexed by block (or each transactions within blocks), it would be good to have an option to prune archive storage, keeping some arbitrary number of blocks there (for example from only last year or month).