RPC to return storage diff between start_block and end_block

[dvcpull]

This rpc will allow to quickly fetch the list of keys that have been changed/deleted between two different blocks, this is useful for chain analysis tools like an indexer and can then be used for processing.

The api would look like :

/// Returns a storage diff between given block and previous block
#[method(name = "state_getStorageDiff", aliases = ["state_getStorageDiffAt"], blocking)]
fn storage_diff(&self, block: Hash, included_prefixes: Option<Vec<StorageKey>>, excluded_prefixes: Option<Vec<StorageKey>>, include_modified_child_tries: Option<bool>) -> RpcResult<(StorageCollection, Option<ChildStorageCollection>)>;

Example :
curl -sS -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "state_getStorageDiff", "params": ["0xaa254e5ff33843f4abfb2527c7273239f395f7f7da10b6018646f4ae7257f913", "0xb2990969fd7771959f534d3d985603b5632fb6a3faf628d7ba79c564424de7ae"]}' http://localhost:9944/

Result :

{"jsonrpc":"2.0","result":[["0x26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac","0x33000000"],["0x26aa394eea5630e07c48ae0c9558cef78a42f33323cb5ced3b44dd825fda9fcc","0xda6e43000582e224d717d60d1fa65ef7bc5b05ccb4ff412c67f8d334a86c8cbd"],["0x26aa394eea5630e07c48ae0c9558cef799e7f93fc6a98f0874fd057f111c4d2d","0x04066175726120baf1d91000000000"],["0x26aa394eea5630e07c48ae0c9558cef7a44704b568d21667356a5a050c1187468e233f9722d30ebf31000000","0xaa254e5ff33843f4abfb2527c7273239f395f7f7da10b6018646f4ae7257f913"],["0x26aa394eea5630e07c48ae0c9558cef7a44704b568d21667356a5a050c118746926f3965faf45f0f16000000","0x403edf6b6b63e8f0402b4815ef63efd29628e29e92c9931012d7470b7730cd0c"],["0x26aa394eea5630e07c48ae0c9558cef7a44704b568d21667356a5a050c1187469d7a9a93920005ae1d000000","0xe7cd5c4f22a46446a36066dffb9c303ceb8d00c2d0a421c56d6ee1b5daf86593"],["0x26aa394eea5630e07c48ae0c9558cef7a44704b568d21667356a5a050c1187469eb2dcce60f37a2702000000","0x75d57415c5640f1c184a9424e01b4979a74f0b6f19a6e41bee3df52eb3b8d4a3"],["0x26aa394eea5630e07c48ae0c9558cef7a44704b568d21667356a5a050c118746a6b274250e6753f00a000000","0xfdd294deb2e99978a9a24eb8eee81de72d839f6c27588ba255e29050e9799cff"],....

Initial Implementation

paritytech/polkadot-sdk#1864

In our first implementation, we chose to read all the keys from the state and loop through the keys to generate the storage diff. Although this worked in our initial testing, when testing with the polkadot chain the performance is very slow (approx 1.5min) when blocks > 1million. We realised this is due to the fact that there is too many keys, iterating and processing each individual key to generate storage diff is too slow for our usecase.

Latest Implementation

traittech/polkadot-sdk#5

In our latest implementation, we are adding the storage_diff generated during every block import to the backend DB. This ensures that we have the storage_diff for block (X,X-1) always ready without any extra processing. We do not observe any adverse effects on storage, but the rpc response time has improved considerably (1.5min -> 0.002858 secs)