feat : Insert storage diff to DB #5
Conversation
dvcpull
changed the title
|
Some test results for benchmark: Query code: import requests
import json
url = "http://localhost:9944/"
headers = {"Content-Type": "application/json"}
data = {
"id": 1,
"jsonrpc": "2.0",
"method": "state_getStorageDiff",
"params": ["0x89955cc93a51534104d97a6c409f31ce7a1ef345b232d111cbf1c1c0e0c76fb0", ["26aa394eea5630e07c"]]
}
# Convert the data dictionary to a JSON string
json_data = json.dumps(data)
# Make the API request
response = requests.post(url, data=json_data, headers=headers)
print("block hash", data['params'])
print("time take for reponse", response.elapsed.total_seconds())
# Check if the request was successful (status code 200)
if response.status_code == 200:
print("modified keys", len(response.json()['result'][0]))
print("modified child keys", len(response.json()['result'][1]))
for item in response.json()['result'][0]:
#print(item[0])
print("".join(format(x, '02x') for x in item[0]))
if item[1] != None:
print("".join(format(x, '02x') for x in item[1]))
else:
print("None")
else:
# Print an error message if the request was not successful
print(f"Error: {response.status_code}")
print(response.text)With local node (without prefixes)With local node (with prefixes)With polkadot (block 971264) |
|
At the moment need to make simple manual tests - make a transfer of native tokens. Then get storage diff via RPC and see that returned data contain the changed storage elements. later we will be able to make a test in indexer:
|
Ok, confirmed with test, below is the python code used import requests
import json
from substrateinterface import SubstrateInterface, Keypair
from substrateinterface.exceptions import SubstrateRequestException
url = "http://localhost:9944/"
headers = {"Content-Type": "application/json"}
# init substrate interface
substrate = SubstrateInterface(
url="ws://127.0.0.1:9944",
ss58_format=42,
type_registry_preset='substrate-node-template',
)
def query_storage_diff(block_hash):
data = {
"id": 1,
"jsonrpc": "2.0",
"method": "state_getStorageDiff",
"params": [block_hash]
}
# Convert the data dictionary to a JSON string
json_data = json.dumps(data)
# Make the API request
response = requests.post(url, data=json_data, headers=headers)
print("block hash", data['params'])
print("time take for reponse", response.elapsed.total_seconds())
# Check if the request was successful (status code 200)
if response.status_code == 200:
print("modified keys", len(response.json()['result'][0]))
print("modified child keys", len(response.json()['result'][1]))
expected_keys = [
substrate.create_storage_key(
"System", "Account", ["5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"] #Alice
).to_hex(),
substrate.create_storage_key(
"System", "Account", ["5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty"] #Bob
).to_hex(),
]
print("Expected keys", expected_keys)
for item in response.json()['result'][0]:
#print(item[0])
#print("0x" + "".join(format(x, '02x') for x in item[0]))
hex_key = "0x" + "".join(format(x, '02x') for x in item[0])
if hex_key in expected_keys:
hex_value = "0x" + "".join(format(x, '02x') for x in item[1])
print("found expected key, value is ", hex_value)
else:
# Print an error message if the request was not successful
print(f"Error: {response.status_code}")
print(response.text)
def do_transfer():
keypair = Keypair.create_from_uri('//Alice')
# send balance from alice to bob
call = substrate.compose_call(
call_module='Balances',
call_function='transfer_keep_alive',
call_params={
'dest': '5FHneW46xGXgs5mUiveU4sbTyGBzmstUspZC92UhjJM694ty',
'value': 1 * 10**15
}
)
extrinsic = substrate.create_signed_extrinsic(
call=call,
keypair=keypair,
era={'period': 64}
)
try:
receipt = substrate.submit_extrinsic(extrinsic, wait_for_inclusion=True)
print('Extrinsic "{}" included in block "{}"'.format(
receipt.extrinsic_hash, receipt.block_hash
))
if receipt.is_success:
print('✅ Success, triggered events:')
for event in receipt.triggered_events:
print(f'* {event.value}')
return receipt.block_hash
else:
print('⚠️ Extrinsic Failed: ', receipt.error_message)
except SubstrateRequestException as e:
print("Failed to send: {}".format(e))
## lets do a transfer onchain
block_hash = do_transfer()
query_storage_diff(block_hash) |
Great news!! |
|
I googled now about child tries in Substrate https://docs.substrate.io/learn/state-transitions-and-storage/ https://substrate.stackexchange.com/questions/1771/how-to-create-an-iterator-for-a-child-storage-trie https://www.shawntabrizi.com/assets/presentations/substrate-storage-deep-dive.pdf I guess this is what we have in "modified child keys".
|
|
One more point. The node has the console launch params: Does this mechanic affect the stored state diffs? Ideally we need a separate param |
Yes, our logic will only work when the node is started with |
The last version of node has two params.
I.e. For the indexer' nodes we will use Ideally state diffs should follow the same param as the state itself - |
Can confirm the node works as expected when using the flags |
After digging through code and docs I think modified child keys are only relevant if they are used by pallets, aka they do not affect how regular pallet storage works, and I could find only pallet crowdloan that uses this child tries. This video is very helpful (https://www.youtube.com/watch?v=9S8rmW8LD5o) child tries start from ~31min |
There was a problem hiding this comment.
Everything is good!!!
One last optimisation and work is done 🎉🎉🎉
And let's update the ticket paritytech/json-rpc-interface-spec#108
Insert storage changes and child storage changes to DB, these values are then used to return the storage diff for blocks from
storage_diffendpoint.