Random "Digest item must match that calculated."

[ales-tsurko]

When we try to sync from scratch, node panics randomly with "Digest item must match that calculated" with the same build, but at different runs:

Screen.Recording.2024-07-12.at.00.36.47-1.mov

We have a code substitute around the block which the last imported in the log. It's there, because we updated our nodes previously without incrementing runtime version. Without the code substitute, the node panics with the error at the block each time.

Moreover, all our nodes are running by us and synced from db backup up until block 551052, which is far away from 4695, where the panic appears.

And if we try to downgrade the node to the initial version and sync it, the issue disappears.

[bkchr]

Stop using the NativeElseWasmExecutor and just use the WasmExecutor. This should fix your issue.

[ales-tsurko]

Stop using the NativeElseWasmExecutor and just use the WasmExecutor. This should fix your issue.

Thanks for the quick reply! I'll try it and report you back.

[ales-tsurko]

It didn't help. The node still panics randomly with "Digest item must match that calculated."

https://github.com/Atleta-network/atleta/blob/fix/node-wont-sync/node/src/service.rs#L41

[ales-tsurko]

Adding flag -l wasm_substitutes=debug shows, that the panic happens because the substitute was not applied for some reason. I tried even very low blocks like 1 and 9 - the node sometimes behaves like it ignoring codeSubstitutes.

[bkchr]

Hmm. It should not ignore the code substitute. You know that the substitute is only being used until the spec_version changes? Maybe you had another upgrade in between?

[ales-tsurko]

The spec version isn't changed around the block number 500 000. We have 2 substitutes in the chainspec and one of them is on 4695 (where the issue happens). I run the node with updated spec_version, but with the chainspec, containing substitutes.

I'm currently debugging it and I see, that WasmSubstitutes::get returns None for some reason (still figuring out why), but it's just sometimes. It's like 1 out of 6 runs it returns None, but the rest of the time it's Some. I'm on branch polkadot-v1.11.0.

[bkchr]

Good that you are debugging! If you found something, please report back!

[ales-tsurko]

It looks like WasmSubstitutes designed the way, it can have only one substitute per spec version (the substitutes storage field is a HashMap with spec version as a key and the substitute as a value). Because of this WasmSubstitutes::get always returns one of the substitutes. This is too bad for us, as we need to substitute two times at the same spec version :)

[bkchr]

Yeah, this only supports one spec version. Generally the assumption is that the spec_version is increased.