Ran out of free wasm instances, when using runtime_api

[seunlanlege]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

In the Hyperbridge node, we provide an RPC for querying ISMP events which calls into the runtime using the runtime api.

https://github.com/polytope-labs/hyperbridge/blob/main/modules/ismp/pallets/rpc/src/lib.rs#L423

The problem is if this RPC gets called with a very large block range or perhaps too frequently, users start to observe the following error:

Which eventually causes the node to fall out of sync.

Steps to reproduce

No response

[tmpolaczyk]

I believe this is harmless, as a new instance will be allocated anyway. See here, maybe someone more familiar with the code can confirm.

polkadot-sdk/substrate/client/executor/src/wasm_runtime.rs

Line 143 in aff3a07

let mut instance = self.module.new_instance()?;

Not sure why the warning exists, can it be removed?

[seunlanlege]

I believe this is harmless, as a new instance will be allocated anyway. See here, maybe someone more familiar with the code can confirm.

My question is why does it stall block syncing?

[bkchr]

Does it really stall or is it just extremely slow? Tell them to set --max-runtime-instances 32 and see if that helps.

[bkchr]

paritytech/substrate#8846 generally this would prevent this by having some priority based cache for consensus related actions.

[seunlanlege]

Does it really stall or is it just extremely slow?

It actually stalls, from the attached image. Its syncing at 0.0bps, while serving a lot of rpc requests

[bkchr]

Can you please ask them to run with -lsync=trace and post the logs when it stalls?