chainHead_storage: Event-based equivalent of closest-descendant-merkle-value

[lexnv]

From the discussion started on: paritytech/substrate#14550, implementing an event-based notification for storage changes might be simpler than fetching the merkle value of some nodes.

One downside of this would be indeed that clients may disconnect and reconnect. In this window of time, the storage might change without the client knowing it.
That could be mitigated by the user fetching-refetching the hash of the value from storage on a fresh boot/disconnection.

I'm not sure how expensive that might be from the light-client perspective.

@tomaka would love to hear your thoughts about the event-based storage changes 🙏
// @paritytech/subxt-team

[tomaka]

cc #10

Shifting from the legacy storage changes notifications to the new system where you explicitly perform requests yourself is one of the main reasons for the new API.

A light client is unable to know the list of storage items that have changed, unless it asks for a call proof of Core_execute_block.
Doing so unfortunately scales linearly with the number of transactions per block. If Polkadot ever reaches multiple hundreds of transactions per second (which is the objective, after all increasing the tx/sec is basically the only reason why Polkadot exists in the first place), then a light client won't be able to follow. The upper bound to the transactions per seconds of a chain is capped to be exactly what a high-end server can process. If this upper bound is reached, a light client would have to be as powerful as this high-end server, which kind of kills the point of light clients.

A light client could download a proof of every storage item being watched at every block, and that's how the legacy API is implemented in smoldot today. However, it is totally realistic here as well that the light client can't follow the rhythm of blocks. There's at least one block every six seconds (often more than one with forks), and if you subscribe to a lot of big items, or that the light client is already under heavy load, or that full nodes are slow, it is not impossible that the storage requests take more than six seconds.
If that happens, then you're screwed. You have no other way to handle the situation than not request the changes of a certain block. And thus the JSON-RPC client might miss a change. That's already the case today with the legacy API.
You've probably never noticed, but the recent events list when using PolkadotJS-on-top-of-smoldot sometimes misses events. That's the reason.

The new API moves this problem to the JSON-RPC client, because the JSON-RPC client knows what it can tolerate or not. For example if you're just watching an account balance, it doesn't matter if you don't know exactly at which block the value has changed, and thus you can delay the storage request until the load is lighter.

[lexnv]

Closing this for now