Deadlock when request is issued when some subscription is active

[svyatonik]

Steps to repeat:

1. open a connection to the ws-server (e.g. substrate node);
2. subscribe to some notifications stream (e.g. to justifications using grandpa_justifications);
3. wait enough time for 5 notifications to be sent by the server (e.g. 20 blocks). DO NOT read any notifications from jsonrpsee::Subscription;
4. make request to the server (e.g. chain_getHeader) using the same client => deadlock.

The reason is that notifications channel is created with buffer size = 4. When we're trying to send 5th notification using blocking SinkExt::send().await here, the background thread will stop processing FrontToBack requests. So if we have single-threaded client, it'll block here, waiting for background thread to issue request and read response. At the same time, background thread is waiting for us to read pending notifications from the channel.

[niklasad1]

Right, so the simple fix is to use an unbounded channel?

It seems tricky to pick the "right" buffer size.

[tomusdrw]

IMHO have it user-configured and drop excessive notifications (they usually have an ephemeral nature anyway). If the user is interested in not missing any of them, then they can set the size of the channel high enough. We can't (and imho shouldn't) really back-pressure the server, it would presumably be dropping excessive notifications if the client can't keep up either (or even dropping the connection) - you can't stop the external stuff that generates these notifications from making progress really :)

[niklasad1]

Fixed by #180