Reduce busywaiting in ChannelStream components

[mmirate]

When I want to read some bytes from the server, but I don't expect output to be available to be read immediately, I observe that even when the resulting system is "idle", the read call consumes an entire CPU core!

The cause is fairly straightforward: calling Waker::wake_by_ref() in poll and similar methods, in circumstances where the method didn't make any progress before activating the Waker, means a busywait. When there is an underlying asynchronous operation, proper behavior is obtained by passing the Waker to the underlying operation's poll method.

In this case of the ChannelRx, I propose a very simple solution, made possible by https://docs.rs/tokio/latest/tokio/sync/mpsc/struct.UnboundedReceiver.html#method.poll_recv. Using this, I observe that I am able to tail -f an infrequently-written logfile with no CPU usage unless logfile contents are being transferred and processed.

grep told me that ChannelTx has a similar problem; because its body is less simple than "await a future then do synchronous things with the result", my proposed solution is markedly less simple. It is also not 100% complete; it will still busywait for the window size to be increased from zero; the obvious answer to that would be a condvar, except that doing these 100% correctly in async Rust appears to be an unsolved problem and thus there isn't (yet) a condvar in tokio::sync.

I've placed these in separate commits so that the ChannelTx change may be set aside if it is undesirable.

[Eugeny]

Thank you! Sorry for the delay reviewing, I needed to really wrap my head around this. While I agree that the Tx implementation is not the simplest, it's still a significant improvement, so I'm merging both 🙌