Allow IoBuffer to hold multiple packets

[algesten]

This is a potential fix for #530.

The story here is that the openssl crate has an abstraction layer SslStream which is generic over something that implements Read/Write. str0m is push/pull which means we need an "adapter" to convert between these two methods – that adapter is called IoBuffer.

I have assumed the operation of the IoBuffer would be such that each time we push an incoming packet to it, the SslStream needs to read that packet completely. The assertion tripped in #530 seem to indicate that doesn't always happen.

This PR relaxes this assumption to allow IoBuffer to potentially hold a few unconsumed packets. Because there is no repro, this is a shot in the dark.

[OxleyS]

I notice the Read impl for IoBuffer asserts that the buffer is big enough to consume the entire incoming buffer. If this assertion did not trip, but nevertheless self.incoming.is_empty() was false on the next set_incoming() call, wouldn't that mean read was never called?

[algesten]

@OxleyS yeah that's what I think too.

Would it be possible for you to test with this branch and that log message I just added? I want to see if this happens.

[OxleyS]

Sure, I can't promise too much since the panic has so far been extremely rare, but I can certainly throw it up there and see what happens.

Other possibilities to consider are behavior if handle_receive() was called again after some sort of error, or if the read inside OpenSSL was spuriously interrupted by a signal (the latter highly unlikely, since IoBuffer is just a buffer of bytes).

[OxleyS]

A small update on this - we've been running our servers with this fix since Wednesday last week, and so far haven't seen a panic nor the Data remaining in IoBuffer log line. We'll keep on running this and keeping an eye on it, though.

[algesten]

@OxleyS sounds promising. How frequent was it before the fix?

[OxleyS]

It only happened once, so it must be some rather freak conditions that trigger it.
If we haven't seen a panic nor the log line, that means those conditions have not happened yet, correct? I think we would need to see one or the other to know if this fix is working.

[algesten]

Yeah. Let's just give it time.

[algesten]

@OxleyS I rebased this PR off latest main so you can get both this fix and the other log changes.

[OxleyS]

While I admit that I haven't been super diligently checking (with a vacation putting me out of the office for a while), I still haven't seen any panics nor log lines.

How should we proceed? Personally I feel that whether it fixes #530 or not, this PR is good to have as a "just in case" guard (minus the logging of course). Especially since the happy path is almost certainly taken. I haven't done any benchmarking though.

[algesten]

Yeah. Let's land it. I'll remove the logging and get it in later today. Thanks for checking!