Introduce Reliability network layer #1074

This code is just an experiment and would require a lot of masssaging to get right...

- Add a test case to make sure the acks are updated on each received/broadcasted message

To be more performant let's use Vectors in the Msg type as well as trace messages.

Rename some binders to make more sense

- We used Sequence to capture messages and List to hold parties. Our benchmarks show worse performance than current master (they are not able to complete at all). By using Vector for everything I am noticing better performance but benchmarks for hydra-cluster stil don't finish. - Also introduce custom exception types for Reliability

After possible callback local acks might change so we need to re-read them from a TVar. Add a party in the BroadcastCounter to improve logging.

We had an issue whereby concurrent call to the callback function from lower layer lead to vector clock losing sync and not allowing progress anymore because observation and changes were not atomic. The resending logic does not seem to much make sense even though it seems to work, we need to analyse the behaviour of the system a bit more...

The Authenticate layer has the signing key to sign messages, so there is no need to require senders to do the work themselves. This is possible because we have decoupled incoming from outgoing messages in the definition of a NetworkComponent.

The idea is that the reliability layer works with heartbeats to send acks which do not change upon a ping, denoting the fact the node cannot make progress by sending new messages and giving the opportunity to peers to resend messages not acknowledged because of transient network failures. This makes the Reliability highly dependent on Heartbeat which raises the question of whether or not we would not like to merge the 2.

This is of course not meant to stay but is an interesting small technique that I thought would be nice to introduce. A similar technique could be used to simulate messages shuffling, or delays, etc. Next commit will revert this.

This reverts commit 86dbd743a3735804bea0220bd140dd7ca1670908.

And do the sorting inside the function and not require a sorted vector

The actual network stack used by the node is becoming more complex, and requires pulling in dependencies which are irrelevant.

We want to ensure alice, bob, carol as Party reflect the sorting of their identifiers

If we receive a messagae with an unappropriate number of peers, we just drop the message as it is quite suspicious.

Can ease test exploration.

Also note that we have an issue with this property that would pass if we drop all the messages.

- Also add a new test case

- We record what each party has seen and remove messages seen by all parties. - Use IntMap instead of Vector for storing sent messages because we need to be able to remove old/seen indices without re-indexing. - Use Map to keep track of the last seen message by party - Introduce a test case to assert the withNetwork logs removal of old messages.

We wire 2 components for alice and bob, mediated through a lossy channel, and check bob receives all messages from alice

It would be better to wait (with a timeout) to receive all broadcast messages

… Alice

* Remove imports * Remove unneeded resize * Tabulate to show length of messages sent * Increase ratio of dropped messages

This test is superseded by the stress test which covers more of it.

How is it that the test fail with `[0] /= [0, 0]` but the debug trace is `[0, 0] /= [0, 0]` ?

Co-authored-by: Sebastian Nagel <[email protected]>

At least not fast enough compared to the time we were giving their messages to arrive. Sending a message once every ten seconds and expecting all the messages to reach the peer in less than 100 seconds does not always work.

Bug was exposed by running: ``` cabal test hydra-node --test-options '-m Reliability --seed 1054015251' ``` The problem was caused by Bob increasing his local view of received messsages from Alice from 15 to 16 when receiving a Ping from Alice when, actually, he never received this message 16 before. As a consequence, Alice would not resend message 16 or, when she resends message 16, Bob would ignore it anyway as it's expecting :x

…other If Alice is lagging behind Bob and Bob is lagging behind Alice then nobody would resend any message to its peer. Here we remove one condition to unlock this.

So we should not include ourself to the `seenMessages` map or, otherwise, in real life, we will never garbage collect.

This is meant to ensure we only try to resend messages whenever the peer is quiescent, which was the original intent of using Pings in the first place in order to avoid resending messages too often. The assumption is that disconnections and messages drop should be few and far between in normal operations and it's therefore fine to rely on the Ping's roundtrip time to check for peers state.

Timeouts are inherently unreliable, esp. given an arbitrary and unknown list of messages and an arbitrary ordering of actions. Tests might fail because one of the peers stops before the other and therefore fails to send Pings which will notify the peer it's missing message, or fail to take into account the peer's Pings. This commit replaces complicated timeout logic with a simple STM-based check that _both_ peers received all the messages.

This was used for GC messages and will be rewritten later

Also it throws an error if we do not find ourselves in the list of all parties. This is an absurd given we included ourselves to the list before sorting.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Reliability network layer #1074

Introduce Reliability network layer #1074

Commits on Oct 4, 2023