Fix deadlock in network-devp2p

[AtkinsChang]

Fix two deadlock in devp2p module.

Reproduce

deadlock will occur in 5 minutes with this docker-compose.yaml.

version: '3'
services:
  parity:
    image: parity/parity:latest
    command: --chain kovan --unsafe-expose --jsonrpc-apis=all --node-key=test --no-warp

  rpc_client:
    image: alpine
    command:
      - sh
      - -c
      - |-
        apk --no-cache add curl jq
        while :
          do
          curl --data '[
            {"method":"parity_addReservedPeer","params":["enode://5ff409de959e62e822273a53e42a603a81676538f58b8c708014ff057e48d2ec15c1273ff963ea162ccba8358587143e5d694564eeffd0cb3f9520272fb3c48e@parity:30303"],"id":1,"jsonrpc":"2.0"},
            {"method":"eth_syncing","params":[],"id":2,"jsonrpc":"2.0"}
          ]' -H "Content-Type: application/json" -X POST parity:8545 2>/dev/null | jq
        done
    depends_on:
      - parity

Explanation

In fn connect_peers(&self, io: &IoContext<NetworkIoMessage>), try to acquire mutex for Session while holding r-lock for reserved_peers:
https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L576
https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L603-L604

in fn session_readable(&self, token: StreamToken, io: &IoContext<NetworkIoMessage>), there are two code path trying to acquire r-lock for reserved_peers while holding mutex for Session:

• https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L725-L730
• https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L744
  https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L768

In the situation that:

• thread A is in connect_peers holding r-lock for reserved_peers and waiting the session mutex
• thread B is in session_readable holding mutex for session and trying to acquire r-lock for reserved_peers

if there is another thread C acquiring w-lock for reserved_peers, thread B will be blocked until there are no more writers which hold / waiting for the lock cause the dead lock among three threads.

Another deadlock is cause by reserved_nodes and node in fn update_nodes(&self, _io: &IoContext<NetworkIoMessage>, node_changes: TableUpdates) and fn connect_peers(&self, io: &IoContext<NetworkIoMessage>).

[parity-cla-bot]

It looks like @AtkinsChang signed our Contributor License Agreement. 👍

Many thanks,

Parity Technologies CLA Bot

[dvdplm]

Deadlocks for me in less than a minute.

[tomusdrw]

Amazing catches, thanks a lot!

[tomusdrw]

Are there any other occurences where we need reserved_nodes and session.lock()?

As a rule of thumb, we try to keep the lock order in the same order as fields declaration. Here we lock something that is not part of the original struct, but since it's retrieve from self.sessions I'd say that we should keep the lock ordering as well.

Otherwise let's put a comment on top of sessions: and note what order the session.lock() should be acquired in.

[AtkinsChang]

From sessions:
https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L383-L386
https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L845-L849

From NetworkContext::new:
https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L856-L858
https://github.com/paritytech/parity-ethereum/blob/00124b5a4bf8a38ad6c660b462a5e5addf13cab3/util/network-devp2p/src/host.rs#L957-L959

[dvdplm]

This is great stuff. I just tested the branch against kovan and cannot make it deadlock. Let's wait for @ngotchac's review and then merge.

[ngotchac]

Good catch !