Refine the BF logic for updating the PeerFetchState with async block adds

[nfrisby]

While developing a new approach to the ThreadNet tests, I noticed something unexpected in the BlockFetch threads' Tracer output.

As of PR #1709, the BF client adds blocks to CDB asynchronously instead of synchronously. Since the addFetchedBlock call in the BF client now returns immediately, the BF client now updates its PeerFetchStatus (to indicate the fetched block is no-longer in flight) before the CDB updates its current chain (to include the new fetched block) (or is it ChainDB.getIsFetched and not the current chain?).

As a result I'm seeing the node's BF fetch logic thread immediately add a duplicate fetch request for the same block. Once the BF client updates its peer status, the relevant inputs to the fetch logic thread now look the same as they did before it originally made the fetch request: the (fetched) block is not in flight and it is not in the CDB's current chain -- it's about to be, but the fetch logic doesn't know that.

This is causing the BF client in the ThreadNet to fetch every (?) block twice. This Issue is resolved when the the BF client no longer causes the node's fetch logic thread to repeat its requests.

@duncan shared some thoughts here https://input-output-rnd.slack.com/archives/GCW2S19CM/p1585076440303700

[mrBliss]

@nfrisby Changing the following line: https://github.com/input-output-hk/ouroboros-network/blob/e71aa72adc7e876cda4ff19b5cc0900167fa9804/ouroboros-consensus/src/Ouroboros/Consensus/NodeKernel.hs#L338 to

  addFetchedBlock _pt = void . ChainDB.addBlock chainDB

should fix it. This will block until the block has been written to disk and processed by chain selection (until the block is from the future).

Note that doing things async gave us a 20-30% boost in bulk sync speed. This change will undo that. Maybe a better compromise is to add the ability to wait until a block has been written to the VolatileDB so that getIsFetched will say yes for the block, but not until the block has been processed by chain selection. I'd hope this fixes the double fetches and doesn't give up too much performance in the process.