blockmanager: aggressively pipeline getheaders requests

[Roasbeef]

Currently when fetching block headers, we wait until we finish processing the current batch before we fetch the very next batch. This is inefficient, as it doesn't allow us to keep pipeline and keep the network saturated while we're fetching and verifying headers. Instead, once we receive a batch of headers, we should fetch the very next immediately. It may even be worth breaking fetching and processing out into distinct goroutines. If this is done, then it'll be possibly to nearly fetch the entire set of headers, while we're busy with I/O and header verification.

While we're at it, we should continue always fetch headers beyond the last checkpoint. Much of this sync code was borrowed from btcd. In our case however, we don't care about fully validation, only PoW so we can more aggressively pipeline our block fetching, and stop using getblocks all together.

[Roasbeef]

This may actually be an opportunity to significantly simplify the header fetching code, as what we borrowed from btcd is actually a super set of what we need as a light client.

[Chinwendu20]

Hello, this issue was created 2018. I just want to confirm that the links in the issue redirect to where you want it to

[Roasbeef]

So the links above don't use permalinks, so def possible that the sections I intended to link to have shifted over time.

[Chinwendu20]

Oh okay I am going through this issue and trying to learn more about it for my SOB proposal. It would make things easier to understand for me if I can get clarity on the particular sections that are being referred to here by the links.

[Roasbeef]

Re headers:

• We write the headers here at the very end of the processing loop:

  neutrino/blockmanager.go

  Line 2671 in 9badd7f

  err := b.cfg.BlockHeaders.WriteHeaders(headerWriteBatch...)

• We'll then fetch the new set of headers here at the bottom:

  neutrino/blockmanager.go

  Lines 2686 to 2695 in 9badd7f

  	locator := blockchain.BlockLocator([]*chainhash.Hash{finalHash})
  	nextHash := zeroHash
  	if b.nextCheckpoint != nil {
  	nextHash = *b.nextCheckpoint.Hash
  	}
  	err := hmsg.peer.PushGetHeadersMsg(locator, &nextHash)
  	if err != nil {
  	log.Warnf("Failed to send getheaders message to "+
  	"peer %s: %s", hmsg.peer.Addr(), err)
  	return

[Roasbeef]

This is a related PR as well: #263

[Chinwendu20]

Thanks a lot.

[Chinwendu20]

How about redirect for, "We should always fetch headers beyond the last checkpoint"?
If you could also direct me to where the codebase is using "getblocks" I would be grateful. I cannot seem to find that.

[Roasbeef]

So we actually don't use getblocks anymore. But if we expand the set of checkpoints in btcd itself, then this'll let us continue to use that checkpointed sync for longer.