snap: track revertals when peer rejects request

[holiman]

Supersedes karalabe#42

This (hopefully) fixes a flaw, where a remote peer rejects the request. Previously, we would seemingly not reschedule the request, until the next sync cycle. For some reason, which I'm not fully clear about, it could get stuck for hours (my NUC was stuck on the same things for >12 hours), requesting (and getting rejected) the same segments from the same peer(s).

This PR reverts the request if the remote side cannot deliver, so it can be rescheduled for another peer.

Example output when running it

DEBUG[12-04|11:28:11.318] Persisted set of healing data            bytes=40.43KiB [134/1839]
DEBUG[12-04|11:28:11.318] Peer rejected trienode heal request      peer=2d1f1ff2 reqid=2436914576115805419
DEBUG[12-04|11:28:11.318] Reverting trienode heal request          peer=2d1f1ff2774352477bfbf6ee10868373b074e7cd678270608eccfb2205d0e874
DEBUG[12-04|11:28:11.500] Persisted set of healing data            bytes=41.12KiB
INFO [12-04|11:28:12.503] Imported new block headers               count=1   elapsed=5.563ms    number=11385599 hash="d69950…88b422"
WARN [12-04|11:28:12.934] Pivot became stale, moving               old=11385473 new=11385537
INFO [12-04|11:28:13.011] Imported new block receipts              count=64  elapsed=76.480ms   number=11385536 hash="d72ac0…fe5a1e" age=12m42s    size=5.50MiB
DEBUG[12-04|11:28:13.011] Terminating snapshot sync cycle          root="a5adab…35513a"  
INFO [12-04|11:28:13.011] State heal in progress                   [email protected] [email protected]    pending=52937
DEBUG[12-04|11:28:13.012] Starting snapshot sync cycle             root="5d4bc8…1e0968"  
DEBUG[12-04|11:28:13.198] Persisted set of healing data            bytes=0.00B
DEBUG[12-04|11:28:13.600] Peer rejected trienode heal request      peer=8d1da4c5 reqid=4602343511521169247
DEBUG[12-04|11:28:13.600] Reverting trienode heal request          peer=8d1da4c5bc682f8f8ad141f60d2889fa4232144c7d2b29b88f81ca33e586ef9d
DEBUG[12-04|11:28:13.817] Peer rejected trienode heal request      peer=0c625358 reqid=2618974347818703192
DEBUG[12-04|11:28:13.817] Reverting trienode heal request          peer=0c6253589a8d244d57abc1c8ab298e048f24487d6572d7890e0cbb7b70039621
DEBUG[12-04|11:28:14.025] Persisted set of healing data            bytes=0.00B
DEBUG[12-04|11:28:14.935] Persisted set of healing data            bytes=0.00B
DEBUG[12-04|11:28:15.581] Persisted set of healing data            bytes=0.00B
DEBUG[12-04|11:28:16.143] Persisted set of healing data            bytes=0.00B
INFO [12-04|11:28:16.143] State heal in progress                   [email protected] [email protected]    pending=20636
INFO [12-04|11:28:16.296] Initializing fast sync bloom             items=581032069 errorrate=0.001 elapsed=11m57.850s

My machine was eventually able to sync up with these changes in

[holiman]

LGTM, it might cause some deadlock where we previously just dropped things on the floor, but having a deadlock might be better, and might allow us to fix some root problem