Lightest sync starts to fail after machine sleep and wake

[piersy]

Description

If I leave a lightest sync node running and my machine sleeps, sometimes upon wakeup the node panics and after that syncing fails. The workaround i have found is to delete the datadir and re-sync from scratch.

INFO [03-13|22:03:49.489] Imported new block headers               count=2 elapsed=13.372ms  number=11,923,200 hash=e16000..0c7bd3 age=3h59m21s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4bd4a52]

goroutine 61 [running]:
github.com/celo-org/celo-blockchain/light.(*TxPool).reorgOnNewHead(0xc0000ea9a0, {0x5961ec8, 0xc003e40ba0}, 0xc0009845a0)
	/Users/piersy/projects/celo-blockchain/light/txpool.go:241 +0x3f2
github.com/celo-org/celo-blockchain/light.(*TxPool).setNewHead(0xc0000ea9a0, 0xc0009845a0)
	/Users/piersy/projects/celo-blockchain/light/txpool.go:318 +0x1c9
github.com/celo-org/celo-blockchain/light.(*TxPool).eventLoop(0xc0000ea9a0)
	/Users/piersy/projects/celo-blockchain/light/txpool.go:299 +0x13e
created by github.com/celo-org/celo-blockchain/light.NewTxPool
	/Users/piersy/projects/celo-blockchain/light/txpool.go:112 +0x545

$ ./geth --datadir lightest --syncmode lightest
INFO [03-14|11:15:21.314] Starting Geth on Celo mainnet...
INFO [03-14|11:15:21.314] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [03-14|11:15:21.315] Maximum peer count                       ETH=175 LES=0 total=175
INFO [03-14|11:15:21.316] Set global gas cap                       cap=25,000,000
INFO [03-14|11:15:21.316] Allocated cache and file handles         database=/Users/piersy/projects/celo-blockchain/cmd/geth/lightest/celo/lightestchaindata cache=2.00GiB handles=5120
INFO [03-14|11:15:21.575] Allocated cache and file handles         database=/Users/piersy/projects/celo-blockchain/cmd/geth/lightest/celo/les.client cache=16.00MiB handles=16
INFO [03-14|11:15:21.712] Initialised chain configuration          config="{ChainID: 42220 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Churrito: 6774000, Donut: 6774000, Espresso: 11838440, Engine: istanbul}"
INFO [03-14|11:15:21.712] Open roundstate db                       func=newRoundStateDB type=roundStateDB rsdb_path=/Users/piersy/projects/celo-blockchain/cmd/geth/lightest/celo/roundstates
INFO [03-14|11:15:21.985] Loaded most recent local header          number=11,923,200 hash=e16000..0c7bd3 td=11,923,201 age=17h10m53s
INFO [03-14|11:15:21.985] Checkpoint oracle is not enabled
WARN [03-14|11:15:21.986] Old unclean shutdowns found              count=1
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-10T18:04:16+0000 age=3d17h11m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-10T18:06:45+0000 age=3d17h8m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T10:42:09+0000 age=3d33m12s
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T10:43:17+0000 age=3d32m4s
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T11:05:07+0000 age=3d10m14s
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T13:00:41+0000 age=2d22h14m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T17:57:13+0000 age=2d17h18m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T17:58:15+0000 age=2d17h17m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T18:06:13+0000 age=2d17h9m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T18:07:19+0000 age=2d17h8m
WARN [03-14|11:15:21.986] Unclean shutdown detected                booted=2022-03-11T18:23:10+0000 age=2d16h52m
INFO [03-14|11:15:21.986] Starting peer-to-peer node               instance=celo/v1.5.4-stable/darwin-amd64/go1.17.6
INFO [03-14|11:15:22.175] New local node record                    seq=56 id=7b083c09dd991471 ip=127.0.0.1 udp=30306 tcp=30306
INFO [03-14|11:15:22.176] Started P2P networking                   self=enode://dd7c850f3c0faed877e15e3b3fbb894ab5a22d23efdc27a61c5724023a73e414f6983d2a302bbd87ebdc0ea32c458663c5c27e776f375e8ea4d776fb53d64128@127.0.0.1:30306 maxdialed=58 maxinbound=117
INFO [03-14|11:15:22.177] IPC endpoint opened                      url=/Users/piersy/projects/celo-blockchain/cmd/geth/lightest/geth.ipc
WARN [03-14|11:15:22.177] Light client mode is an experimental feature
INFO [03-14|11:15:22.492] Block synchronisation started
ERROR[03-14|11:15:24.274] The header retrieved from the chain is nil block num=11,905,920
ERROR[03-14|11:15:24.274] Error "unknown block" at block 11935570
ERROR[03-14|11:15:24.274] Failed to validate the header chain at 0 due to "unknown block"
WARN [03-14|11:15:24.274] Invalid header encountered               number=11,935,570 hash=7b728c..f110db parent=3cb373..961d89 err="unknown block"
WARN [03-14|11:15:24.275] Synchronisation failed, dropping peer    peer=5c9aa36facf5242f1f65996cc8d96b2745305ce083b2f1b9dc06888b382ef7b0 err="retrieved hash chain is invalid: unknown block"
ERROR[03-14|11:15:26.223] The header retrieved from the chain is nil block num=11,905,920
ERROR[03-14|11:15:26.223] Error "unknown block" at block 11935571
ERROR[03-14|11:15:26.223] Failed to validate the header chain at 0 due to "unknown block"
WARN [03-14|11:15:26.223] Invalid header encountered               number=11,935,571 hash=6843e3..f94da4 parent=7b728c..f110db err="unknown block"
WARN [03-14|11:15:26.223] Synchronisation failed, dropping peer    peer=93f14e42a3520d920287c9e1c656251752631b98ed4f32cef36bc77dda3badad err="retrieved hash chain is invalid: unknown block"
ERROR[03-14|11:15:28.256] The header retrieved from the chain is nil block num=11,905,920
ERROR[03-14|11:15:28.256] Error "unknown block" at block 11935571
ERROR[03-14|11:15:28.256] Failed to validate the header chain at 0 due to "unknown block"
WARN [03-14|11:15:28.256] Invalid header encountered               number=11,935,571 hash=6843e3..f94da4 parent=7b728c..f110db err="unknown block"
WARN [03-14|11:15:28.256] Synchronisation failed, dropping peer    peer=218bcba2337f64c08bbf6f42fbe18add1b4de8104528a3232bd613f2edeeea6c err="retrieved hash chain is invalid: unknown block"
ERROR[03-14|11:15:30.204] The header retrieved from the chain is nil block num=11,905,920
ERROR[03-14|11:15:30.204] Error "unknown block" at block 11935572
ERROR[03-14|11:15:30.204] Failed to validate the header chain at 0 due to "unknown block"
WARN [03-14|11:15:30.204] Invalid header encountered               number=11,935,572 hash=2b4765..59cdd2 parent=6843e3..f94da4 err="unknown block"
WARN [03-14|11:15:30.204] Synchronisation failed, dropping peer    peer=08a0d8761124ed08f2fc41cac18e33da4207b268fce906602da7c3eaf5eb8eef err="retrieved hash chain is invalid: unknown block"
INFO [03-14|11:15:30.717] New local node record                    seq=57 id=7b083c09dd991471 ip=78.130.98.122 udp=30306 tcp=30306
ERROR[03-14|11:15:32.302] The header retrieved from the chain is nil block num=11,905,920
ERROR[03-14|11:15:32.302] Error "unknown block" at block 11935572
ERROR[03-14|11:15:32.302] Failed to validate the header chain at 0 due to "unknown block"
WARN [03-14|11:15:32.302] Invalid header encountered               number=11,935,572 hash=2b4765..59cdd2 parent=6843e3..f94da4 err="unknown block"
WARN [03-14|11:15:32.303] Synchronisation failed, dropping peer    peer=bf2f38432e0a50bc417bcf54ac103bc999b670c4db53b7fecae5098c2ba739b2 err="retrieved hash chain is invalid: unknown block"
ERROR[03-14|11:15:36.563] The header retrieved from the chain is nil block num=11,905,920
ERROR[03-14|11:15:36.563] Error "unknown block" at block 11935572
ERROR[03-14|11:15:36.563] Failed to validate the header chain at 0 due to "unknown block"
WARN [03-14|11:15:36.563] Invalid header encountered               number=11,935,572 hash=2b4765..59cdd2 parent=6843e3..f94da4 err="unknown block"
WARN [03-14|11:15:36.563] Synchronisation failed, dropping peer    peer=13de055c4e3cd4f5b204154357607c1cfd971f104abf7c35ac821bae0e17828c err="retrieved hash chain is invalid: unknown block"
WARN [03-14|11:15:38.173] Unable to request etherbase from peer    id=a2ad30913beb2717 conn=dyndial err="write tcp 192.168.8.105:52006->35.234.9.93:30303: write: broken pipe"
WARN [03-14|11:15:40.173] Unable to request etherbase from peer    id=a2ad30913beb2717 conn=dyndial err="shutting down"