will be stuck when sync data with fullnode mode

[leafan]

System information

Geth version: 1.1.1-beta-69ce7f1e
OS & Version: Linux
Commit hash : (if develop)

Expected behaviour

run ok

Actual behaviour

will be stuck at about every 1 hour

Steps to reproduce the behaviour

Backtrace

2021/08/10 15:58:24 worker exits from a panic: runtime error: invalid memory address or nil pointer dereference
2021/08/10 15:58:24 worker exits from panic: goroutine 315 [running]:
github.com/panjf2000/ants/v2.(*goWorker).run.func1.1(0xc01c45efc0)
	github.com/panjf2000/ants/[email protected]/worker.go:58 +0x177
panic(0x14d9f40, 0x2583950)
	runtime/panic.go:969 +0x1b9
github.com/ethereum/go-ethereum/eth/filters.(*PublicFilterAPI).NewPendingTransactionFilter.func1()
	github.com/ethereum/go-ethereum/eth/filters/api.go:129 +0x205
github.com/panjf2000/ants/v2.(*goWorker).run.func1(0xc01c45efc0)
	github.com/panjf2000/ants/[email protected]/worker.go:70 +0x94
created by github.com/panjf2000/ants/v2.(*goWorker).run
	github.com/panjf2000/ants/[email protected]/worker.go:48 +0x4c

[seonggwonyoon]

I also experienced that problem yesterday.

Error Log

INFO [08-05|14:21:36.004] Starting Geth on Ethereum mainnet... 
INFO [08-05|14:21:36.007] Maximum peer count                       ETH=30 LES=0 total=30
INFO [08-05|14:21:36.007] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [08-05|14:21:36.007] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=0xf61a55]

goroutine 491099054 [running]:
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestorBinarySearch(0xc023d234a0, 0xc1b91ba870, 0x0, 0x97b34c, 0x9653b8, 0x9653b8, 0x0, 0x1afeba0)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:988 +0x6d5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).findAncestor(0xc023d234a0, 0xc1b91ba870, 0xc0ef8fd200, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:840 +0x3a5
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).syncWithPeer(0xc023d234a0, 0xc1b91ba870, 0x7ee3a7587e3b7f61, 0xbe018385b384f184, 0xdc9e4a7e2c877ae9, 0x2e142b9fcbb39a3f, 0xc26f7581a0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:497 +0x516
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).synchronise(0xc023d234a0, 0xc1ee715600, 0x40, 0x7ee3a7587e3b7f61, 0xbe018385b384f184, 0xdc9e4a7e2c877ae9, 0x2e142b9fcbb39a3f, 0xc26f7581a0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:453 +0x3b1
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).Synchronise(0xc023d234a0, 0xc1ee715600, 0x40, 0x7ee3a7587e3b7f61, 0xbe018385b384f184, 0xdc9e4a7e2c877ae9, 0x2e142b9fcbb39a3f, 0xc26f7581a0, 0xc100000000, 0x1b31800, ...)
        github.com/ethereum/go-ethereum/eth/downloader/downloader.go:348 +0x8c
github.com/ethereum/go-ethereum/eth.(*handler).doSync(0xc01b858000, 0xc1827058c0, 0x1, 0x0)
        github.com/ethereum/go-ethereum/eth/sync.go:325 +0x125
github.com/ethereum/go-ethereum/eth.(*chainSyncer).startSync.func1(0xc023d17530, 0xc1827058c0)
        github.com/ethereum/go-ethereum/eth/sync.go:301 +0x38
created by github.com/ethereum/go-ethereum/eth.(*chainSyncer).startSync
        github.com/ethereum/go-ethereum/eth/sync.go:301 +0x76

Start BSC Geth Option (Full Sync)

#!/bin/bash
geth --config /data/bsc/config.toml --datadir /data/bsc/node --cache 18000 --rpc.allow-unprotected-txs --txlookuplimit 0

[perfectcircle2020]

This happened to me as well recently.

@seonggwonyoon as @guagualvcha has pointed out this is caused by the fact that the node has been shut down incorrectly, if you notice every time you start it from the console, it displays in the logs 3 times when the node has not been shut down correctly, eventually I figured out it comes from the snapshot itself that a lot of full nodes are using.

I suppose the solution would be to make a snapshot from scratch and make it available on the GitHub repo without the unclean shutdowns, or maybe find a solution to path the chaindata on each node.

[perfectcircle2020]

The Ethereum guys have made a tool to repair the database and released it here:

ethereum/go-ethereum#21650

Shouldn't we use the same approach?

[unclezoro]

hi, @perfectcircle2020 @leafan so there are two problem here, The one @leafan raised is a known issue See :
#388
ethereum/go-ethereum#22131

As for the issue that @seonggwonyoon raised, we will consider what @perfectcircle2020 said to implement the new repair tool. See #396

I will close this issue as both of the two problems have is own issue.