Panic and state-root mismatch at height 3550 when re-validating calibnet

[lemmih]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

lotus version 1.23.2-dev+calibnet+git.30a9f6316

Repro Steps

1. Compile with calibnet support: make calibnet
2. Download snapshot file: https://forest-snapshots.fra1.cdn.digitaloceanspaces.com/debug/lotus_full_calibnet-17000.car.zst
3. Import and validate chain: ./lotus daemon --import-chain lotus_full_calibnet-17000.car.zst --halt-after-import
4. Observe the panic and error message: ERROR: chain validation failed: tipset chain had state mismatch at height 3550

Describe the Bug

I'm trying to re-validate all tipsets in calibnet from a full calibnet snapshot.

Logging Information

2023-06-30T12:30:43.861+0200	INFO	statemgr	stmgr/stmgr.go:465	computing state (height: 3549, ts=[bafy2bzaceagsiue5it6xiosz44d3vrfsrw5adn7onikb3jh46ytpum64zf72q bafy2bzacebmippnzmsjmqlcyrmtngvz7v63ff6s2nao2jmnsqqc7iehzwmbgq])
panic in cgo externs: runtime error: index out of range [0] with length 0
goroutine 17 [running, locked to thread]:
runtime/debug.Stack()
	/snap/go/current/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	/snap/go/current/src/runtime/debug/stack.go:16 +0x19
github.com/filecoin-project/filecoin-ffi/cgo.logPanic({0x5330fe0?, 0xc034362c90?})
	/home/lemmih/coding/lotus/extern/filecoin-ffi/cgo/errors.go:24 +0x65
github.com/filecoin-project/filecoin-ffi/cgo.cgo_extern_get_beacon_randomness.func1()
	/home/lemmih/coding/lotus/extern/filecoin-ffi/cgo/extern.go:55 +0x31
panic({0x5330fe0, 0xc034362c90})
	/snap/go/current/src/runtime/panic.go:884 +0x213
github.com/filecoin-project/lotus/chain/beacon.Schedule.BeaconForEpoch(...)
	/home/lemmih/coding/lotus/chain/beacon/beacon.go:32
github.com/filecoin-project/lotus/chain/rand.(*stateRand).extractBeaconEntryForEpoch(0xc03dc21600, {0x73b1ad0, 0xc0000540d8}, 0xc1a)
	/home/lemmih/coding/lotus/chain/rand/rand.go:201 +0x325
github.com/filecoin-project/lotus/chain/rand.(*stateRand).getBeaconRandomnessV3(0xc0000c6000?, {0x73b1ad0?, 0xc0000540d8?}, 0xc024dedcf8?, 0xc1a, {0xc02018aeb0, 0x4, 0x4})
	/home/lemmih/coding/lotus/chain/rand/rand.go:162 +0x66
github.com/filecoin-project/lotus/chain/rand.(*stateRand).GetBeaconRandomness(0xc03dc21600, {0x73b1ad0, 0xc0000540d8}, 0x4f00d20?, 0x4?, {0xc02018aeb0, 0x4, 0x4})
	/home/lemmih/coding/lotus/chain/rand/rand.go:185 +0x126
github.com/filecoin-project/filecoin-ffi/cgo.cgo_extern_get_beacon_randomness(0x19e8973?, 0xc024dede28?, 0x79d638?, 0xc0000560c0?, 0x0?, 0xc000006601?)
	/home/lemmih/coding/lotus/extern/filecoin-ffi/cgo/extern.go:66 +0x10a
2023-06-30T12:30:43.879+0200	INFO	consensus-common	consensus/compute_state.go:301	ApplyBlocks stats	{"early": 0.004656756, "earlyCronGas": 21901499, "vmMsg": 0.010062858, "msgGas": 30359324, "vmCron": 0.002469978, "cronGas": 21902291, "vmFlush": 0.000355917, "epoch": "3549", "tsk": "{bafy2bzaceagsiue5it6xiosz44d3vrfsrw5adn7onikb3jh46ytpum64zf72q,bafy2bzacebmippnzmsjmqlcyrmtngvz7v63ff6s2nao2jmnsqqc7iehzwmbgq}"}
2023-06-30T12:30:43.879+0200	INFO	statemgr	stmgr/stmgr.go:465	computing state (height: 3550, ts=[bafy2bzaceabgwqhy22s75k66pcp57k3kboq552yuoc3n7adzty2k2of22dfay])
2023-06-30T12:30:43.880+0200	WARN	chainstore	store/store.go:634	reorgWorker quit
ERROR: chain validation failed: tipset chain had state mismatch at height 3550

[lemmih]

Also reproducible with the 1.23.2 tag.

[rjan90]

Closing as this has been fixed by: #11080