Node crashes while trying to report equivocation

[fkbenjamin]

One of our validators is in a crash loop after trying to report an equivocation. Here are some logs of when the issue started:

Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 ✨ Imported #10500464 (0x56a2…b0b1)
Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 ♻️  Reorg on #10500464,0x56a2…b0b1 to #10500464,0x95d1…65c7, common ancestor #10500463,0x0904…6b2f
Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 ✨ Imported #10500464 (0x95d1…65c7)
Dec 13 10:25:42  polkadot[1091]: 2021-12-13 10:25:42 rejecting unsigned report equivocation transaction because it is not local/in-block.
Dec 13 10:25:44  polkadot[1091]: 2021-12-13 10:25:44 💤 Idle (37 peers), best: #10500464 (0x95d1…65c7), finalized #10500460 (0x8ffe…098c), ⬇ 4.8MiB/s ⬆ 2.8MiB/s
Dec 13 10:25:48  polkadot[1091]: 2021-12-13 10:25:48 ✨ Imported #10500465 (0x01e1…297f)
Dec 13 10:25:49  polkadot[1091]: 2021-12-13 10:25:49 💤 Idle (37 peers), best: #10500465 (0x01e1…297f), finalized #10500461 (0xa7ef…afaa), ⬇ 3.4MiB/s ⬆ 1.4MiB/s
Dec 13 10:25:50  polkadot[1091]: 2021-12-13 10:25:50 ✨ Imported #10500464 (0xcaa5…5185)
Dec 13 10:25:54  polkadot[1091]: 2021-12-13 10:25:54 💤 Idle (37 peers), best: #10500465 (0x01e1…297f), finalized #10500461 (0xa7ef…afaa), ⬇ 2.0MiB/s ⬆ 2.1MiB/s
Dec 13 10:25:54  polkadot[1091]: 2021-12-13 10:25:54 ✨ Imported #10500466 (0x0a6c…9367)
Dec 13 10:25:57  polkadot[1091]: 2021-12-13 10:25:57 GRANDPA voter error: Client(BadJustification("invalid precommits for target commit"))
Dec 13 10:25:57  polkadot[1091]: 2021-12-13 10:25:57 Essential task `grandpa-voter` failed. Shutting down service.
Dec 13 10:25:58  polkadot[1091]: Error:
Dec 13 10:25:58  polkadot[1091]:    0: Other: Essential task failed.
Dec 13 10:25:58  polkadot[1091]: Backtrace omitted.
Dec 13 10:25:58  polkadot[1091]: Run with RUST_BACKTRACE=1 environment variable to display it.
Dec 13 10:25:58  polkadot[1091]: Run with RUST_BACKTRACE=full to include source snippets.

Since then the node is in a crash loop trying to submit the report:

Dec 13 10:29:58  systemd[1]: Started Kusama Validator.
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ----------------------------
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 This chain is not in any way
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59       endorsed by the
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59      KUSAMA FOUNDATION
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ----------------------------
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 Parity Polkadot
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ✌️  version 0.9.13-7d8f00b90c-x86_64-linux-gnu
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ❤️  by Parity Technologies <[email protected]>, 2017-2021
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 📋 Chain specification: Kusama
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 🏷 Node name: 
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 👤 Role: AUTHORITY
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 💾 Database: RocksDb at /home/<user>/.local/share/polkadot/chains/ksmcc3/db/full
Dec 13 10:29:59  polkadot[1162451]: 2021-12-13 10:29:59 ⛓  Native runtime: kusama-9130 (parity-kusama-0.tx8.au2)
Dec 13 10:30:11  polkadot[1162451]: 2021-12-13 10:30:11 🏷 Local node identity is: <id>
Dec 13 10:30:11  polkadot[1162451]: 2021-12-13 10:30:11 🔍 Discovered new external address for our node: /ip4/<IP>/tcp/55032/p2p/<id>>
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 📦 Highest known block at #10500466
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 〽️ Prometheus exporter started at 0.0.0.0:9615
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 Listening for new connections on 127.0.0.1:9944.
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 👶 Starting BABE Authorship worker
Dec 13 10:30:38  polkadot[1162451]: 2021-12-13 10:30:38 Detected precommit equivocation in the finality worker: Equivocation { round_number: 3321, identity: Public(5f3fa3e3eaf17017ee2175>
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 (offchain call) Error submitting a transaction to the pool: Pool(InvalidTransaction(InvalidTransaction::Stale))
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 Error submitting equivocation report: ()
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 GRANDPA voter error: Client(BadJustification("invalid precommits for target commit"))
Dec 13 10:30:40  polkadot[1162451]: 2021-12-13 10:30:40 Essential task `grandpa-voter` failed. Shutting down service.
Dec 13 10:30:40  polkadot[1162451]: Error:
Dec 13 10:30:40  polkadot[1162451]:    0: Other: Essential task failed.
Dec 13 10:30:40  polkadot[1162451]: Backtrace omitted.
Dec 13 10:30:40  polkadot[1162451]: Run with RUST_BACKTRACE=1 environment variable to display it.
Dec 13 10:30:40  polkadot[1162451]: Run with RUST_BACKTRACE=full to include source snippets.
Dec 13 10:30:40  systemd[1]: kusama.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 10:30:40  systemd[1]: kusama.service: Failed with result 'exit-code'.

After asking for some advise in the Kusama Validator launch, we tried to restart the node without the validator flag to see if that help it sync further blocks but it still tries to submit the report and crashes.

[joslee7410]

I think we met with the same issue #893 ? XD