op-geth stuck at "State not available, ignoring new payload" forever

[cheneyweb]

System information

Geth version: geth version 0.1.0-unstable-a87a03ee-20230919
CL client & version: op-node version v0.0.0-3a62bccd-1688072329
OS & Version: Linux AlmaLinux9.2

Expected behaviour

receive new payload

Actual behaviour

3|op-geth | INFO [09-21|09:46:31.483] Started P2P networking self="enode://905a92799a7bed7fa07df13ec276490600908730cd803d6696e2624f15e3910c7b51bb85a2fd9887dda34e6a4e44353e9500cbefc9607316b89af10a620e0379@127.0.0.1:10303?discport=0"
3|op-geth | INFO [09-21|09:52:09.233] Starting work on payload id=0xd2ef6c4a0498bd37
3|op-geth | WARN [09-21|09:52:09.234] Ignoring already known beacon payload number=109,730,845 hash=7be2d6..95d603 age=2d7h44m
3|op-geth | INFO [09-21|09:52:09.239] Regenerating historical state block=109,730,845 target=109,730,845 remaining=0 elapsed=561ns
3|op-geth | INFO [09-21|09:52:09.773] Historical state regenerated block=109,730,845 elapsed=534.400072ms nodes=151.07KiB preimages=0.00B
3|op-geth | ERROR[09-21|09:52:09.781] Failed to build payload err="failed to force-include tx: 0x04d886e3305305996fb167c600d1b966341a4d349e5e1b02f8fa81cfe6730bef type: 2 sender: 0x7E92cE0B94980C6DF482Cd1eC7f676E32629096e nonce: 34651, err: nonce too high: address 0x7E92cE0B94980C6DF482Cd1eC7f676E32629096e, tx: 34651 state: 0"
3|op-geth | WARN [09-21|09:52:09.781] Served engine_forkchoiceUpdatedV1 conn=127.0.0.1:33184 reqid=67524 duration=542.999471ms err="Invalid payload attributes" errdata="{Error:failed to force-include tx: 0x04d886e3305305996fb167c600d1b966341a4d349e5e1b02f8fa81cfe6730bef type: 2 sender: 0x7E92cE0B94980C6DF482Cd1eC7f676E32629096e nonce: 34651, err: nonce too high: address 0x7E92cE0B94980C6DF482Cd1eC7f676E32629096e, tx: 34651 state: 0}"
3|op-geth | INFO [09-21|09:54:56.347] Regenerating historical state block=109,730,845 target=109,730,845 remaining=0 elapsed=441ns
3|op-geth | INFO [09-21|09:54:56.362] Historical state regenerated block=109,730,845 elapsed=15.099994ms nodes=151.07KiB preimages=0.00B
3|op-geth | INFO [09-21|09:54:56.362] Starting work on payload id=0x95a0fc5fa9f7b3d2
3|op-geth | WARN [09-21|09:54:56.364] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:54:58.422] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:55:02.578] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:55:10.601] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:55:20.604] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:55:30.606] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:55:40.608] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:55:50.610] State not available, ignoring new payload
3|op-geth | WARN [09-21|09:56:00.612] State not available, ignoring new payload

Steps to reproduce the behaviour

/root/op-geth/build/bin/geth
--port=10303
--http.api=admin,debug,web3,eth,txpool,personal,miner,net
--http
--http.port=38545
--http.addr=0.0.0.0
--http.vhosts=""
--http.corsdomain=""
--authrpc.addr=localhost
--authrpc.jwtsecret=/root/op-geth/jwt.txt
--authrpc.port=38551
--authrpc.vhosts="*"
--verbosity=3
--rollup.sequencerhttp=https://mainnet-sequencer.optimism.io/
--nodiscover
--syncmode=full
--maxpeers=0
--datadir=/mnt/chain/optimism

How to fix and continue to sync, it happened after I was down for maintenance and rebooted

And if it's a matter of stopping the program, what is the safe way to stop it?

[zainirfan13]

Faced this again. Only option i had was to resync, node died again after 2 days of being synced.

They dont reply how to fix this.

[protolambda]

Just replied in the other thread #130 with a possible workaround.

Generally the issue is the upstream geth code and the DB corruption. You can prevent it from failing like that by not forcefully shutting down op-geth, but letting it gracefully write the necessary data to disk while shutting down. This can take several minutes: you may have to modify your docker/systemd/other setup to not timeout and force-kill the process.

Closing this as a duplicate, let's keep the conversation in one GH issue.