OP proposer is stalled for 25k blocks in devnet version v0.1.0-OP_v1.0.6-CN_v0.9.5 on devnet

[jcstein]

The "safe" block is 25000 blocks behind the "latest" block.

Here are the full logs

[TheAppleTucker]

Think this has to do with the speed of the derivation pipeline in op-node. Think it's quite slow to sync call_data transactions so the safe blocks tend to lag behind

[jcstein]

Thank you @TheAppleTucker - do you think even 25k blocks behind is expected based on the config?

[tuxcanfly]

there seem to have been a lot of panics in the logs above:

grep panic 115k-devnet-logs.log

This should be fixed by celestiaorg/celestia-node#2244

So we should test this again against v0.10.2

[jcstein]

Fortunately, there were no instances of panic in this run on v0.10.1 last week #120 (comment)

[jcstein]

unfortunately, the proposer is still stalling, but we don't really know which error thrown is the issue

[tuxcanfly]

I debugged the logs mentioned in #120

#120 (comment)

Long shot but are we sure it was running with proposer timeout fix?

[jcstein]

yes it was running with proposer timeout fix #120 (comment)

[tuxcanfly]

It seems that node requests stall whenever head request to trusted peer failed error occurs:

celestiaorg/celestia-node#2241 (comment)

This seems to be happening in node and then spilling over to proposer, will need to investigate if bumping node timeout is a workaround for this.

[tuxcanfly]

OK, I think I found the issue, it seems that proposer is stalling waiting for safe l2 head to be ahead of the checkpoint head, but since proposer was not able to update safe head due to op-node error, it can't progress and goes into a livelock.

If this is indeed the bug, then either increasing the op-node timeout or setting OP_PROPOSER_ALLOW_NON_FINALIZED: "false" should fix the issue.