[OP Stack Deployment] L3 chain L1 (L2 in this case) RPC always reorg

[upnodedev]

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

Almost every team (except conduit) and we are facing these reorg errors and warning about L1 (L2 in this case) reorg

Moreover, sometimes, false L3 reorg happens and if the chain is down for a long time and L3 reorg happens then the database will be forever corrupted.

Happen on both standard op-geth (simple-optimism-node) and reth as Base L2 RPC

Logs

op-node-1      | t=2024-05-03T04:50:14+0000 lvl=info msg="received L1 finality signal, but missing data for immediate L2 finalization" prev_finalized_l1=0xc2554cbe91c17e3836b7b174bd7da286b905f2094f31bf8736fc8c2a54ed0293:13960462 signaled_finalized_l1=0xc2554cbe91c17e3836b7b174bd7da286b905f2094f31bf8736fc8c2a54ed0293:13960462

op-node-1      | t=2024-05-03T04:49:45+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0xa13b6a71dd87b709af233c65a29726985ed3cdfa89b3c6834d7ca03e18e977ab:13961217 new_l1_head_parent=0x24ad52315cf235844e33e5bb2538f13d2b10a8b7e18c3267873c5adcaf1711c9 new_l1_head=0xcf6fe30d1fe45ec9b2a5f23533e8f30266bc5585b225ccbfbd06760d862791a7:13961219

op-node-1      | t=2024-05-03T04:22:08+0000 lvl=eror msg="ignoring old L1 finalized block signal! Is the L1 provider corrupted?" prev_finalized_l1=0xf5cc7ea2aa33d6c0b8ea62ba1c747b14a0d243c8a4c040ce8e7e9542c0963be4:13959872 signaled_finalized_l1=0x072d1ece2de7dad1f442148671785312173386b4774094ef5a77e4a967a1c273:13959691

Additional Information

This is a Base L3

Feedback

No response

[sbvegan]

Hey @upnodedev, I recently spoke to some protocol engineers on this topic. Here's some L3 hotspots to consider:

• The base layer block time going from 12s → 2s is a big change:
  • The L(N+1) needs to be less than L(N). If the L3 blocktime == L2 blocktime your chain will fail.
  • The rollup node needs to be able to derive all the base chains' blocks to process deposit transactions.
  • Any flakiness of that cross domain connection will cause the rollup to fall behind because the block times are on parity. It would never be able to catch up. Eventually it would fall behind the max sequencer drift and only produce empty blocks
• Many of the configuration values are written in number of L1 blocks, so considerations need to be made their. You can generally multiple them by 6, but I don't have a working base config to reference.
• EIP-1559 params assume a L3 block time of 2s. This is baked into the protocol design at the moment. The result is that fees can spike rapidly.
• For plasmas: the bond size bakes in gas cost assumptions. (I don't have much detail on this)

Are you following this general guidance?

[upnodedev]

Yes, our chain has 1s block time. You can see at https://explorer.hokum.gg/

It's currently working but with these L1 reorg warnings always happen!

Many of the configuration values are written in number of L1 blocks, so considerations need to be made their. You can generally multiple them by 6, but I don't have a working base config to reference.

Can you point out which configurations are them? We have only configured finalityLookback from 4*32 + 1 to 32*32 + 1

All of our changes compared to the Celestia forked optimism can be found at https://github.com/upnode-celestia/optimism/compare/ee2c8bc2f1bab0380dd85d86cbf2eed6aa25ca66..c85d7029b586d9aa1a49b7af6069fbcf66214e9e

For plasmas: the bond size bakes in gas cost assumptions.

We currently use Celestia forked optimism monorepo which doesn't use plasma DA. However, we plan to migrate to plasma DA and we also have Plasma DA Translation Hub grant approved 😃

[upnodedev]

possible L1 re-org and ignoring old L1 finalized block signal always happening but the missing data is not logged anymore