OP Stack L3s: Catch-all Discussion + Q&A

[tynes]

This is meant to be a catch all discussion about using the OP Stack to run L3 chains

This isn't meant to be a place to debug issues, but instead for discussion on how to make running L3s more smooth using the OP Stack

[sebastianst]

To loosen the coupling between an L(n) and L(n+1), we could ingest events from multiple L(n) blocks into the same L(n+1) block. This would even allow an L3 to have a larger block time than its L2. This could maybe be done in a way to allow the inclusion of fewer L1 Deposited transactions (while still staying within the sequencer drift window or some new protocol parameter). h/t @kahuang for the impulse.

[upnodedev]

Almost every team (except conduit) and we are facing these reorg errors and warning about L1 (L2 in this case) reorg

op-node-1      | t=2024-05-03T04:50:14+0000 lvl=info msg="received L1 finality signal, but missing data for immediate L2 finalization" prev_finalized_l1=0xc2554cbe91c17e3836b7b174bd7da286b905f2094f31bf8736fc8c2a54ed0293:13960462 signaled_finalized_l1=0xc2554cbe91c17e3836b7b174bd7da286b905f2094f31bf8736fc8c2a54ed0293:13960462

op-node-1      | t=2024-05-03T04:49:45+0000 lvl=warn msg="L1 head signal indicates a possible L1 re-org" old_l1_head=0xa13b6a71dd87b709af233c65a29726985ed3cdfa89b3c6834d7ca03e18e977ab:13961217 new_l1_head_parent=0x24ad52315cf235844e33e5bb2538f13d2b10a8b7e18c3267873c5adcaf1711c9 new_l1_head=0xcf6fe30d1fe45ec9b2a5f23533e8f30266bc5585b225ccbfbd06760d862791a7:13961219

op-node-1      | t=2024-05-03T04:22:08+0000 lvl=eror msg="ignoring old L1 finalized block signal! Is the L1 provider corrupted?" prev_finalized_l1=0xf5cc7ea2aa33d6c0b8ea62ba1c747b14a0d243c8a4c040ce8e7e9542c0963be4:13959872 signaled_finalized_l1=0x072d1ece2de7dad1f442148671785312173386b4774094ef5a77e4a967a1c273:13959691

Moreover, sometimes, false L3 reorg happens and if the chain is down for a long time and L3 reorg happens then the database will be forever corrupted.

Happen on both standard op-geth (simple-optimism-node) and reth as Base L2 RPC

[sbvegan]

Relaying my same comment here:

I recently spoke to some protocol engineers on this topic. Here's some L3 hotspots to consider:

• The base layer block time going from 12s → 2s is a big change:
  • The L(N+1) needs to be less than L(N). If the L3 blocktime == L2 blocktime your chain will fail.
  • The rollup node needs to be able to derive all the base chains' blocks to process deposit transactions.
  • Any flakiness of that cross domain connection will cause the rollup to fall behind because the block times are on parity. It would never be able to catch up. Eventually it would fall behind the max sequencer drift and only produce empty blocks
• Many of the configuration values are written in number of L1 blocks, so considerations need to be made their. - You can generally multiple them by 6, but I don't have a working base config to reference.
  EIP-1559 params assume a L3 block time of 2s. This is baked into the protocol design at the moment. The result is that fees can spike rapidly.
• For plasmas: the bond size bakes in gas cost assumptions. (I don't have much detail on this)

Are you following this general guidance?

[upnodedev]

Yes, our chain has 1s block time. You can see at https://explorer.hokum.gg/

It's currently working but with these L1 reorg warnings always happen!

Many of the configuration values are written in number of L1 blocks, so considerations need to be made their. You can generally multiple them by 6, but I don't have a working base config to reference.

Can you point out which configurations are them? We have only configured finalityLookback from 4*32 + 1 to 32*32 + 1

All of our changes compared to the Celestia forked optimism can be found at https://github.com/upnode-celestia/optimism/compare/ee2c8bc2f1bab0380dd85d86cbf2eed6aa25ca66..c85d7029b586d9aa1a49b7af6069fbcf66214e9e

For plasmas: the bond size bakes in gas cost assumptions.

We currently use Celestia forked optimism monorepo which doesn't use plasma DA. However, we plan to migrate to plasma DA and we also have Plasma DA Translation Hub grant approved 😃

[arshsingh]

L1 head signal indicates a possible L1 re-org

I don't think this is cause for concern. The L1 state tracker is missing head updates. What's your l1.http-poll-interval value? The default is 12, so if you change it to 2 (assuming 2 second block time for the target chain), the warning should go away.

[upnodedev]

OP_NODE_L1_EPOCH_POLL_INTERVAL=2s
OP_NODE_L1_HTTP_POLL_INTERVAL=2s
OP_NODE_SEQUENCER_L1_CONFS=3
OP_NODE_VERIFIER_L1_CONFS=3

Our op-node already uses l1.http-poll-interval = 2s

Possible L1 re-org and ignoring old L1 finalized block signal always happening but the missing data is not logged anymore

[arshsingh]

It's only a warning about a possible re-org. The new l1_head is few blocks ahead of the old but my guess is that if you look up the blocks by hashes you'll see there is no re-org.

ignoring old L1 finalized block

Sounds like an issue with your L1 rpc. Your node is querying for finalized block, it recieves 14405479 as the finalized number, but then some calls after that start seeing 14405301. Most likely your L1 rpc provider has multiple nodes running behind a load balancer and sometimes the request goes to a node that is not up to date (for e.g. you'll see this behaviour if you watch finalized number on http://sepolia.base.org)

Try switching L1 RPC or bumping up the epoch-poll-interval (to give all your L1 nodes more time to catch up). the safe/finalized numbers are not going to be updating every 2 seconds

[yohanelly95]

Hi, have a few questions around L3s.

• Currently facing issues running a replica node for the L3 chain, it op-geth/op-node started syncing and the eth_getBlockByNumber response was incrementing as expected. Suddenly stopped, op-node did not show any errors and is showing the same logs it did initially.
  op-geth however shows the following:
  
  Now it only shows Looking for peers consistently. The archival node is running as expected and is creating new blocks every 1s.

• Bridging L1 native tokens to L3. How does this work? For eg, a token is deployed on ETH (L1) then a new token is created on say OP Mainnet (L2) using the IOptimismMintableERC20 interface. Tokens are bridged. Now this token needs to be bridged to L3 ? How would this work. I have tried this on testnets (Sepolia -> OP Sepolia -> Custom L3) but got the error that execution reverted: StandardBridge: wrong remote token for Optimism Mintable ERC20 local token.

• On the archival node, the proposer has stopped posting proposals, without any logs of errors. Its been almost 24 hours now, no idea why. https://sepolia.basescan.org/address/0x0a3917afbc42eb7326e18dbe565afe6fe1c6b603
  
  The op-batcher is posting batches successfully - https://sepolia.basescan.org/address/0x0f0a03a2a3aa937e6b824cec93eb6a3633ebcc50. What has caused the issue on op-proposer?
  I then added the --allow-non-finalized=true flag to op-proposer and began posting an L2 (L3) output root. But the settlement layer (L2) would have had finalised blocks in the ~24 hours it did not post roots for, not sure whats causing this.

[yohanelly95]

I then added the --allow-non-finalized=true flag to op-proposer and it posted an L2 (L3) output root. But the settlement layer (L2) would have had finalised blocks in the ~24 hours it did not post roots for, not sure whats causing this.

Are blocks on your L3 finalizing? It doesn't matter if the settlement layer has finalized blocks, op-proposer without --allow-non-finalized will not post output roots if your L3 blocks don't finalize. Try calling eth_getBlockByNumber on your L3 node with the finalized as the param and see if it's updating

Yes blocks are finalizing now, I kept finalityLookBack the same (4*32 + 1) does not really seem to cause any issues. Settling on Base Sepolia at the moment. I did however change some configs for the chain to account for the 2s block time of the settlement layer.

One issue I am facing is running a full node/replica node for the chain. I have p2p enabled on the sequencer node, and when I connect to it from a replica node it begins syncing for a while then suddenly stops. op-geth just says looking for peers and the eth_getBlockByNumber remains the same from then on. Have you tried running a replica node for the L3 chain ? @arshsingh

[arshsingh]

Yes, we run multiple nodes without any issues.

connect to it from a replica node

How do you connect to it? are you running with --syncmode=consensus-layer or execution-layer? Can't help without more logs

[yohanelly95]

So I just ran the replica node over the weekend and it stopped syncing at block 2011.

op-geth logs and command:

./build/bin/geth   --datadir ./datadir   --http   --http.corsdomain="*"   --http.vhosts="*"   --http.addr=0.0.0.0   --http.api=web3,debug,eth,txpool,net,engine   --ws   --ws.addr=0.0.0.0   --ws.port=8546   --ws.origins="*"   --ws.api=debug,eth,txpool,net,engine   --syncmode=snap   --gcmode=full   --nodiscover   --maxpeers=0   --networkid=11106813   --authrpc.vhosts="*"   --authrpc.addr=0.0.0.0   --authrpc.port=8551   --authrpc.jwtsecret=./jwt.txt --txlookuplimit=0 --rollup.sequencerhttp=http://${SEQUENCER_NODE_IP}:8545

op-node logs and command:

./bin/op-node   --l2=http://localhost:8551   --l2.jwt-secret=./jwt.txt  --sequencer.l1-confs=5   --verifier.l1-confs=4   --rollup.config=./rollup.json   --rpc.addr=0.0.0.0   --rpc.port=8547   --rpc.enable-admin   --p2p.sequencer.key=$GS_SEQUENCER_PRIVATE_KEY   --l1=$L1_RPC_URL   --l1.rpckind=$L1_RPC_KIND --l1.http-poll-interval=2s --l1.epoch-poll-interval=2s --l2.skip-sync-start-check=true --safedb.path=./safedb --sequencer.enabled=false  --syncmode=execution-layer --sequencer.enabled=false

Synopsis:
op-geth seemed to be updated the chain head only until block 2011. After that there were no updates. op-node was advancing origin endlessly since it was not able to catch up to the chain head. I wanted to run the node as snap/full only so not sure why op-node needs to sync from the L3 genesis which is happening in this case. @arshsingh

[yohanelly95]

tried this again from scratch, but again it synced only to the block number 2011 on the replica node. Although I do see this error on my sequencer node now:
op-node:

op-geth:

op-batcher and op-proposer show they are out of sync as well. This is after a few days of running the L3 chain with a 1s blocktime. @arshsingh any insights here would be super helpful thanks!

[arshsingh]

You're running op-node with --syncmode=execution-layer which tells it to use op-geth for syncing blocks, but you've set maxpeers=0 for op-geth.

Once you fix that, try calling admin_peers on op-geth and see if your replica node is actually connected to your primary node

not sure why op-node needs to sync from the L3 genesis

I believe op-node will switch to consensus layer sync if execution-layer sync is stalled. Not 100% sure though.

op-batcher and op-proposer show they are out of sync as well

My advice is that you should focus on getting a single node working well, before adding replica nodes. Follow your primary node's latest, safe, finalized block numbers, make sure they're progressing consistently.

[arshsingh]

Request: faster op-node restarts

Restarting op-node resets the derivation pipeline, which means it has to start from the last finalized block. This can be a problem on an L3 because not only can there be a lot of blocks to process, the parent chain will continue producing blocks every 2 seconds while op-node is processing older blocks.

The problem is magnified if we want to increase the batch submission duration, because it will increase the distance between l2_finalized and l2_unsafe.

[zhiqiangxu]

A question that comes to my mind is: what's the recommended challenge period for L3 and why?

[opfocus]

L3s for Base and the OP Stack