[General] Finalized status on OP mainnet

[cromaniuc]

Discussion Topic

We operate multiple OP nodes and have observed since yesterday (03-25-2024) that executing the command below yields a block from several hours earlier, this occurs even when accessing the public endpoint:

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["finalized", false],"id":1}' https://mainnet.optimism.io/

The block is different for different nodes instances, but its timestamp varies from 3 to 15 hours.

Any idea what changed? Thanks!

[cromaniuc]

It just happened with the new version as well with some of the nodes (all pointing to the same L1 beacon node). So the issue seems to occur when the L1 beacon has a forked block: https://beaconscan.com/slot/8727479. Some nodes can recover from that, but other don't. Thoughts?

=2024-03-28T03:58:10+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmLuHHn4MB92AXZytriPmrveNVQ9Sy9k444vyxhJgCY3N7 t=2024-03-28T03:58:10+0000 lvl=info msg="disconnected from peer" peer=16Uiu2HAmK5g19NEqDDhgTbEjbyujUGTwFWLJE1q2g2b8s9UAUhVv addr=/ip4/51.81.184.191/tcp/9222 t=2024-03-28T03:58:11+0000 lvl=info msg="Received signed execution payload from p2p" id=0x423a5d64bf964c74c05c28d77b2bf9e6d08e8209e15533900b0641232e4dd54a:117999757 peer=16Uiu2HAmK7D8ajruPfh463A8ZW3ZNpguvJBVXhmTQqiNAkW5VF2J t=2024-03-28T03:58:11+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x423a5d64bf964c74c05c28d77b2bf9e6d08e8209e15533900b0641232e4dd54a:117999757 t=2024-03-28T03:58:12+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer" l2_finalized=0xb7d30640c31b920e54100135ebf9164caf351b729d059038725e1f8f0923c90e:117977219 l2_safe=0xf9d5d7fe82cf10d55f330ddf484d923c7d91a74fe52a8951abadcacc5bedc483:117977353 l2_pending_safe=0xf9d5d7fe82cf10d55f330ddf484d923c7d91a74fe52a8951abadcacc5bedc483:117977353 l2_unsafe=0x423a5d64bf964c74c05c28d77b2bf9e6d08e8209e15533900b0641232e4dd54a:117999757 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1711598291 l1_derived=0xe31a552f337deb8cb4857d3acc94c98524841543a6cc6aa71f67bc1b8b222e02:19526568 t=2024-03-28T03:58:12+0000 lvl=warn msg="Derivation process temporary error" attempts=1 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0xe31a552f337deb8cb4857d3acc94c98524841543a6cc6aa71f67bc1b8b222e02:19526568: failed to fetch blob sidecars for slot 8727479 block 0xe31a552f337deb8cb4857d3acc94c98524841543a6cc6aa71f67bc1b8b222e02:19526568: failed request with status 404: {\"code\":404,\"message\":\"NOT_FOUND: beacon block at slot 8727479\",\"stacktraces\":[]}"

[adetante]

I have exactly the same problem with the "safe block".

curl -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["safe", false],"id":1}' \
  https://mainnet.optimism.io/

=> 2 days ago.

and depending on the node I send the request to, I get different results (a local node, mainnet.optimism.io, a 3rd party provider, ...)
is this "safe" "finalized" information returned by the JSON-RPC reliable?

[opfocus]

Alchemy's display results in about <15 minutes

[adetante]

but why mainnet.optimism.io is 13 days ago now?

[opfocus]

I don’t know the why.
Let's see if we can find someone from the team to answer this question

[sbvegan]

Hey @cromaniuc, the public endpoint is behind proxyd. This proxy acts as a load balancer for RPC requests. They include nodes run by OP Labs and third party RPC providers. So the finalized and safe tags can vary across the nodes you're hitting. Generally you should see normal block tags within the time ranges specified on this page.