Node stays just behind tip

[icculp]

On latest v0.5.1, using my own load balanced geth nodes (<10ms latency), with OP_NODE_L1_TRUST_RPC enabled. I have run base before like this with no issues but had to move servers thanks to growing storage, stood this node up with snapshot and it's been like this since it got close to the tip, about 3 days now. Discord support just keeps reasking the same basic questions. Have also tried infura and alchemy endpoints just to test, same behavior.

[icculp]

same thing appears to be happening with sepolia using an erigon archival node. Even tried updating op-node to 1.3.2 and op-geth to v1.101304.2

[icculp]

Very interesting, once I properly forwarded op-node peering ports and restarting, they were both able to maintain par with the tip without any lag. Closing this issue, but I'd be interested in understanding better the mechanics of base's op-node peering that result in such a consistent, perceptible lag when keeping peer ports closed.

[mdehoog]

Hey @icculp, thanks for opening the issue. The graph you shared indicates that you are keeping up with the safe tip, but are not receiving (or processing) unsafe payloads that are gossiped on the P2P network.

In order for a healthy peerset, your node's P2P ports need to be available on the public internet. Port forwarding was the correct fix here 👍

[wbnns]

@icculp

Apologies that I wasn't helpful enough on Discord when I was asking about and confirming the basic info, initially, Ian! I haven't seen this issue very frequently and wasn't quite sure where-in the problem existed, so I was trying to get the additional info to share with the rest of the team (so they would hopefully have a clearer picture / full context).

Please let us know if there's ever anything else we can do to support or assist!

[icculp]

I'd like to reopen this, because just after I closed this, my goerli and sepolia nodes went from functional to again dropping off the tip and staying consistently behind.

[Chomtana]

Possibly relate to this

https://github.com/ethereum-optimism/optimism/releases/tag/op-node%2Fv1.4.0

❗ This is a mandatory release for Optimism Sepolia & Goerli and upgrading is required before 2023-12-20. This includes all features required for the Delta Network Upgade which will be activating at 1703116800 Thu Dec 21 00:00:00 UTC 2023 for Goerli & at 1703203200 Fri Dec 22 00:00:00 UTC 2023 for Sepolia.

[icculp]

Thanks for looking, good catch. I did notice that mentioned in the optimism repo, but when this base node repo went to 0.6.1 and the Dockerfile referenced a 1.4.1 I just assumed it was valid, When I look at my op-node version, it has v0.0.0-54a7dbf8-1702417143 from the op-node/v1.4.1 released. Reverting to 1.4.0 appears to have the same issue. Going to try maybe 1.4.2

[wbnns]

@icculp

Heya, happy new year! Did you try 1.4.2? Did that help?

[icculp]

hey, happy new year to you and team too! 1.4.2 did not help, I ended up downloading another snapshot for both and they have been okay since. For testnets its not as bad size wise to redownload but still I hate doing that all the time running up bandwidth especially when I don't understand what happened, I was upgraded before the network upgrade heights, and it seems related to issue #127

[wbnns]

@icculp

Heya, just wanted to follow-up with an update alongside a comment on #172 (comment)

do you have --l1.trustrpc or OP_NODE_L1_TRUST_RPC=true enabled? We found some issues with ethereum-optimism/optimism#8130 possibly disabling receipt validation that should be fixed in the next release (see ethereum-optimism/optimism#8861), so would recommend disabling this for now if enabled

Could you please try setting it to false and let us know if it resolves your issue?

[icculp]

This happened again on base mainnet, despite it being okay for a little while after snapshot, and turning off OP_NODE_L1_TRUST_RPC does not resolve the issue.

[wbnns]

@icculp

Hey, just a heads up, I'm closing this to consolidate into #127.