op-node: continue on retrieval error

[tuxcanfly]

Overview

There is a persistent issue with the rollup node not progressing on safe blocks. Upon inspection, the rollup node tries to fetch frame from data but fails due to:

WARN [06-06|11:04:20.344] unable to retrieve data from da          origin=b83465..d34c02:9130159 err="Get \"http://localhost:26659/namespaced_data/00000000000000000000000000000000000000f92140012837778721be/height/36708\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
WARN [06-06|11:04:20.346] Failed to parse frames                   origin=b83465..d34c02:9130159 err="parsing frame 0: reading channel_id: unexpected EOF"

In case of an error during da retrieval, we can continue so that the derivation can proceed without trying to parse (and failing) invalid frames.

Checklist

• New and updated code has appropriate documentation
• New and updated code has new and/or updated testing
• Required CI checks are passing
• Visual proof for any user facing features like CLI or documentation updates
• Linked issues closed with keywords

[jcstein]

LGTM

[tuxcanfly]

This wasn't quite the right fix, because the rollup node doesn't know to retry the temporary error, so it skips the frame.

WARN [06-06|17:58:04.375] requesting celestia namespaced data      origin=0e2085..27f227:9131334 namespace=00000000000000000000000000000000000000f92140012837778721be height=37852
WARN [06-06|17:58:04.377] unable to retrieve data from da          origin=0e2085..27f227:9131334 err="Get \"http://localhost:26659/namespaced_data/00000000000000000000000000000000000000f92140012837778721be/height/37852\": dial tcp [::1]:26659: connect: connection refused"
INFO [06-06|17:58:05.569] Advancing bq origin                      origin=e09194..029bca:9131335 originBehind=false
...
WARN [06-06|17:58:46.077] requesting celestia namespaced data      origin=a578de..f6f3e2:9131376 namespace=00000000000000000000000000000000000000f92140012837778721be height=37891
WARN [06-06|17:58:46.256] retrieved data                           origin=a578de..f6f3e2:9131376 data=00...
INFO [06-06|17:58:46.266] created new channel                      origin=a578de..f6f3e2:9131376 channel=af7f0b..1284a6 length=3405  frame_number=0 is_last=true
INFO [06-06|17:58:46.266] Reading channel                          channel=af7f0b..1284a6 frames=1

Note that after failing to read celestia block 37852, it skips and reads 37891 next.

We need to return and handle the sentinel NewTemporaryError introduced in #43 in NextData so that NextFrame doesn't try to parse it in ParseFrames. Looks like the sentinel is not used in this context as there was no error possible here previously (in Ethereum DA layer).

[jcstein]

merged this prematurely, so think we can revert the change in another PR 👍

[tuxcanfly]

Cool. FWIW here's a way to reliably repro this.

Temporarily bring the celestia light node down until there's at least one

WARN [06-06|17:58:04.377] unable to retrieve data from da          origin=0e2085..27f227:9131334 err="Get \"http://localhost:26659/namespaced_data/00000000000000000000000000000000000000f92140012837778721be/height/37852\": dial tcp [::1]:26659: connect: connection refused"

error in the op-node logs. Now when celestia light node is bought up again, the derivation should resume from the last failed da block.

This simulates a temporary node error, for e.g.

 context deadline exceeded (Client.Timeout exceeded while awaiting headers)

which was causing the original issue with lagging safe blocks.

[tuxcanfly]

Followed up in #144