v1.3.0: Bad Request: Block not in forkChoice - Error producing SyncCommitteeContribution

[philknows]

Describe the bug

During sync committee duties, there are times when blocks are not in forkChoice in time to produce SyncCommitteeContributions. There doesn't seem to be any correlation with time and if it gets better over time. It is very sporadic when it happens.

Jan-26 18:31:43.150[]                error: Error on SyncCommitteeContribution slot=5661156, index=1 Bad Request: Block not in forkChoice - Error producing SyncCommitteeContribution
Error: Bad Request: Block not in forkChoice - Error producing SyncCommitteeContribution
    at HttpClient.requestWithBody (file:///usr/app/packages/api/src/utils/client/httpClient.ts:258:15)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Expected behavior

We should hit SyncCommitteeContribution every time. Blocks not in forkChoice should not be happening.

[twoeths]

could be the same root cause to #5098 (comment)

when there are multiple beacon node urls, SyncComittee block root is produced and published to one node while SyncCommitteeContribution is produced from another node, and the other node hasn't seen the block yet

[philknows]

For greater context, restarting the VC fixed this problem, but unsure of what actually caused it. Could it have been using or receiving old duties? I don't have the exact steps to reproduce this, but there was a manual restart on the BN/EL server which the VC was trying to consistently connect to it during that time. More of this in the private discord thread here: https://discord.com/channels/593655374469660673/1068138020127125505/1068245669468454912

[twoeths]

to be clear, this is just a log on validator with SyncCommitteeContribution and not related to the miss of SyncCommitteeSignature. This should rarely happen (unlike attestation) because it's most likely that beacon block should come at 8s after the slot. And this happens randomly per slot, restarting the validator does not help.

[philknows]

It was a log that showed and was confirmed to be missing via beaconcha.in. So during this period of time, we missed all sync committee contributions until we restarted the VC.

[twoeths]

Just looked at the incident on Jan 25/Jan 26:

• It was caused by bad performance of beacon nodes
• Also the rescue/fallback node was configured with subscribe_all_subnets=false at that time so we lacked some subnet peers on this node ( we only configured that to true after the incident): if submitPoolSyncCommitteeSignatures failed on main bn it could be failed in fallback/rescue node too since we lack subnet peers. Below metric was from the rescue/fallback node

The log in this issue is only related to SyncCommitteeContribution (not SyncCommitteeMessage), although it's not the root cause of missed SyncCommitteeMessage it could improve the issue: vc as a validator can aggregate SyncCommitteeMessages and publish to the network itself

I'm not sure how restaring vc fixed the issue but the rescue node was switched to subscribe_all_subnets=true around that time too

[twoeths]

"Block not in forkChoice" error should be fixed with #5157