Beacon Node Fallback Issues

[MysticRyuujin]

Description

My validator points to 5 different beacon nodes and today the "first" beacon node in the list (172.21.4.1) went into an "unsynced" state (issue with Besu database, EL not syncing). I then started receiving constant notifications that my validator was missing attestations.

Looking at the logs all I see is logs like this repeated:

Nov 13 16:10:05.001 INFO Connected to beacon node(s)             synced: 4, available: 5, total: 5, service: notifier
Nov 13 16:10:05.001 INFO All validators active                   slot: 5127648, epoch: 160239, total_validators: 3, active_validators: 3, current_epoch_proposers: 0, service: notifier
Nov 13 16:10:10.142 INFO Connected to beacon node                endpoint: http://172.21.4.1:5052/, version: Lighthouse/v3.2.1-6d5a2b5/x86_64-linux
Nov 13 16:10:10.431 WARN Beacon node is not synced               endpoint: http://172.21.4.1:5052/, head_slot: 5127470, sync_distance: 178
Nov 13 16:10:11.417 INFO Connected to beacon node                endpoint: http://172.21.4.1:5052/, version: Lighthouse/v3.2.1-6d5a2b5/x86_64-linux
Nov 13 16:10:11.574 INFO Connected to beacon node                endpoint: http://172.21.4.1:5052/, version: Lighthouse/v3.2.1-6d5a2b5/x86_64-linux
Nov 13 16:10:11.706 WARN Beacon node is not synced               endpoint: http://172.21.4.1:5052/, head_slot: 5127470, sync_distance: 179
Nov 13 16:10:11.706 ERRO Unable to publish proposer preparation to all beacon nodes, error: Some endpoints failed, num_failed: 1 http://172.21.4.1:5052/ => Unavailable(NotSynced), service: preparation
Nov 13 16:10:11.864 WARN Beacon node is not synced               endpoint: http://172.21.4.1:5052/, head_slot: 5127470, sync_distance: 179

All of the other beacon nodes in the list are configured with:

--subscribe-all-subnet
--validator-monitor-auto

And they're all healthy and fine, EL is fine, and Lighthouse is fine on these other beacon nodes.

Just looking at the logs, and by the fact that I'm missing attestations here, would suggest to me that the fallbacks may not be working correctly?

Interestingly, after I stopped the broken Besu container I did start attesting again, but there's no change in the log output. It seems strange that Lighthouse continues to connect to the non-synced node here.

Version

All Lighthouse beacon nodes and the validator are running v3.2.1 - the Beacon Nodes are docker containers using -modern and the validator is a docker container as well (not -modern)

Present Behaviour

Missing some attestations when first listed beacon node is unhealthy.

Expected Behaviour

I would not expect to miss any attestations in this scenario with 5 beacon nodes all configured the way they are configured.

Steps to resolve

idk

[michaelsproul]

Sorry about this, the merge has revealed some issues with our fallback implementation which we are tracking via this issue: #3613. We haven't started on the implementation of any of the ideas in that issue yet, but will hopefully find some time soon (currently a lot of us are caught up in other projects: new hard forks, networking improvements, database overhauls, etc).