Synthetic block proposals

[corverroos]

Problem to be solved

We had one failed block proposal in our core team canary cluster and we couldn't find the reason. It seems related to networking, possibly the bandwidth requirements of beacon blocks.

The problem with block proposals are that they are rare. Debugging problems with rare flows in production is very hard.

Proposed solution

Introduce the concept of synthetic duties, so synthetic block proposals. See reference for synthetic testing.

What is synthetic duties?

• Synthetic duties are additional fake duties that are scheduled in a cluster, specifically production clusters.
• A deterministic function will determine if the synthetic duties needs to be scheduled when a normal duties isn't scheduled for that epoch/period.
• The cluster will perform that duty like it is a normal duty.
• The cluster will swallow the synthetic duty instead of finally broadcasting it to the beacon node.
• This will reproduce rare duties on regular bases, allowing easy debugging and problem identification.
• The BN should be oblivious to these duties, except for the additional fetch requests to get the unsigned data.
• Note that even though broadcasting unexpected block proposals isn't penalised, it may have side-effects like p2p score. We therefore swallow these submissions to reduce noise.

Why synthetic duties in production and not more production-like simnet clusters?

• Testing in production is all about leverage.
• One leverages everything in the production cluster: network, hardware, load, monitoring, alerts, human attention.
• Reproducing all of this in a simnet clusters is very hard.
• We can even use this is user clusters if they require it (to debug failed proposal in their specific environment).

Suggested solution:

• Enable this via a opt-in flag: synthetic_duties. Note this is not a feature flag since feature flags are for staged rollouts, this will always be opt-in and never be on-by-default.
• For v1, we only include block proposals, v2 can include sync committees.
• Implement this in eth2wrap via a functional option WithSyntheticDuties().
• When fetching block proposal duties for an epoch, if no duty is present in the result, pick a deterministic slot for the synthetic proposal and cache the fact that this duty is synthetic.
• When fetching unsigned blocks, if the duty is synthetic, replace the graffiti in the request with SYNTHETIC DUTY: DO NOT SUBMIT.
• When submitting blocks, if the block graffiti indicates it is synthetic, swallow it. This ensures that peers without cached synthetic duties do not submit either.

Out of Scope

This will impact production metrics. We ignore this for now, except for instrumenting the fact that synthetic is enabled.