Network scalability: 300 paravalidators and 70 parachains

[sandreim]

Parachain consensus is restricted to a subset of the relay chain validators via themaxValidators configuration parameter. The plan is to remove the parameter, but until then we will gradually increase it's value as we improve the protocol and performance characteristics of the implementation. maxValidators is bumped to higher values only if key performance metrics, specifically parachain block times and finality lag do not degrade when more validators participate in parachain consensus.

After improving our tooling for profiling, gathering metrics and debugging we identified bottlenecks which we are currently addressing.

Experiments and potential improvements are tested on Versi before enabling them on value bearing chains like Kusama and Polkadot. After a battery of tests successfully ran on Versi we have already bumped the number of paravalidators from 200 to 250 on Kusama and we are currently monitoring the key metrics.

The scalability project board is available here: https://github.com/orgs/paritytech/projects/63

Plan

• Test Versi Scaling Limit polkadot#5962
• approval-distribution: batched approval/assignment sending polkadot#6401
• availability-recovery: prefer fetching small PoVs from backing group polkadot#6832
• Network Bridge error: notification queue size > 100k messages polkadot#6904
• Subsystem channel tweaks polkadot#6905
• approval distribution: trigger assignment/votes resend based on approval checking lag  polkadot#7038
• Better approval voting paramters polkadot-sdk#640

Q2 milestones:

• Increase the number of parachain validators from 200 to 250 (supporting up to 70 parachains) on Kusama
• Increase the number of parachain validators from 250 to 300 (supporting up to 70 parachains) on Kusama
• Increase the number of parachain validators from 200 to 300 (supporting up 70 parachains) on Polkadot