feat: Move shard-assignment for chunk producers to nightly

[tayfunelmas]

Per #11190, move the shard-assignment shuffling feature for chunk producers to nightly.

Also enable this feature for mocknet together with stateless validation to exercise the state sync code path more often. Note that the guarding for the feature in nightly/prod is different from mocknet, as we use mocknet to exercise (stateless validation) features in between prod and nightly.

After this change, shard shuffling will be activated in the following cases:

1. In "nightly", it will be active by default, since we move the feature among the set of nightly features and update the protocol version accordingly (143).
2. In "mocknet", it will be active by default, as we enable it directly when chain id == "mocknet".
3. In all other networks (including "mainnet", "testnet", and "statelessnet"), it will be disabled by default and will only be enabled (later) explicitly by moving the feature from nightly to stable protocol version.

Compiling the binary with or without "statelessnet_protocol" will not have any effect on the enablement of the feature (instead need to compile with "nightly").

[codecov]

Codecov Report

Attention: Patch coverage is 96.96970% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 70.97%. Comparing base (92ee5d2) to head (b43485d).

Files	Patch %	Lines
core/primitives/src/epoch_manager.rs	96.87%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #11203   +/-   ##
=======================================
  Coverage   70.96%   70.97%           
=======================================
  Files         781      781           
  Lines      155420   155434   +14     
  Branches   155420   155434   +14     
=======================================
+ Hits       110301   110322   +21     
+ Misses      40349    40340    -9     
- Partials     4770     4772    +2     

Flag	Coverage Δ
backward-compatibility	0.24% <0.00%> (-0.01%)	⬇️
db-migration	0.24% <0.00%> (-0.01%)	⬇️
genesis-check	1.40% <0.00%> (-0.01%)	⬇️
integration-tests	37.14% <90.90%> (+<0.01%)	⬆️
linux	68.88% <45.45%> (+<0.01%)	⬆️
linux-nightly	70.44% <96.96%> (+0.02%)	⬆️
macos	52.50% <45.45%> (+0.04%)	⬆️
pytests	1.63% <0.00%> (-0.01%)	⬇️
sanity-checks	1.42% <0.00%> (-0.01%)	⬇️
unittests	65.42% <51.51%> (+<0.01%)	⬆️
upgradability	0.29% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wacban]

I think the plan should be somewhat different.

• We want to keep it enabled in forknet for as long as possible maybe with a small exception to run it once without it at the end. I would leave it as is in the StatelessNet collection of protocol features.
• We want to have it enabled in mocknet / forknet pre-release testing. In order to do that we can enable it based on the chain id but without checking if the feature is enabled.
• Taking a step back we actually not only want shard shuffling but also single shard tracking enabled in pre-release testing. That's more of a configuration change and I lack expertise to say how best to go about it.

cc some node team experts to share their opinions
@marcelo-gonzalez @posvyatokum @VanBarbascu

[wacban]

This is not how it'll work currently. Right now you check both chain id and the if the feature is enabled. This means it will not be enabled in forknet and mocknet by default because neither has nightly protocol version.

[tayfunelmas]

I was thinking about enabling it with stateless validation only, not always (even in existing stateful validation). This is why I check the stateless validation feature explicitly. But we can also do that independently because without single-shard tracking (which comes with stateless validation) it should not have any effect. what do you think?

[tayfunelmas]

I think the plan should be somewhat different.

• We want to keep it enabled in forknet for as long as possible maybe with a small exception to run it once without it at the end. I would leave it as is in the StatelessNet collection of protocol features.

The reason I moved it our of the statenessnet collection is I assumed that when launching stateless validation, we would bump the protocol version above 86 or so. Thus to avoid production getting shard shuffling at that point.

• We want to have it enabled in mocknet / forknet pre-release testing. In order to do that we can enable it based on the chain id but without checking if the feature is enabled.

The change does this currently, but it still checks stateless validation feature. I think that can be removed (as it will only take effect with single-shard tracking).

• Taking a step back we actually not only want shard shuffling but also single shard tracking enabled in pre-release testing. That's more of a configuration change and I lack expertise to say how best to go about it.

Is this also for pre-release testing of stateless validation as well as any previous release before stateless? Single shard tracking is only for stateless right? I am assuming that forknet/mocknet testing for stateless should be somewhere above 86 and the mainnet release before stateless will remain somewhere above 66.

cc some node team experts to share their opinions @marcelo-gonzalez @posvyatokum @VanBarbascu

[robin-near]

Why does this need to be only active for mocknet?

Also, we've been compiling our nodes with the flag statelessnet_protocol. Are we going to stop using that and instead use "nightly"? But nightly includes more features than statelessnet_protocol. So I'm a bit confused here about the exact strategy.

[tayfunelmas]

Why does this need to be only active for mocknet?

Also, we've been compiling our nodes with the flag statelessnet_protocol. Are we going to stop using that and instead use "nightly"? But nightly includes more features than statelessnet_protocol. So I'm a bit confused here about the exact strategy.

Updated the description to clarify, adding it here as well. You are right that compiling the binary with statelessnet_protocol will not enable shard shuffling and I am actually not sure we should do that. Instead we can add a feature to enable it explicitly. In this setup, only "nightly" and "mocknet" will have shard-shuffling enabled by default.

---

After this change, shard shuffling will be activated in the following cases:

1. In "nightly", it will be active by default, since we move the feature among the set of nightly features and update the protocol version accordingly (143).
2. In "mocknet", it will be active by default, as we enable it directly when chain id == "mocknet".
3. In all other networks (including "mainnet", "testnet", and "statelessnet"), it will be disabled by default and will only be enabled (later) explicitly by moving the feature from nightly to stable protocol version.

Compiling the binary with or without "statelessnet_protocol" will not have any effect on the enablement of the feature (instead need to compile with "nightly").

[robin-near]

In "mocknet", it will be active by default, as we enable it directly when chain id == "mocknet".

Ah, sorry I missed that this is unconditionally applied if the network is mocknet.

[tayfunelmas]

Note: I had to move the call point of config_validator_selection below the check for use_production_config because it was failing some tests and a better fix (eg. overriding the test epoch config) also had other complications (test overrides are only effective when use_production_config is true and when I set it to true it affects other configs and so on.

Instead I will let this PR go this way and in a separate PR I will try to remove use_production_config option and see what kinds of tests it breaks.