WIP chore: guidelines for multinode testnet experimentation

[moul]

This PR presents guidelines for establishing a multinode testnet, prior to completing the Proof of Concept (PoC) phase. Enabling the testing team to whitelist new nodes through contract invocation. Future plans involve automating and securing the process using the Contributors DAO.

Areas:

• examples: r/system/validator contract to manage the active set.
• gno.land: hook to automatically register the first node during genesis.
• tm2: configure tm2 to watch the validator set defined by the contract.

Addresses #362.
Related with #872.
Related with gnolang/hackerspace#9.

[moul]

Looking for ideas on configuring tm2 to listen to r/system/validator for active set changes in a simple and efficient manner. Need an effective approach from gno.land/cmd/gnoland to configure tm2/node, enabling tm2/consensus/reactor to monitor examples/gno.land/r/system/validator updates.

[zivkovicmilos]

Looking for ideas on configuring tm2 to listen to r/system/validator for active set changes in a simple and efficient manner. Need an effective approach from gno.land/cmd/gnoland to configure tm2/node, enabling tm2/consensus/reactor to monitor examples/gno.land/r/system/validator updates.

There are a couple of ways we can solve this, but all of them require additional infrastructure to work:

Pull changes in intervals

One of the ways you can facilitate changes in the validator set of the consensus layer is to require each node to pull in the latest validator set from the Realm (query the method on the Realm) at the same intervals. Usually these intervals are specific block numbers of the chain.

The obvious problem here is that these pulls can be seriously prolonged if you have a network that doesn't have blocks often (non-empty blocks, network activity is low). This means that validators who go into the validator set cannot know they are validating until this rendezvous block comes about (and they pull in the changes from the Realm)

Event subscribe system

Another option, if we need to have instantaneous validator set updates is to subscribe to events happening on the Realm itself. The realm, when the validator set changes, would emit an event that can be subscribed to.
An independent process would monitor these change events, and update the validator set accordingly for the next block.

This shouldn't really be constructed over RPC, because you'd introduce a delay in this event fetching - it should instead be utilized as part of the block parsing logic somehow, that these events (validator set changes) are applied to the state before moving on to the next block. This ensures that every node who is parsing the block will have an aligned validator set upon parse completion, and that incoming nodes (new validators or regular nodes) can correctly sync the chain (and validator set changes) since the events are somehow part of the historical block they are executing and committing

These are just starter ideas that we can iterate on and discuss. I'll try to think of more solutions to this problem, but I'm afraid that we need to provide the necessary tooling / feature support for any possible solution

[r3v4s]

If event does emit when only validator set has been changed, I think we'll need some backup logic for situation like some validators(node) missed certain block that includes validator set change event (which means, those node will have OLD validator set)

[piux2]

Looking for ideas on configuring tm2 to listen to r/system/validator for active set changes in a simple and efficient manner. Need an effective approach from gno.land/cmd/gnoland to configure tm2/node, enabling tm2/consensus/reactor to monitor examples/gno.land/r/system/validator updates.

Here are some thoughts.

If we are trying to achieve POA running multiple nodes, here is what we can do for now

1. bring in the staking module and evidence module
2. Add the validators in the genesis,
3. Assign over 2/3 of voting power to one gno test node so that a single party can handle the node deployment and update. We don’t need to coordinate updates and deployment among different parties.

If we plan to use a gno contract to update the validator set, verify the proposal voting power, be part of the consensus to resolve conflicts and secure the chain. We may need to re-implement most of the stuff in the x/staking and x/evidence modules in the contracts. ValidatorUpdate happens at abci.Endblocker in staking module

However, serious security and trust concerns exist about using smart contracts to track validators and voting powers and use the information from contracts to reach node consensus. It is because contracts can be upgraded, and contract states can be modified by the admin and contract deployer, who may not be known as the admin.

In the future, to dynamically update the white list of POA in a contract, we should consider a chain upgrade with the gov module in place or only allow the gov module to upgrade the system contracts and a message call to update the system contract states has to be approved by gov module.

[moul]

Thank you @zivkovicmilos, @r3v4s, @piux2, for your suggestions.

Event Subscribe System:
Implementing an event subscribe system with the emit functionality (#575) in the GnoSDK (#873) would allow chains to react to contract events. We can create a GnoSDK helper to subscribe to these events from the chain app, making it simple and efficient for tm2/consensus/reactor to monitor updates in examples/gno.land/r/system/validator.

Autoregistering Validators:
Instead of adding validators in the genesis, I propose autoregistering the first node and using the contract to add subsequent validators. This eliminates the need for coordination among different parties and simplifies the process.

Contributors DAO and Governance:
Let's focus on integrating with the Contributors DAO (#872) for managing changes and replacing the hardcoded admin address. The governance module will handle admin-related decisions, making it a more generic and standardized approach.

Contract Upgrades and Configurability:
To address contract upgrades, we can make the contract path configurable from another contract, allowing flexibility. For example, r/system/configuration.UpdateValidatorContract(“r/system/validator.v2”) (similar to #761 (comment)). Governance decisions can determine the contract path, and chain upgrades can be optional.

Let's work towards making chain upgrades rare and ensuring PoA management becomes straightforward through Gno and the Gno SDK.

[anarcher]

For now, let's start with a simple implementation :-) .
As @piux2 said, I believe it would work by simply doing a ValidatorUpdate on abci.EndBlocker (we might need a r/staking realm too).
I'm not sure if we need an event for it.

[moul]

What would you put in a r/staking realm?

[anarcher]

Now that I think about it, IMHO we only need to manage the Validator's VotingPower in PoA. (I don't think we need the whole Stacking module in POA :-))

[moul]

Yep, staking is not suitable for PoA.

Voting power can be separate from validation.

Evaluation DAO (#407) utilizes ^Worx tokens for dynamic voting power. Then, WorxDAO elects validator nodes.

Validators can equally share validation power, each having a 5% share in a group of 20 validators, following a round-robin approach.

Certain staking components, like slashing, may still be relevant. Governance could handle slashing through trials. Pausing nodes in case of dangerous behaviors before the trial could be beneficial.

[zivkovicmilos]

@anarcher @moul

Wanted to leave this here, since it's relevant:
https://eips.ethereum.org/EIPS/eip-225#voting-on-signers

Clique handles PoA validator set changes by block votes (as a proposer, you vote for 1 address to be included / excluded from the set), and if there are 2/3+ votes on a tally for a specific action, the vote validator set is changed (protocol level).
It all depends on if you want to have this validator set management be purely on the Realm, or have it be protocol-level.

Autoregistering Validators:
Instead of adding validators in the genesis, I propose autoregistering the first node and using the contract to add subsequent validators. This eliminates the need for coordination among different parties and simplifies the process.

In this Clique PoA voter set approach that I've mentioned, you'd need to have an initial validator set in genesis (or have a single validator at the beginning), and manually add other validators - you need to ensure that the majority of the current validator set votes in new validators, which can be a pain as you've mentioned.

If we want to go full-Realm, the Realm would essentially need 3 methods:

• Get the validator set
• Update the validator set (add / remove validator)
• Get voting power

Voting power, of course for PoA would be equal (1) for all validators, but for PoS it would vary depending on the stake. There is a complication here for historical validator set management, meaning the Realm should only hold the current validator set, and not keep historical data. The historical data itself (the set changes) should be contained within something each node must parse (probably the block itself). We would also need to predeploy the Validator Set Realm on chain init (genesis)

[jaekwon]

Clique handles PoA validator set changes by block votes (as a proposer, you vote for 1 address to be included / excluded from the set), and if there are 2/3+ votes on a tally for a specific action, the vote validator set is changed (protocol level).
It all depends on if you want to have this validator set management be purely on the realm, or have it be protocol-level.

Let's do it purely on the realm, I suggest /r/system/validators or /r/validators, and if (and only if) the realm changed the validator set, then it immediately gets applied via EndBlocker.

The first step in InitChain should be to set up /r/validators with the data from the chain. Then, I think the first public facing smart contract to implement is one that allows something like /r/demo/valset.ValSet (which replicates functionality from tm2/pkg/bft/types) to vote for a slice of update proposals []abci.ValidatorUpdate and if 2/3 does agree, then in one transaction apply those changes. (and many such updates get aggregated until finally and EndBlock, the final changes are compressed into one set of abci values back to tm2.

This /r/validators could also expose functions allowing other libraries to make and cancel proposals, but in the end the validators only change when the /r/validators voting procedure has completed for a proposal, and never without, for safety. Another way to look at it is that we are exposing as much of abci as reasonable to the smart contract layer such that it becomes integrated into a full solution eventually -- but also with some safety features built in, as mentioned in the first sentence. Other applications with for example proof-of-stake will not want to use this basic manual PoA system but use something else entirely, and that's OK.

[moul]

See also, #889