Key assignment (#515)

* add MsgAssignConsumerKey * add MsgAssignConsumerKey * fix package name * add keys * add keeper methods for key assignment * handle MsgAssignConsumerKey * map addresses in slash requests * prune old consumer addresses * move AssignConsumerKey logic to keeper * update consumer initial valset * add ApplyKeyAssignmentToValUpdates * fix client creation * do not check init valset on consumer * clean state on val removal * fix TestAssignConsensusKeyForConsumerChain * delete on val removal * remove reverse mapping on val removal * remove pending key assignment in EndBlock * add query endpoints add summary of indexes change ConsumerValidatorByVscID to ConsumerAddrsToPrune * Refactor AssignConsumerKey for clarity (IMO) * finish key assignment genesis code- untested * FIxed mocks compile issue - not sure if it works right though. * add test for init and export genesis * set after get in AssignConsumerKey * enable AssignConsumerKey to be called twice * remove key assignment on chain removal * apply some review comments * fix bug: two validator with same consumer key * rename key: ConsumerValidatorsByVscIDBytePrefix -> ConsumerAddrsToPruneBytePrefix * PendingKeyAssignment -> KeyAssignmentReplacements * msg.ProviderAddr is a validator addr * fix: key assignment genesis tests (#517) * Fix consumer init genesis test * fix provider genesis tests * fix key assignement handler * fix linter * fix merge conflict * fix ProviderValidatorAddress * remove unused expectation Co-authored-by: Marius Poke <[email protected]> * add key assignment CRUD operations unit tests (#516) * test val consumer key related CRUD * test val consumer addr related CRUD * test pending key assignments related CRUD * refactor after review session * refactor after review session * add prune key CRUD tests * renamings in testfiles * improve KeyAssignmentReplacement set and get * remove ApplyKeyAssignmentToInitialValset (redundant) * add invariant to docstring of AppendConsumerAddrsToPrune * fix address conversion * adding e2e tests * fix linter * add queries; setup integration tests (#519) * add queries; setup integration testse * test key assignment before chain start * fix state queries; refactor * rm extra comment * rm unused action field * bump voting times in all tests * add provider address query to tests * Adds some very basic random testing and unit tests (#522) * Adds imports * Does multi iterations: fails! * Handle errs * checkpoint debug * Pre introduce dynamic mock * Issue seems to be resolved * Removes prints in key asisgn * Removes debug, pre reintroduce all test features * Fix some magic numbers, bring back prune check * Pre rework initial assignments * Refactor and tidyup * Better docs, clarity, org Co-authored-by: Daniel <[email protected]> * Enable key assignment testing for all e2e tests (#524) * split CCVTestSuite.setupCallback in two * pre-assign keys for all vals of first consumer * fix linter * remove TestConsumerGenesis * adding ADR * move handler.go outside client/ * replace [][]byte with AddressList * remove IterateAllConsumerAddrsToPrune; not needed * apply review suggestions * fix linter * Danwt/key assignment slash test (#545) * cp * wip * note * cp * Adds slash test Co-authored-by: Daniel <[email protected]> * Fixes #503 prevents two key assignment key overlap security issues (#556) * Deletes out of date duplicate code * Adds check that validator with key does not already exist * Partially adjust assign unit test * Finishes adjusting unit * Updates stress test to never find a validator * Improves comment * Fixes handler_test * Adds validatorI iterator to expected keeper * Implements AfterValidatorCreated hook * Names * Simplifies validator query * Adds hooks test * Remove TODO * Fix random sim test Co-authored-by: Daniel <[email protected]> * Bump AssignConsumerKey comment * improve comments for iterators * Masa/key assignment integration tests amend (#548) * handle gosec false positive * add err checks for key assign; rm multiconsumer tests * guestimate block window for keyswaps in happyPeth * start multiconsumer with flag * remove node_modules * fix comment Co-authored-by: Jehan Tremback <[email protected]> Co-authored-by: Simon Noetzlin <[email protected]> Co-authored-by: MSalopek <[email protected]> Co-authored-by: Daniel T <[email protected]> Co-authored-by: Daniel <[email protected]> Co-authored-by: Jehan <[email protected]>
cosmos · Dec 7, 2022 · cb2b49e · cb2b49e
1 parent db9939f
commit cb2b49e
Show file tree

Hide file tree

Showing 65 changed files with 7,476 additions and 758 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,5 +1,5 @@
 .DS_STORE
-vue/node_modules
+*/node_modules
 vue/dist
 release/
 docs/tla/states/

diff --git a/docs/architecture/README.md b/docs/architecture/README.md
@@ -0,0 +1,30 @@
+# Architecture Decision Records (ADR)
+
+This is a location to record all high-level architecture decisions in the Interchain Security project.
+
+You can read more about the ADR concept in this [blog post](https://product.reverb.com/documenting-architecture-decisions-the-reverb-way-a3563bb24bd0#.78xhdix6t).
+
+An ADR should provide:
+
+- Context on the relevant goals and the current state
+- Proposed changes to achieve the goals
+- Summary of pros and cons
+- References
+- Changelog
+
+Note the distinction between an ADR and a spec. The ADR provides the context, intuition, reasoning, and
+justification for a change in architecture, or for the architecture of something
+new. The spec is much more compressed and streamlined summary of everything as
+it is or should be.
+
+If recorded decisions turned out to be lacking, convene a discussion, record the new decisions here, and then modify the code to match.
+
+Note the context/background should be written in the present tense.
+
+To suggest an ADR, please make use of the [ADR template](./adr-template.md) provided.
+
+## Table of Contents
+
+| ADR \# | Description | Status |
+| ------ | ----------- | ------ |
+| [001](./adr-001-key-assignment.md) | Consumer chain key assignment | Accepted, Implemented |
diff --git a/docs/architecture/adr-001-key-assignment.md b/docs/architecture/adr-001-key-assignment.md
@@ -0,0 +1,267 @@
+# ADR 001: Key Assignment
+
+## Changelog
+* 2022-12-01: Initial Draft
+
+## Status
+
+Accepted
+
+## Context
+
+KeyAssignment is the name of the feature that allows validator operators to use different consensus keys for each consumer chain validator node that they operate.
+
+## Decision
+
+It is possible to change the keys at any time by submitting a transaction (i.e., `MsgAssignConsumerKey`).
+
+### State required
+
+- `ValidatorConsumerPubKey` - Stores the validator assigned keys for every consumer chain.
+```golang
+ConsumerValidatorsBytePrefix | len(chainID) | chainID | providerConsAddress -> consumerKey
+```
+- `ValidatorByConsumerAddr` - Stores the mapping from validator addresses on consumer chains to validator addresses on the provider chain. Needed for the consumer initiated slashing sub-protocol.
+```golang
+ValidatorsByConsumerAddrBytePrefix | len(chainID) | chainID | consumerConsAddress -> providerConsAddress
+```
+- `KeyAssignmentReplacements` - Stores the key assignments that need to be replaced in the current block. Needed to apply the key assignments received in a block to the validator updates sent to the consumer chains.
+```golang
+KeyAssignmentReplacementsBytePrefix | len(chainID) | chainID | providerConsAddress -> abci.ValidatorUpdate{PubKey: oldConsumerKey, Power: currentPower},
+```
+- `ConsumerAddrsToPrune` - Stores the mapping from VSC ids to consumer validators addresses. Needed for pruning `ValidatorByConsumerAddr`. 
+```golang
+ConsumerAddrsToPruneBytePrefix | len(chainID) | chainID | vscID -> []consumerConsAddresses
+```
+
+### Protocol overview 
+
+On receiving a `MsgAssignConsumerKey(chainID, providerAddr, consumerKey)` message:
+```golang
+// get validator from staking module  
+validator, found := stakingKeeper.GetValidator(providerAddr)
+if !found {
+    return ErrNoValidatorFound
+}
+providerConsAddr := validator.GetConsAddr()
+
+// make sure consumer key is not in use
+consumerAddr := utils.TMCryptoPublicKeyToConsAddr(consumerKey)
+if _, found := GetValidatorByConsumerAddr(ChainID, consumerAddr); found {
+    return ErrInvalidConsumerConsensusPubKey
+}
+
+// check whether the consumer chain is already registered
+// i.e., a client to the consumer was already created
+if _, consumerRegistered := GetConsumerClientId(chainID); consumerRegistered {
+    // get the previous key assigned for this validator on this consumer chain
+    oldConsumerKey, found := GetValidatorConsumerPubKey(chainID, providerConsAddr)
+    if found {
+        // mark this old consumer key as prunable once the VSCMaturedPacket
+        // for the current VSC ID is received
+        oldConsumerAddr := utils.TMCryptoPublicKeyToConsAddr(oldConsumerKey)
+        vscID := GetValidatorSetUpdateId()
+        AppendConsumerAddrsToPrune(chainID, vscID, oldConsumerAddr)
+    } else {
+        // the validator had no key assigned on this consumer chain
+        oldConsumerKey := validator.TmConsPublicKey()
+    }
+
+    // check whether the validator is valid, i.e., its power is positive
+    if currentPower := stakingKeeper.GetLastValidatorPower(providerAddr); currentPower > 0 {
+        // to enable multiple calls of AssignConsumerKey in the same block by the same validator
+        // the key assignment replacement should not be overwritten
+        if _, found := GetKeyAssignmentReplacement(chainID, providerConsAddr); !found {
+            // store old key and power for modifying the valset update in EndBlock
+            oldKeyAssignment := abci.ValidatorUpdate{PubKey: oldConsumerKey, Power: currentPower}
+            SetKeyAssignmentReplacement(chainID, providerConsAddr, oldKeyAssignment)
+        }
+    }
+} else {
+    // if the consumer chain is not registered, then remove the previous reverse mapping
+    if oldConsumerKey, found := GetValidatorConsumerPubKey(chainID, providerConsAddr); found {
+        oldConsumerAddr := utils.TMCryptoPublicKeyToConsAddr(oldConsumerKey)
+        DeleteValidatorByConsumerAddr(chainID, oldConsumerAddr)
+    }
+}
+
+
+// set the mapping from this validator's provider address to the new consumer key
+SetValidatorConsumerPubKey(chainID, providerConsAddr, consumerKey)
+
+// set the reverse mapping: from this validator's new consensus address 
+// on the consumer to its consensus address on the provider
+SetValidatorByConsumerAddr(chainID, consumerAddr, providerConsAddr)
+```
+
+When a new consumer chain is registered, i.e., a client to the consumer chain is created, the provider constructs the consumer CCV module part of the genesis state (see `MakeConsumerGenesis`). 
+```golang
+func (k Keeper) MakeConsumerGenesis(chainID string) (gen consumertypes.GenesisState, nextValidatorsHash []byte, err error) {
+    // ...
+    // get initial valset from the staking module
+    var updates []abci.ValidatorUpdate{}
+    stakingKeeper.IterateLastValidatorPowers(func(providerAddr sdk.ValAddress, power int64) (stop bool) {
+        validator := stakingKeeper.GetValidator(providerAddr)
+        providerKey := validator.TmConsPublicKey()
+		updates = append(updates, abci.ValidatorUpdate{PubKey: providerKey, Power: power})
+		return false
+	})
+
+    // applies the key assignment to the initial validator
+	for i, update := range updates {
+		providerAddr := utils.TMCryptoPublicKeyToConsAddr(update.PubKey)
+		if consumerKey, found := GetValidatorConsumerPubKey(chainID, providerAddr); found {
+			updates[i].PubKey = consumerKey
+		}
+	}
+    gen.InitialValSet = updates
+
+    // get a hash of the consumer validator set from the update
+	updatesAsValSet := tendermint.PB2TM.ValidatorUpdates(updates)
+	hash := tendermint.NewValidatorSet(updatesAsValSet).Hash()
+
+	return gen, hash, nil
+}
+```
+
+On `EndBlock` while queueing `VSCPacket`s to send to registered consumer chains:
+```golang
+func QueueVSCPackets() {
+	valUpdateID := GetValidatorSetUpdateId()
+	// get the validator updates from the staking module
+	valUpdates := stakingKeeper.GetValidatorUpdates()
+
+	IterateConsumerChains(func(chainID, clientID string) (stop bool) {
+		// apply the key assignment to the validator updates
+		valUpdates := ApplyKeyAssignmentToValUpdates(chainID, valUpdates)
+        // ..
+    })
+    // ...
+}
+
+func ApplyKeyAssignmentToValUpdates(
+    chainID string, 
+    valUpdates []abci.ValidatorUpdate,
+) (newUpdates []abci.ValidatorUpdate) {
+    for _, valUpdate := range valUpdates {
+        providerAddr := utils.TMCryptoPublicKeyToConsAddr(valUpdate.PubKey)
+
+        // if a key assignment replacement is found, then
+        // remove the valupdate with the old consumer key
+        // and create two new valupdates
+        prevConsumerKey, _, found := GetKeyAssignmentReplacement(chainID, providerAddr)
+        if found {
+            // set the old consumer key's power to 0
+            newUpdates = append(newUpdates, abci.ValidatorUpdate{
+				PubKey: prevConsumerKey,
+				Power:  0,
+			})
+		    // set the new consumer key's power to the power in the update
+            newConsumerKey := GetValidatorConsumerPubKey(chainID, providerAddr)
+			newUpdates = append(newUpdates, abci.ValidatorUpdate{
+				PubKey: newConsumerKey,
+				Power:  valUpdate.Power,
+			})
+            // delete key assignment replacement
+			DeleteKeyAssignmentReplacement(chainID, providerAddr)
+        } else {
+            // there is no key assignment replacement;
+            // check if the validator's key is assigned
+            consumerKey, found := k.GetValidatorConsumerPubKey(ctx, chainID, providerAddr)
+			if found {
+                // replace the update containing the provider key 
+                // with an update containing the consumer key
+				newUpdates = append(newUpdates, abci.ValidatorUpdate{
+					PubKey: consumerKey,
+					Power:  valUpdate.Power,
+				})
+			} else {
+				// keep the same update
+				newUpdates = append(newUpdates, valUpdate)
+			}
+        }
+    }
+
+    // iterate over the remaining key assignment replacements
+    IterateKeyAssignmentReplacements(chainID, func(
+		pAddr sdk.ConsAddress,
+		prevCKey tmprotocrypto.PublicKey,
+		power int64,
+	) (stop bool) {
+       // set the old consumer key's power to 0
+		newUpdates = append(newUpdates, abci.ValidatorUpdate{
+			PubKey: prevCKey,
+			Power:  0,
+		})
+        // set the new consumer key's power to the power in key assignment replacement
+		newConsumerKey := GetValidatorConsumerPubKey(chainID, pAddr)
+		newUpdates = append(newUpdates, abci.ValidatorUpdate{
+			PubKey: newConsumerKey,
+			Power:  power,
+		})
+		return false
+	})
+
+    // remove all the key assignment replacements
+
+    return newUpdates
+}
+```
+
+On receiving a `SlashPacket` from a consumer chain with id `chainID` for a infraction of a validator `data.Validator`:
+```golang
+func HandleSlashPacket(chainID string, data ccv.SlashPacketData) (success bool, err error) {
+    // ...
+    // the slash packet validator address may be known only on the consumer chain;
+	// in this case, it must be mapped back to the consensus address on the provider chain
+    consumerAddr := sdk.ConsAddress(data.Validator.Address)
+    providerAddr, found := GetValidatorByConsumerAddr(chainID, consumerAddr)
+    if !found {
+        // the validator has the same key on the consumer as on the provider
+        providerAddr = consumer
+    }
+    // ...
+}
+```
+
+On receiving a `VSCMatured`:
+```golang
+func OnRecvVSCMaturedPacket(packet channeltypes.Packet, data ccv.VSCMaturedPacketData) exported.Acknowledgement {
+    // ...
+    // prune previous consumer validator address that are no longer needed
+    consumerAddrs := GetConsumerAddrsToPrune(chainID, data.ValsetUpdateId)
+	for _, addr := range consumerAddrs {
+		DeleteValidatorByConsumerAddr(chainID, addr)
+	}
+	DeleteConsumerAddrsToPrune(chainID, data.ValsetUpdateId)
+    // ...
+}
+```
+
+On stopping a consumer chain:
+```golang
+func (k Keeper) StopConsumerChain(ctx sdk.Context, chainID string, lockUbd, closeChan bool) (err error) {
+    // ...
+    // deletes all the state needed for key assignments on this consumer chain
+    // ...
+}
+```
+
+## Consequences
+
+### Positive
+
+- Validators can use different consensus keys on the consumer chains.
+
+### Negative
+
+- None
+
+### Neutral
+
+- The consensus state necessary to create a client to the consumer chain must use the hash returned by the `MakeConsumerGenesis` method as the `nextValsHash`.
+- The consumer chain can no longer check the initial validator set against the consensus state on `InitGenesis`.
+
+## References
+
+* [Key assignment issue](https://github.com/cosmos/interchain-security/issues/26)
diff --git a/docs/architecture/adr-template.md b/docs/architecture/adr-template.md
@@ -0,0 +1,37 @@
+# ADR {ADR-NUMBER}: {TITLE}
+
+## Changelog
+* {date}: {changelog}
+
+## Status
+
+> A decision may be "proposed" if it hasn't been agreed upon yet, or "accepted" once it is agreed upon. If a later ADR changes or reverses a decision, it may be marked as "deprecated" or "superseded" with a reference to its replacement.
+
+{Deprecated|Proposed|Accepted}
+
+## Context
+
+> This section contains all the context one needs to understand the current state, and why there is a problem. It should be as succinct as possible and introduce the high level idea behind the solution. 
+
+## Decision
+
+> This section explains all of the details of the proposed solution, including implementation details.
+It should also describe affects / corollary items that may need to be changed as a part of this.
+If the proposed change will be large, please also indicate a way to do the change to maximize ease of review.
+(e.g. the optimal split of things to do between separate PR's)
+
+## Consequences
+
+> This section describes the consequences, after applying the decision. All consequences should be summarized here, not just the "positive" ones.
+
+### Positive
+
+### Negative
+
+### Neutral
+
+## References
+
+> Are there any relevant PR comments, issues that led up to this, or articles referrenced for why we made the given design choice? If so link them here!
+
+* {reference link}