Skip to content

Commit

Permalink
improvements to the docs
Browse files Browse the repository at this point in the history
  • Loading branch information
AaronKutch committed Sep 18, 2023
1 parent 2a2df43 commit 2a3aabc
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 12 deletions.
2 changes: 1 addition & 1 deletion tests/background.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Replicated Security / ICS (originally "interchain-security" but partway through
Onomy's original main chain (https://github.com/onomyprotocol/onomy) is a provider chain. The ONEX chain and Arc chains are consumer chains in the Onomy network. For the ONEX chain, the market module is defined in https://github.com/onomyprotocol/market. The market/app/app.go doesn't have a consumer module, so it is just a standalone (I use the term "standalone" for something that can be run by itself without any ICS requirements) used for devnet testing. The multiverse repo (https://github.com/onomyprotocol/multiverse) imports the functional modules with an ICS consumer module to make the actual testnet and production binaries (see the branches for different chains).

We use a consumer-democracy model, which means that our consumer chains have normal governance and staking modules and thus can have a set of bonded validators.
The set of replicated security validators used by a consumer chain is _not_ related to the set of consumer-side bonded validators (in fact consumer chains start with zero bonded validators, as all the block-to-block signing is provided by the tendermint keys of the replicated security set). Currently, all that the bonded validators do is that they determine the governance voting powers. In the future, we want to move to provider-driven governance for all of our consumer chains except for the potential Haven chain (https://github.com/cosmos/interchain-security/issues/1271).
The set of replicated security validators used by a consumer chain is _not_ related to the set of consumer-side bonded validators (in fact consumer chains start with zero bonded validators, as all the block-to-block signing is provided by the tendermint keys of the replicated security set). Currently, all that the bonded validators do is that they determine the governance voting powers. Depending on the chain, the bond token may be a newly minted token for the chain, an IBC token, or the future provider-driven governance will be used instead (https://github.com/cosmos/interchain-security/issues/1271).

The replicated security set is determined by the provider chain. Validators on the provider chain start out with their existing tendermint keys being used in the set, unless they change it with the `assign-consensus-key` command. See `bootstrapping.md` for how a consumer chain is run for the first time. When the validator set of the provider changes, ICS automatically creates IBC packets to be relayed to consumer chains, which when received will automatically update the replicated security set. Each validator on the provider chain should also run corresponding consumer nodes with their assigned consensus key, because otherwise they can be slashed for downtime.

Expand Down
26 changes: 15 additions & 11 deletions tests/bootstrapping.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Bootstrapping a consumer chain in the `ics_with_onomyd` test works in the follow
- The test then tests various things involving using IBC transfer tokens for gas prices and such, governance changes, and any commands relevant to the specific chain

Here is the process for a real chain.
- Validators agree on a genesis for the consumer chain. We don't know the CCV state yet, so "ccvconsumer" is left in a default state. The normal validator set can be left empty. An example genesis can be obtained by getting a test genesis from the `ics_with_onomyd` test, which should be modified to obtain a testnet or mainnet ready genesis.
- Validators agree on a genesis for the consumer chain. We don't know the CCV state yet, so "ccvconsumer" is left in an empty state. The normal validator set can be left empty. An example genesis can be obtained by getting a test genesis from the `ics_with_onomyd` test, which should be modified to obtain a testnet or mainnet ready genesis.
- Someone creates a "consumer-addition" file proposal with the argument file
```
{
Expand Down Expand Up @@ -63,26 +63,30 @@ Here is the process for a real chain.

We want to avoid situations where a misconfiguration causes an error that only happens after the consumer chain is brought online, and especially misconfigurations that are built into the proposal (If it would pass, then we would have to make a consumer removal proposal). The `onomy_tests` repo has `onex_genesis` which modifies the standard ICS test to use a prespecified proposal and partial genesis to test this. It works for a single node and single dealer account for testnets, but may need to be modified for cases with more than one validator.

Before the CCV state is obtained, any validator that wants to use a different tendermint key than what they use for the provider must run
`cosmovisor run tx provider assign-consensus-key [consumer chain id] [tendermint key] [flags]`
Before the CCV state is obtained (a team member usually does this shortly after the consumer addition proposal passes), any validator that wants to use a different tendermint key than what they use for the provider must run
`[provider binary] tx provider assign-consensus-key [consumer chain id] [tendermint key] [flags]` (note: it should be an ed25519 tendermint validator key and not a secp256k1 key)

The CCV state is obtained by `cosmovisor run query provider consumer genesis [consumer chain id]` after the "spawn_time" is passed. This is inserted into the consumer genesis to complete it. This is passed along to the validators who start their nodes in preparation for "genesis_time".
The CCV state is obtained by `[provider binary] query provider consumer genesis [consumer chain id]` after the "spawn_time" is passed. This is inserted into the consumer genesis to complete it. This is passed along to the validators who start their nodes in preparation for "genesis_time".

If the consumer chain has its own staking coin that is used in th redistribution rewards, someone needs to run `cosmovisor run tx provider register-consumer-reward-denom [IBC-version-of-consumer-reward-denom]` in order for redistribution to start working later.
If the consumer chain has its own staking coin that is used in th redistribution rewards, someone needs to run `[provider binary] tx provider register-consumer-reward-denom [IBC-version-of-consumer-reward-denom]` in order for redistribution to start working later.

Before chain start, most validators will need to copy the tendermint keys from their provider node to their consumer node (excepting if they chose a different key by "assign-consensus-key" _before_ the CCV state was obtained, in which case they need that key).
Before chain start, most validators will need to copy the tendermint keys from their provider node to their consumer node (excepting if they chose a different key by "assign-consensus-key" _before_ the CCV state was obtained, in which case they need that key). Specifically, `$PROVIDER_HOME/config/priv_validator_key.json` needs to be copied to `$CONSUMER_HOME/config/priv_validator_key.json`. The provider and consumer node could be run together on the same machine as long as separate sets of ports are assigned, or else they can go on different machines.

Upon chain start, the ICS channel needs to be initiated. The ICS modules automatically create 07-tendermint clients on both chains. This is always 07-tendermint-0 for the consumer, and the corresponding one on the provider needs to be found. For Hermes, the "--a-client" must be the consumer chain, and the "--b-client" must be the provider chain.
- create the connection with `hermes create connection --a-chain [consumer id] --a-client 07-tendermint-0 --b-client [correct provider client]` creats the connection
Upon chain start, the ICS channel needs to be initiated (this should be done by a team member with access to a provider node and a consumer node). The ICS modules automatically create 07-tendermint clients on both chains. This is always 07-tendermint-0 for the consumer, and the corresponding one on the provider needs to be found. For Hermes, the "--a-client" must be the consumer chain, and the "--b-client" must be the provider chain.
- create the connection with `hermes create connection --a-chain [consumer id] --a-client 07-tendermint-0 --b-client [correct provider client]`
- create the channel with `hermes create channel --a-chain [consumer id] --a-connection connection-0 --a-port consumer --b-port provider --order ordered`
Upon provider-consumer channel creation, ICS will automatically create a transfer-transfer channel on both sides. The `chan-open-init` is done already, but the remaining 3 manual channel opening steps need to be manually done to complete the process. This works like

Upon provider-consumer channel creation, ICS will automatically create a transfer-transfer channel on both sides. The first `chan-open-init` step is done already, but the remaining 3 manual channel opening steps need to be manually done to complete the process. This works like:

- `hermes tx chan-open-try --dst-chain {provider} --src-chain {consumer} --dst-connection {provider_connection} --dst-port transfer --src-port transfer --src-channel {consumer_channel}`
- `hermes tx chan-open-ack --dst-chain {consumer} --src-chain {provider} --dst-connection {consumer_connection} --dst-port transfer --src-port transfer --dst-channel {consumer_channel} --src-channel {provider_channel}`
- `hermes tx chan-open-confirm --dst-chain {provider} --src-chain {consumer} --dst-connection {provider_connection} --dst-port transfer --src-port transfer --dst-channel {provider_channel} --src-channel {consumer_channel}`

where `consumer_connection` should always be "connection-0" and `consumer_channel` should always be "channel-1" (the provider-consumer channel was "channel-0"), and the other values need to be determined from how they were initialized earlier

A Hermes relayer should be started, and if after this there are no "tx contains unsupported message types" errors from any non-IBC transactions on the consumer chain, and if a ibc-transfer can be performed over the channel that was just opened, then bootstrap is complete.
A Hermes relayer should be started, and if after this there are no "tx contains unsupported message types" errors from any non-IBC transactions on the consumer chain, and if a ibc-transfer can be performed over `channel-1`, then bootstrap is complete.

Hermes `ft-transfer` is a very flaky transaction. In testnet practice, the timeouts had to be manually specified otherwise a "packet timeout height and packet timeout timestamp cannot both be 0" error would be issued `hermes tx ft-transfer --dst-chain [consumer-id] --src-chain [provider-id] --src-port transfer --src-channel channel-4 --amount 100000000000 --denom anom --timeout-height-offset 20 --timeout-seconds 120`.

For governance, normal validators can be created like `cosmovisor run tx staking create-validator --commission-max-change-rate 0.01 --commission-max-rate 0.10 --commission-rate 0.05 --min-self-delegation 1 --amount 100000000000ibc/5872224386C093865E42B18BDDA56BCB8CDE1E36B82B391E97697520053B0513 --from validator --pubkey '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"1vMo7NN5rvX06zVmJ61KG00/KZB0H3rsmsoslRyaBds="}' -y -b block --fees 1000000anom`. We can use IBC tokens set in the genesis because the IBC hash can be determined ahead of time.
For governance, consumer-side validators can be created like `[consumer binary] tx staking create-validator --commission-max-change-rate 0.01 --commission-max-rate 0.10 --commission-rate 0.05 --min-self-delegation 1 --amount 100000000000stake --from validator --pubkey '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"1vMo7NN5rvX06zVmJ61KG00/KZB0H3rsmsoslRyaBds="}' -y -b block --fees 1000000anom`. These validators do not sign blocks and are used just for bonding and voting in consumer-side governance. Each replicated security validator could use their provider key for the `--pubkey` argument.

0 comments on commit 2a3aabc

Please sign in to comment.