Creating a channel fails trying to simulate the tx and ultimately fails out

[njmurarka]

Summary of Bug

When I try to create a channel with Hermes, it fails with a lengthy output. I was attempting this between Osmosis testnet and my own Testnet for Bluzelle. Interestingly, an earlier version of Curium chain works fine with Hermes. This deployment of Curium chain has some parametric differences and is a fork of an existing LaunchPad network, but I don't see how the fork itself is the difference/cause. Obviously, something different explains this new failure.

The key aspect of the failure appears to be the call to "CreateClient", where an attempt to estimate the gas, fails.

Here is the output (with much of the golang output purged for practicality -- please ask if the rest is needed):

Hint: Consider using the default invocation

hermes create channel --port-a <PORT-ID> --port-b <PORT-ID> <CHAIN-A-ID> <CONNECTION-A-ID>

to re-use a pre-existing connection. yes
2022-07-06T11:08:58.034961Z  INFO ThreadId(01) Creating new clients, new connection, and a new channel with order ORDER_UNORDERED
2022-07-06T11:08:58.047592Z  INFO ThreadId(11) wait_for_block_commits: waiting for commit of tx hashes(s) 3A557833D22CF1EBE1DA762A7D421F3C55E52E501BC388850B6BD31E31B508B0 id=osmo-test-4
2022-07-06T11:09:00.761983Z  INFO ThreadId(01) 🍭 [curium -> osmo-test-4:07-tendermint-2065]  => CreateClient(
    CreateClient(
        Attributes {
            height: Height {
                revision: 4,
                height: 5447855,
            },
            client_id: ClientId(
                "07-tendermint-2065",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 8961,
            },
        },
    ),
)

2022-07-06T11:09:00.774013Z ERROR ThreadId(21) send_tx_with_account_sequence_retry{id=curium}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "recovered: UnmarshalJSON cannot decode empty bytes
stack:
goroutine 3429905 [running]:
runtime/debug.Stack()
        /snap/go/9759/src/runtime/debug/stack.go:24 +0x65
github.com/cosmos/cosmos-sdk/baseapp.newDefaultRecoveryMiddleware.func1({0x2fc0220, 0xc0102410a0})
        /home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:71 +0x27
github.com/cosmos/cosmos-sdk/baseapp.newRecoveryMiddleware.func1({0x2fc0220, 0xc0102410a0})
        /home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:39 +0x30
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2fc0220, 0xc0102410a0}, 0xc013397440)
        /home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:28 +0x37
github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2fc0220, 0xc0102410a0}, 0x3c5ac40)
        /home/ubuntu/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/recovery.go:33 +0x5e

Version

Hermes version:

hermes 0.15.0+4e83aae

Steps to Reproduce

Here is the command that was run and can be run again to reproduce:

hermes create channel osmo-test-4 -c curium --port-a transfer --port-b transfer --new-client-connection

You will need an adequately funded curium account. Here is the json you can use to add the key to Hermes:

":\"A8D7EWAqMuKRc/xADg6XHGVb4OcGIXn/+jGqQnQbnxeO\"}","mnemonic":"give ahead beyond together relax ahead clip execute stick valid leaf hybrid staff noise keen month snake purity clump jump climb present shrimp cement"}

Note: You will likely need to add the following parameter when adding the key above to Hermes:

--hd-path "m/44'/483'/0'/0/0"

You will need an adequately funded osmosis account. For simplicity, I provide here the json you can use for adding a key for osmosis:

{"name":"osmosis-ibc","type":"local","address":"osmo1hhv8mqfn7kv9gte7gw7whsueh4566w93k2elu9","pubkey":"{\"@type\":\"/cosmos.crypto.secp256k1.PubKey\",\"key\":\"Akz3ggVQWyVm7UGctCPMJ9gx7yG2OxGAGpn6sEev1G+9\"}","mnemonic":"shy scorpion portion believe outer frost pink crunch tennis satisfy dash oval trust duty wash mad result region area buzz safe frequent trend kitchen"}

Here is a relevant snippet of Hermes' config.toml I use (modifed with the rpc and grpc endpoints working for the Internet):

[[chains]]
id = 'osmo-test-4'
rpc_addr = 'http://preview.bluzelle.com:26677'
grpc_addr = 'http://preview.bluzelle.com:9110'
websocket_addr = 'ws://preview.bluzelle.com:26677/websocket'
rpc_timeout = '10s'
account_prefix = 'osmo'
key_name = 'relayer'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 1402071
gas_price = { price = 0.0001, denom = 'uosmo' }
gas_adjustment = 0.1
max_msg_num = 30
max_tx_size = 2097152
clock_drift = '5s'
max_block_time = '10s'
trusting_period = '12days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }
memo_prefix = 'Interbloc for Osmosis'
[chains.packet_filter]
policy = 'allow'
list = []

[[chains]]
id = 'curium'
rpc_addr = 'http://preview.bluzelle.com:26657'
grpc_addr = 'http://preview.bluzelle.com:9090'
websocket_addr = 'ws://preview.bluzelle.com:26657/websocket'
rpc_timeout = '10s'
account_prefix = 'bluzelle'
key_name = 'relayer'
store_prefix = 'ibc'
default_gas = 100000
max_gas = 800000
gas_price = { price = 0.0007, denom = 'ubnt' }
gas_adjustment = 0.2
max_msg_num = 30
max_tx_size = 200000
clock_drift = '5s'
max_block_time = '10s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
address_type = { derivation = 'cosmos' }
memo_prefix = 'Interbloc for Bluzelle'
[chains.packet_filter]
policy = 'allow'
list = [['transfer', 'channel-0']]

The other sections of config.toml is the default if I recall.

Acceptance Criteria

I can run the command to create a channel, successfully.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ancazamfir]

Here is the output (with much of the golang output purged for practicality -- please ask if the rest is needed):

Yes please paste the full output, thanks!

[ancazamfir]

Could you also provide the cosmos-sdk and ibc-go versions used by (the new) Curium chain?

[njmurarka]

Here is the full output, in an attachment.

full_output.txt

[njmurarka]

Cosmos SDK version:

v0.44.3

[njmurarka]

IBC Go version:

v1.2.2

[ancazamfir]

Thanks. It looks like the allowed client types is not set in the store and SDK panics when trying to get it during client creation. Can you check that genesis.json for the curium chain includes something like this:

    "ibc": {
      "client_genesis": {
        "clients": [],
        "clients_consensus": [],
        "clients_metadata": [],
        "params": {
          "allowed_clients": [
            "06-solomachine",
            "07-tendermint"
          ]
        },
...

[njmurarka]

Thanks so much!

I believe your suggestion totally resolved the original issues.

I am running into some more that I would appreciate help on. Given you helped me with the first one, perhaps this one will be resolved too, @ancazamfir ?

Please see the attached. Basically, it did not die right away and tried to create the IBC connection. In fact, it seems there is a valid TX registered to start the connection but ultimately, Hermes stops.

Seems related to #1971 from the standpoint that the error I see in my setup is mentioned there in 1971.

It would seem the message "ConnectionOpenTry" is unknown...?

Thanks.
error.txt

[ancazamfir]

Could you check again genesis.json for the staking parameters? They should look something like:

    "staking": {
      "params": {
        "unbonding_time": "1814400s",
        "max_validators": 100,
        "max_entries": 7,
        "historical_entries": 10000,
        "bond_denom": "stake"
      },
...

The relevant part being historical_entries which should be non-zero, ideally at least in the hundreds.

[njmurarka]

Thanks. Look like we are making good progress @ancazamfir.

It went further but then gave more errors. Interesting thing is this time, Hermes did not end, but it did give a bunch of errors.

Attached:
error-2.txt

You can see here on Big Dipper, which IBC-related messages went through:

http://preview.bluzelle.com/transactions

Please advise. Greatly appreciate it.

[ancazamfir]

Looks like transfer module did not bind to the transfer port so has not claimed the capability. Maybe this is missing this from the genesis.json:

    "transfer": {
      "port_id": "transfer",
      "denom_traces": [],
      "params": {
        "send_enabled": true,
        "receive_enabled": true
      }
    },
...

[njmurarka]

Ok @ancazamfir looks like you REALLY know your stuff. Thanks.

Just made those changed and it appears the handshake worked and there is a path between the two chains now.

Success: Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "osmo-test-4",
                version: 4,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-2102",
        ),
        connection_id: ConnectionId(
            "connection-1635",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-355",
            ),
        ),
        version: None,
    },
    b_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "curium",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-0",
        ),
        connection_id: ConnectionId(
            "connection-0",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
        version: None,
    },
    connection_delay: 0ns,
}

More soon :). Thanks a bunch!

[ancazamfir]

great! are there any more issues you are seeing? if not we can maybe close this one?

[njmurarka]

All good for now. Will ping if something new comes up... probably as a different issue. Hopefully, that won't be necessary.

BTW, I am writing up some docs... sort of a "quick start" for Hermes + connecting two chains. I know the docs for Hermes have this, but mine is perhaps more deeper, for lack of a better word. Would this be of value to the community? Where can I put it?

Thanks again!

[romac]

All good for now. Will ping if something new comes up... probably as a different issue. Hopefully, that won't be necessary.

Glad to hear that, and yeah please don't hesitate to open issues if you run into any other kind of trouble.

BTW, I am writing up some docs... sort of a "quick start" for Hermes + connecting two chains. I know the docs for Hermes have this, but mine is perhaps more deeper, for lack of a better word. Would this be of value to the community? Where can I put it?

Wow that's great, we would love to check it out once it's ready and could perhaps add it to the Hermes guide if you'd agree. Feel free to open a PR against the guide or just link us to a GitHub Gist and we'll take a look :)

[njmurarka]

Sorry for the delay. Heads down with related work but this will come out and will announce when ready :).