Routine Simultaneous Validator Jailings

[chillyvee]

We have noticed recurring instances of multiple validators being jailed on sommelier.

It is typically unlikely for multiple validators to all fail on the same block due to mismanagement.

This may point to a p2p-network-issue typically associated with unbalanced vote power where most of the validators votes are not necessary for consensus and the network proceeds "without lower powered validators"

A foundation delegation to most validators to "even out" the delegated power may help until the network becomes more stable at which point delegation can be removed.

Jailing on block 2898376:
"2898376 BOND_STATUS_BONDED 1784658893 polkachu.com"
"2898376 BOND_STATUS_BONDED 18714137555 MetaRats 🐀"
"2898376 BOND_STATUS_BONDED 2249847859 Bi23 Labs"
"2898376 BOND_STATUS_BONDED 386000000 Chill Validation"
"2898376 BOND_STATUS_BONDED 695988539618 Standard Crypto"
"2898376 BOND_STATUS_UNBONDING 15682946864 Forbole"

Jailing on block 2955077:
"2955077 BOND_STATUS_BONDED 18509187555 MetaRats 🐀"
"2955077 BOND_STATUS_BONDED 2350658893 polkachu.com"
"2955077 BOND_STATUS_BONDED 386000000 Chill Validation"
"2955077 BOND_STATUS_UNBONDING 2708409660 Bi23 Labs"
"2955077 BOND_STATUS_UNBONDING 695984928611 Standard Crypto"

[chillyvee]

Occurred again
"2995816 BOND_STATUS_BONDED 18491187555 MetaRats 🐀"
"2995816 BOND_STATUS_BONDED 2499989047 polkachu.com"
"2995816 BOND_STATUS_BONDED 407000000 Chill Validation"
"2995816 BOND_STATUS_UNBONDING 1000000 mahof"
"2995816 BOND_STATUS_UNBONDING 2722383198 Bi23 Labs"
"2995816 BOND_STATUS_UNBONDING 695984928611 Standard Crypto"

[PolkachuIntern]

It is always the same list. I suspect some issue with gorc setup.

[chillyvee]

some more logs leading to jailing - does not seem to indicate anything in particular

Apr 03 18:32:34 sommelier sommelier[36525]: 6:32PM INF committed state app_hash=284CB68211399B5EF5931F67E22933CE30B55B859F8322E5ACBD4800892144B4 height=3025814 module=state num_txs=0
Apr 03 18:32:34 sommelier sommelier[36525]: 6:32PM INF indexed block height=3025814 module=txindex
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF Timed out dur=4957.931951 height=3025815 module=consensus round=0 step=1
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"403ABAC3B16217C1021DB2C11D4ED8FE2E7CF4611F387E31EADB8261810A7C32","parts":{"hash":"0EA9BB08E3E7604ABB96FF46E98B389BA632576C3161C24A53D21385E2D71783","total":1}},"height":3025815,"pol_round":-1,"round":0,"signature":"wVf03qk+OVcnt6RB4x35JBv4YZ9SsaDGiKSFk7BGRQdYXoRiu0F535xYVGnMGfMXklzqGHHKG5NLRsVT3vocDQ==","timestamp":"2022-04-03T18:32:39.024215933Z"}
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF received complete proposal block hash=403ABAC3B16217C1021DB2C11D4ED8FE2E7CF4611F387E31EADB8261810A7C32 height=3025815 module=consensus
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF finalizing commit of block hash=403ABAC3B16217C1021DB2C11D4ED8FE2E7CF4611F387E31EADB8261810A7C32 height=3025815 module=consensus num_txs=0 root=284CB68211399B5EF5931F67E22933CE30B55B859F8322E5ACBD4800892144B4
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF considering signer set tx creation blockHeight=3025815 lastUnbondingHeight=0 latestSignerSetTx.Nonce=60 module=x/gravity powerDiff=0.0020393899181018094 shouldCreate=false
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF tallying allocation votes height=3025815 module=x/allocation
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF package all winning allocation votes into contract calls module=x/allocation
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF vote period set height=3025816 module=x/allocation
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF executed block height=3025815 module=state num_invalid_txs=0 num_valid_txs=0
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF commit synced commit=436F6D6D697449447B5B3834203620383120313633203133392031333120323238203137392031302032313520313539203739203332203136312032323520343020323134203136372031353820333620313736203138342031382031363420393420313737203334203232392034332033382036342038385D3A3245324239377D
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF committed state app_hash=540651A38B83E4B30AD79F4F20A1E128D6A79E24B0B812A45EB122E52B264058 height=3025815 module=state num_txs=0
Apr 03 18:32:39 sommelier sommelier[36525]: 6:32PM INF indexed block height=3025815 module=txindex
Apr 03 18:32:40 sommelier sommelier[36525]: 6:32PM INF Saving AddrBook to file book=/home/robot/.sommelier/config/addrbook.json module=p2p size=44
Apr 03 18:32:40 sommelier sommelier[36525]: 6:32PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=11 numToDial=-1
Apr 03 18:32:44 sommelier sommelier[36525]: 6:32PM INF Timed out dur=4949.850554 height=3025816 module=consensus round=0 step=1
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"B76DDCDFC9FC2CE9B3B5B37AAF59C4736F8EC102F33D95E2FCC9078143A55FCE","parts":{"hash":"1553EA02A9CF7AADA9EC557194BC9DD5E6FEBF05ADD02CBB4090CBB5952C63FF","total":1}},"height":3025816,"pol_round":-1,"round":0,"signature":"nAscnuiG6P6ODJjKnXr0eyOYu/qvBeZON16gxFGJV9Jmpa0XIxnIn/POJI83XYLJzfoBxQRXFIDOmOmCF8cRAw==","timestamp":"2022-04-03T18:32:44.796707778Z"}
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF received complete proposal block hash=B76DDCDFC9FC2CE9B3B5B37AAF59C4736F8EC102F33D95E2FCC9078143A55FCE height=3025816 module=consensus
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF finalizing commit of block hash=B76DDCDFC9FC2CE9B3B5B37AAF59C4736F8EC102F33D95E2FCC9078143A55FCE height=3025816 module=consensus num_txs=0 root=540651A38B83E4B30AD79F4F20A1E128D6A79E24B0B812A45EB122E52B264058
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF considering signer set tx creation blockHeight=3025816 lastUnbondingHeight=0 latestSignerSetTx.Nonce=60 module=x/gravity powerDiff=0.0020393899181018094 shouldCreate=false
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF executed block height=3025816 module=state num_invalid_txs=0 num_valid_txs=0
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF commit synced commit=436F6D6D697449447B5B363020313531203136203134342031353520383820333220313434203234352032303520323535203233203234322031383220313931203136352037352039382034332032303620323333203632203931203138203135382034302031333420313236203234302031393320323030203130325D3A3245324239387D
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF committed state app_hash=3C9710909B582090F5CDFF17F2B6BFA54B622BCEE93E5B129E28867EF0C1C866 height=3025816 module=state num_txs=0
Apr 03 18:32:45 sommelier sommelier[36525]: 6:32PM INF indexed block height=3025816 module=txindex
Apr 03 18:32:50 sommelier sommelier[36525]: 6:32PM INF Timed out dur=4952.474563 height=3025817 module=consensus round=0 step=1
Apr 03 18:32:50 sommelier sommelier[36525]: 6:32PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"2C17AAE31F2491BB189A27E8B743ACA006009EA7D39E054CAC20655FA2EFBA16","parts":{"hash":"93FD8B56827126AEC43643C2CD7CFD5FC4A1B6C0681D22CE828F60AEC1EC6BDA","total":1}},"height":3025817,"pol_round":-1,"round":0,"signature":"0NSNSLd3MfB2JRrPlSDgcCXHjCXzp7Nj50TwvB+eCzoovmb1EcgE84QObgpkhflTt3oZxgifhjAqF4t3VU7RCQ==","timestamp":"2022-04-03T18:32:50.497488605Z"}
Apr 03 18:32:50 sommelier sommelier[36525]: 6:32PM INF received complete proposal block hash=2C17AAE31F2491BB189A27E8B743ACA006009EA7D39E054CAC20655FA2EFBA16 height=3025817 module=consensus
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF finalizing commit of block hash=2C17AAE31F2491BB189A27E8B743ACA006009EA7D39E054CAC20655FA2EFBA16 height=3025817 module=consensus num_txs=0 root=3C9710909B582090F5CDFF17F2B6BFA54B622BCEE93E5B129E28867EF0C1C866
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF considering signer set tx creation blockHeight=3025817 lastUnbondingHeight=0 latestSignerSetTx.Nonce=60 module=x/gravity powerDiff=0.0020393899181018094 shouldCreate=false
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF slashing at current height; not scanning unbonding delegations & redelegations height=3025817 module=x/staking
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator slashed by slash factor burned=0 module=x/staking slash_factor=0.000000000000000000 validator=sommvaloper1kkrw7naqmrtz5fyqsse3syzqnhl6wejdaxqlhh
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator jailed module=x/staking validator=sommvalcons1et6hwm57nfr9qyjklx8ygsfhlv0cpenvn87lal
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF slashing at current height; not scanning unbonding delegations & redelegations height=3025817 module=x/staking
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator slashed by slash factor burned=0 module=x/staking slash_factor=0.000000000000000000 validator=sommvaloper1mpp46nnk5jv9n4pj4qrgget8edd33me98zeggw
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator jailed module=x/staking validator=sommvalcons1j3dg9tmz272evz4qv9tr6th7hp8e3ywmmqrtuz
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF slashing at current height; not scanning unbonding delegations & redelegations height=3025817 module=x/staking
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator slashed by slash factor burned=0 module=x/staking slash_factor=0.000000000000000000 validator=sommvaloper1gp957czryfgyvxwn3tfnyy2f0t9g2p4pu0vlu3
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator jailed module=x/staking validator=sommvalcons1n5qzmdzyqtutc9eq3hvw0exxemdgynt92sm26g
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF slashing at current height; not scanning unbonding delegations & redelegations height=3025817 module=x/staking
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator slashed by slash factor burned=0 module=x/staking slash_factor=0.000000000000000000 validator=sommvaloper1l38kl5u0rmqj2rr4336jjxpmap9gl7amkf9yff
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator jailed module=x/staking validator=sommvalcons1ppn2lnthy6qhx0v8t23usml4m6j3dxqw9275dq
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF slashing at current height; not scanning unbonding delegations & redelegations height=3025817 module=x/staking
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator slashed by slash factor burned=0 module=x/staking slash_factor=0.000000000000000000 validator=sommvaloper1kz3k09tpc0zrdhklnre99kl5cu9q3cktz7pdhs
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF validator jailed module=x/staking validator=sommvalcons1umz2w4s9qgnuwu3xgsvcvu8pktjrk4rurl0nxa
Apr 03 18:32:51 sommelier sommelier[36525]: 6:32PM INF executed block height=3025817 module=state num_invalid_txs=0 num_valid_txs=0```

[chillyvee]

"3025818 BOND_STATUS_BONDED 18575323347 MetaRats 🐀"
"3025818 BOND_STATUS_BONDED 2568983948 polkachu.com"
"3025818 BOND_STATUS_BONDED 469000000 Chill Validation"
"3025818 BOND_STATUS_UNBONDING 1000000 Nala DAO"
"3025818 BOND_STATUS_UNBONDING 2943414315 Bi23 Labs"```

[chillyvee]

3025818 - 2995816 = 30002

next jailing at 3055820?

[chillyvee]

Had to upgrade geth to v1.10.17 to get it syncing. gorc is waiting now. Will give more feedback when sync is up to date.

[EricBolten]

Sorry for the delayed response. Currently on mainnet the slashing penalty is zero and staking rewards haven't been enabled yet, so while being jailed is annoying, it shouldn't negatively impact you. The only protocol-specific conditions where validators are slashed are if they fail to submit signatures for Gravity Bridge batches or valset updates. This is likely due to either running the wrong version of gorc (current is 0.3.9), having it misconfigured, or not having your delegate keys set appropriately.

[EricBolten]

We have an upcoming release of new major versions of the Sommelier binary and the Gravity Bridge, which we will be communicating out to validators in the coming days, that should hopefully be a good point to ensure everyone is running the correct software and configs.

[PolkachuIntern]

Agreed. We have been running under this assumption of "slashing penalty is zero and staking rewards haven't been enabled" and just periodically unjail the node when needed. Look forward to the new release!

[chillyvee]

Confirmed after geth+gorc are configured correctly that jailing issue goes away. (geth 1.10.17, gorc 0.3.8)

Would be good to have a way to test this prior to being jailed. :)

Less critical now since there are no penalties but we're assuming that new validators would appreciate the ability to verify their setup prior to slashing. (or even existing validators after a machine migration)