Reporter: incorrect account sequence when submitting MsgInsertHeader message

[SebastianElvis]

Description

During the bootstrapping process, the vigilant reporter will forward BTC headers to BBN. When submitting BTC headers via MsgInsertHeader, the following errors occur

➜  vigilante git:(fix-invalid-stop-height) ✗ go run $VIGILANTE_PATH/cmd/main.go reporter \            
         --config $TESTNET_PATH/vigilante/vigilante.yml \
         --babylon-key $BABYLON_PATH/.testnet/node0/babylond
# github.com/keybase/go-keychain
cgo-gcc-prolog:81:11: warning: 'SecKeychainCreate' is deprecated: first deprecated in macOS 12.0 - Custom keychain management is no longer supported [-Wdeprecated-declarations]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecKeychain.h:301:10: note: 'SecKeychainCreate' has been explicitly marked deprecated here
cgo-gcc-prolog:139:11: warning: 'SecKeychainLock' is deprecated: first deprecated in macOS 12.0 - Custom keychain management is no longer supported [-Wdeprecated-declarations]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecKeychain.h:357:10: note: 'SecKeychainLock' has been explicitly marked deprecated here
cgo-gcc-prolog:159:11: warning: 'SecKeychainOpen' is deprecated: first deprecated in macOS 12.0 - Custom keychain management is no longer supported [-Wdeprecated-declarations]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecKeychain.h:287:10: note: 'SecKeychainOpen' has been explicitly marked deprecated here
cgo-gcc-prolog:183:11: warning: 'SecKeychainUnlock' is deprecated: first deprecated in macOS 12.0 - Custom keychain management is no longer supported [-Wdeprecated-declarations]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecKeychain.h:347:10: note: 'SecKeychainUnlock' has been explicitly marked deprecated here
cgo-gcc-prolog:203:11: warning: 'SecTrustedApplicationCreateFromPath' is deprecated: first deprecated in macOS 10.15 - No longer supported [-Wdeprecated-declarations]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecTrustedApplication.h:59:10: note: 'SecTrustedApplicationCreateFromPath' has been explicitly marked deprecated here
# command-line-arguments
ld: warning: could not create compact unwind for _blst_sha256_block_data_order: does not use RBP or RSP based frame
INFO[2022-09-13T14:39:34+10:00] Successfully loaded config file at /Users/runchao/Projects/Babylon/babylon/.testnet/vigilante/vigilante.yml  module=config
INFO[2022-09-13T14:39:34+10:00] Successfully created the BTC client and connected to the BTC server  module=btcclient
INFO[2022-09-13T14:39:34+10:00] Successfully subscribed to newly connected/disconnected blocks from BTC  module=btcclient
DEBU[2022-09-13T14:39:34+10:00] Babylon key directory: /Users/runchao/Projects/Babylon/babylon/.testnet/node0/babylond  module=babylonclient
DEBU[2022-09-13T14:39:34+10:00] All Babylon addresses: map[node0:bbn1653efp24j5agx9x2gssa3ucfvcfyjkztpu02f5]  module=babylonclient
INFO[2022-09-13T14:39:34+10:00] Successfully created the Babylon client       module=babylonclient
INFO[2022-09-13T14:39:34+10:00] BTCCheckpoint parameters: (k, w) = (10, 100)  module=reporter
INFO[2022-09-13T14:39:34+10:00] Successfully generated TLS certificates for the RPC server  module=rpcserver
INFO[2022-09-13T14:39:34+10:00] BTC latest block hash and height: (7d1ddc37f29aaab0d71be51e8c1b2679f00fc633e2b182a9cc0d947876eae122, 10)  module=reporter
INFO[2022-09-13T14:39:34+10:00] BBN header chain latest block hash and height: (683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6, 0)  module=reporter
DEBU[2022-09-13T14:39:34+10:00] BTC cache size: 11                            module=reporter
DEBU[2022-09-13T14:39:34+10:00] block for consistency check: height 0, hash 683e86bd5c6d110d91b94b97137ba6bfe02dbbdb8e3dff722a669b5d69d77af6  module=reporter
INFO[2022-09-13T14:39:34+10:00] BBN header chain falls behind BTC by 10 blocks.  module=reporter
DEBU[2022-09-13T14:39:34+10:00] Helping BBN header chain to catch up block 0cbbc1f4e2ff4d3d4f6d6dabb2773435dddc03bb1ff8f309f096ca8b101e8d1f...  module=reporter
DEBU[2022-09-13T14:39:34+10:00] signer: D523948555953A8314CA4421D8F309661249584B, headerHex: 00000020f67ad7695d9b662a72ff3d8edbbb2de0bfa67b13974bb9910d116d5cbd863e689efbd0b5fe1ae83112c5906c75e94bf4bc31343511d3b7a22327005c63064eb9daf50b63ffff7f2008000000  module=reporter
ERRO[2022-09-13T14:39:40+10:00] Failed to handle header 0cbbc1f4e2ff4d3d4f6d6dabb2773435dddc03bb1ff8f309f096ca8b101e8d1f from Bitcoin: rpc error: code = InvalidArgument desc = account sequence mismatch, expected 21, got 20: incorrect account sequence: invalid request  module=reporter
WARN[2022-09-13T14:39:40+10:00] Found a nil tx in block 0cbbc1f4e2ff4d3d4f6d6dabb2773435dddc03bb1ff8f309f096ca8b101e8d1f  module=reporter
DEBU[2022-09-13T14:39:40+10:00] Helping BBN header chain to catch up block 7b9e2f1c3e070fb02e16afe79dcc5a8d57d0a4f98ae9859fbd1bb39e2e45c84d...  module=reporter
DEBU[2022-09-13T14:39:40+10:00] signer: D523948555953A8314CA4421D8F309661249584B, headerHex: 000000201f8d1e108bca96f009f3f81fbb03dcdd353477b2ab6d6d4f3d4dffe2f4c1bb0c3e19b6b2c7084efe4e7a6db7f6a9c9b3791fef6f0ac3e76ed40fe7758b64abc208a81563ffff7f2004000000  module=reporter

Steps to reproduce

On branch fix-invalid-stop-height

1. Start Babylon

$BABYLON_PATH/build/babylond start --home $TESTNET_PATH/node0/babylond

2. Start Btcd

btcd --simnet --rpclisten 127.0.0.1:18554 --rpcuser rpcuser --rpcpass rpcpass \
    --rpccert $TESTNET_PATH/bitcoin/rpc.cert --rpckey $TESTNET_PATH/bitcoin/rpc.key \
    --miningaddr SQqHYFTSPh8WAyJvzbAC8hoLbF12UVsE5s

3. Start vigilant reporter

go run $VIGILANTE_PATH/cmd/main.go reporter \            
         --config $TESTNET_PATH/vigilante/vigilante.yml \
         --babylon-key $BABYLON_PATH/.testnet/node0/babylond

[SebastianElvis]

This issue seems to depend on #26. When the Babylon client has connection issue to babylond, tx will be signed in offline mode, which requires a sequence number. However we don't have any sequence number locally. If everything works fine and the connection remains persistent then this will not happen. After #38 this issue disappears as well. Will leave this open for now and close it once I confirm it is gone.

Reference: https://docs.cosmos.network/master/run-node/txs.html

[SebastianElvis]

Closed by the retry mechanism introduced in #45

[vitsalis]

Reopened after an offline discussion with @SebastianElvis since the retry mechanism covers the issue but does not resolve it.

[SebastianElvis]

Is this issue still the case? Seems that account sequence error is everywhere and the best way to resolve this is to simply retry.