Possible race in integration testing

[gibson042]

Describe the bug

#9688 encountered a surprising issue that caused the first attempt of getting-started (link-cli/yarn) to fail, followed by broken/cleanup-missing recovery logic: https://github.com/Agoric/agoric-sdk/actions/runs/9913890868/job/27391851751?pr=9688#check-step-7

ENV_SETUP starting
ENV_SETUP finished
start voteLatestProposalAndWait()
waiting for block...
1
block produced
done
Latest proposal: 14
waiting for block...
1
block produced
done
Error: rpc error: code = Unknown desc = rpc error: code = Unknown desc = account sequence mismatch, expected 63, got 62: incorrect account sequence [agoric-labs/[email protected]/x/auth/ante/sigverify.go:269] With gas wanted: '18446744073709551615' and gas used: '38588' : unknown request
Usage:
  agd tx gov deposit [proposal-id] [deposit] [flags]

Flags:
  -a, --account-number uint      The account number of the signing account (offline mode only)
      --aux                      Generate aux signer data instead of sending a tx
  -b, --broadcast-mode string    Transaction broadcasting mode (sync|async|block) (default "sync")
      --dry-run                  ignore the --gas flag and perform a simulation of a transaction, but don't broadcast it (when enabled, the local Keybase is not accessible)
      --fee-granter string       Fee granter grants fees for the transaction
      --fee-payer string         Fee payer pays fees for the transaction instead of deducting from the signer
      --fees string              Fees to pay along with transaction; eg: 10uatom
      --from string              Name or address of private key with which to sign
      --gas string               gas limit to set per-transaction; set to "auto" to calculate sufficient gas automatically. Note: "auto" option doesn't always report accurate results. Set a valid coin value to adjust the result. Can be used instead of "fees". (default 200000)
      --gas-adjustment float     adjustment factor to be multiplied against the estimate returned by the tx simulation; if the gas limit is set manually this flag is ignored  (default 1)
      --gas-prices string        Gas prices in decimal format to determine the transaction fee (e.g. 0.1uatom)
      --generate-only            Build an unsigned transaction and write it to STDOUT (when enabled, the local Keybase only accessed when providing a key name)
  -h, --help                     help for deposit
      --keyring-backend string   Select keyring's backend (os|file|kwallet|pass|test|memory) (default "os")
      --keyring-dir string       The client Keyring directory; if omitted, the default 'home' directory will be used
      --ledger                   Use a connected Ledger device
      --node string              <host>:<port> to tendermint rpc interface for this chain (default "tcp://localhost:26657")
      --note string              Note to add a description to the transaction (previously --memo)
      --offline                  Offline mode (does not allow any online functionality)
  -o, --output string            Output format (text|json) (default "json")
  -s, --sequence uint            The sequence number of the signing account (offline mode only)
      --sign-mode string         Choose sign mode (direct|amino-json|direct-aux), this is an advanced feature
      --timeout-height uint      Set a block timeout height to prevent the tx from being committed past a certain height
      --tip string               Tip is the amount that is going to be transferred to the fee payer on the target chain. This flag is only valid when used with --aux, and is ignored if the target chain didn't enable the TipDecorator
  -y, --yes                      Skip tx broadcasting prompt confirmation

Global Flags:
      --chain-id string     The network chain ID
      --home string         directory for config and data (default "/root/.agoric")
      --log_format string   The logging format (json|plain) (default "plain")
      --log_level string    The logging level (trace|debug|info|warn|error|fatal|panic) (default "info")
      --trace               print out full stack trace on errors

make: *** [Makefile:101: start-contract] Error 1
make: Leaving directory '/workspace/contract'
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
yarn start:ui
$ cd ui && yarn dev
$ vite

  VITE v4.5.0  ready in 213 ms

  ➜  Local:   http://localhost:5173/
  ➜  Network: use --host to expose
  ✘ [fail]: workflow yarn start:contract works
  ─

  workflow

  yarn start:contract works

  Difference (- actual, + expected):

  - 2
  + 0

  › gettingStartedWorkflowTest (packages/agoric-cli/tools/getting-started.js:124:7)

  ─

  1 test failed
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Warning: Attempt 1 failed. Reason: Child_process exited with error code 1
yarn run v1.22.22
$ yarn run create-agoric-cli /home/runner/bin/agoric
$ node ./scripts/create-agoric-cli.cjs /home/runner/bin/agoric
Script directory /home/runner/bin does not appear in $PATH
(You may want to `export PATH=$PATH:/home/runner/bin' to add it to your PATH environment variable)
ensuring /home/runner/bin exists
creating /home/runner/bin/agoric
Error: /home/runner/bin/agoric must not already exist; you should use a fresh path.
    at Object.<anonymous> (/home/runner/work/agoric-sdk/agoric-sdk/scripts/create-agoric-cli.cjs:45:11)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
    at node:internal/main/run_main_module:23:47
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Warning: Attempt 2 failed. Reason: Child_process exited with error code 1
yarn run v1.22.22
$ yarn run create-agoric-cli /home/runner/bin/agoric
$ node ./scripts/create-agoric-cli.cjs /home/runner/bin/agoric
Script directory /home/runner/bin does not appear in $PATH
(You may want to `export PATH=$PATH:/home/runner/bin' to add it to your PATH environment variable)
ensuring /home/runner/bin exists
creating /home/runner/bin/agoric
Error: /home/runner/bin/agoric must not already exist; you should use a fresh path.
    at Object.<anonymous> (/home/runner/work/agoric-sdk/agoric-sdk/scripts/create-agoric-cli.cjs:45:11)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:86:12)
    at node:internal/main/run_main_module:23:47
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Final attempt failed. Child_process exited with error code 1

[mhofman]

I believe this is a dup of #9325