Parallelise integration tests using ResourceT #2191

Anviking · 2020-09-29T14:14:47Z

Issue Number

ADP-466 — both ADP-529 and ADP-528

Overview

Use runResourceT in each test scenario to ensure only the wallet created in it are deleted.
Re-write all listWallets expectations to check that the expected wallets are a subset of the actual wallets, instead of checking for equality.
Enable -j 8 in buildkite

Comments

Buildkite now takes ≈35 min instead of 50 min
On my machine stack test cardano-wallet:integration --ta '-j 8' takes 380s instead of ~1800s.
Haven't enabled parallel tests in hydra yet because of errors
I now pushed anviking/ADP-466/parallel (parallel everything) directly here.

Anviking · 2020-09-29T14:34:24Z

bors try

iohk-bors · 2020-09-29T14:55:33Z

try

Build failed:

buildkite/cardano-wallet

Anviking · 2020-09-29T16:35:11Z

bors try

iohk-bors · 2020-09-29T16:49:44Z

try

Build failed:

buildkite/cardano-wallet

rvl · 2020-09-30T00:57:45Z

This is an excellent idea @Anviking - a general way to clean up resources allocated during test execution.

Just wondering, any reason for choosing the resourcet library ahead of other possible options such as managed?
Is there a general strategy for handling errors in the resource destroy function? If a test case throws an exception, and the destructor also throws an exception, which one is reported to the user?
Looking at this type which is repeated many times.
```
Context t -> ResourceT m a
```
Since you have introduced a monad transformer, what about stacking a reader on top for the Context?

Anviking · 2020-09-30T08:18:07Z

Just wondering, any reason for choosing the resourcet library ahead of other possible options such as managed?

I went for resourcet because I think I heard Edsko mention it in Albi, and I didn't know about managed. Looking now, it seems to me like resourcet is more powerful (allows unprotect, some stuff for sharing resources across threads), which might come in handy one day.

Anviking · 2020-10-01T18:21:20Z

bors try

iohk-bors · 2020-10-01T19:22:03Z

try

Build failed:

ci/hydra-build:required

 To rerun use: --match "/CLI Specifications/SHELLEY_CLI_WALLETS/WALLETS_DELETE_01, WALLETS_LIST_02 - Can delete wallet/"

  src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:330:13:
  10) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_05 - Wallet from pubKey is available, The same account and mnemonic wallet can live side-by-side
       expected: 2
        but got: 39

buildkite:

src/Test/Integration/Scenario/CLI/Shelley/Wallets.hs:317:13:
--
  | 1) CLI Specifications, SHELLEY_CLI_WALLETS, WALLETS_CREATE_05 - Can't create wallet with wrong size of mnemonic, 9
  | expected: "[]\n"
  | but got: "[\n    {\n        \"passphrase\": {\n            \"last_updated_at\": \"2020-10-01T19:25:32.709710148Z\"\n        },\n

Anviking · 2020-10-01T21:08:10Z

bors try

iohk-bors · 2020-10-01T22:00:24Z

try

Build failed:

ci/hydra-build:required

  To rerun use: --match "/CLI Specifications/SHELLEY_CLI_WALLETS/WALLETS_DELETE_01, WALLETS_LIST_02 - Can delete wallet/"

  src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:330:13:
  10) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_05 - Wallet from pubKey is available, The same account and mnemonic wallet can live side-by-side
       expected: 2
        but got: 37

Anviking · 2020-10-02T07:05:40Z

bors try

Anviking · 2020-10-02T07:09:58Z

bors try-

Anviking · 2020-10-08T08:55:06Z

Closing for now.

To investigate failures like: ``` src/Test/Integration/Scenario/API/Shelley/Transactions.hs:1037:18: 2) API Specifications, SHELLEY_TRANSACTIONS, TRANS_EXTERNAL_02 - Multiple Outputs Transaction - Shelley witnesses Waited longer than 90s (more than 2 epochs) to resolve action: "wFaucet and wSrc balances are as expected". expected: Quantity {getQuantity = 999989842531} but got: Quantity {getQuantity = 999989842430} ```

such that it works in parallel. Removing the `walletId` check shouldn't really matter.

To fix failures like: ``` src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:185:13: 1) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_03 - Cannot do operations requiring private key, Cannot send tx "Please enter your passphrase: **************\nI can't process this payment because there's not enough UTxO available in the wallet. The total UTxO sums up to 0 Lovelace, but I need 1000000 Lovelace (excluding fee amount) in order to proceed with the payment.\n" does not contain "I couldn't find a root private key for the given wallet: 53a635ddc8caaee509f4bb991372bdbeeb4ebfde. However, this operation requires that I do have such a key. ```

Previously a `describe` with 3 `it`, each using the same wallet. Race-conditions with the ResourceT cleanup... and simply parallel... made this not work. Let's simply convert them to a single test. We can use `counterexample` to still show /which/ golden failed.

To investigate failures like: ``` src/Test/Integration/Scenario/API/Shelley/Transactions.hs:1037:18: 2) API Specifications, SHELLEY_TRANSACTIONS, TRANS_EXTERNAL_02 - Multiple Outputs Transaction - Shelley witnesses Waited longer than 90s (more than 2 epochs) to resolve action: "wFaucet and wSrc balances are as expected". expected: Quantity {getQuantity = 999989842531} but got: Quantity {getQuantity = 999989842430} ```

By mistake I first added the fix to TRANS_EXTERNAL_01 instead of TRANS_EXTERNAL_02. But I'm not keeping it on both to be safe.

Co-Authored-By: KtorZ <[email protected]>

Thinking the following errors are just from an overloaded hydra... src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 1) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, mnemonic --size 15 uncaught exception: IOException of type InvalidArgument fd:23: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/mnemonic --size 15/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 2) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, wallet list --port uncaught exception: IOException of type InvalidArgument fd:27: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/wallet list --port/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 3) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, wallet create uncaught exception: IOException of type InvalidArgument fd:28: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/wallet create/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 4) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, wallet uncaught exception: IOException of type InvalidArgument fd:23: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/wallet/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 5) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, create uncaught exception: IOException of type InvalidArgument fd:27: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/create/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 6) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, wallet crate name uncaught exception: IOException of type InvalidArgument fd:28: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/wallet crate name/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:102:34: 7) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help on bad arg or param, network information --port uncaught exception: IOException of type InvalidArgument fd:24: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help on bad arg or param/network information --port/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:109:27: 8) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help with, -h uncaught exception: IOException of type InvalidArgument fd:24: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help with/-h/" src/Test/Integration/Scenario/CLI/Miscellaneous.hs:109:27: 9) No backend required, Miscellaneous CLI tests, COMMON_CLI_MISC, CLI_HELP - cardano-wallet shows help with, --help uncaught exception: IOException of type InvalidArgument fd:24: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/No backend required/Miscellaneous CLI tests/COMMON_CLI_MISC/CLI_HELP - cardano-wallet shows help with/--help/" src/Test/Integration/Scenario/CLI/Shelley/Addresses.hs:206:51: 10) CLI Specifications, SHELLEY_CLI_ADDRESSES, ADDRESS_LIST_04 - False wallet ids, 41 chars hex uncaught exception: IOException of type InvalidArgument fd:514: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/SHELLEY_CLI_ADDRESSES/ADDRESS_LIST_04 - False wallet ids/41 chars hex/" src/Test/Integration/Scenario/CLI/Shelley/Wallets.hs:306:40: 11) CLI Specifications, SHELLEY_CLI_WALLETS, WALLETS_CREATE_05 - Can't create wallet with wrong size of mnemonic, 9 uncaught exception: IOException of type InvalidArgument fd:594: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/SHELLEY_CLI_WALLETS/WALLETS_CREATE_05 - Can't create wallet with wrong size of mnemonic/9/" src/Test/Integration/Scenario/CLI/Shelley/Wallets.hs:418:50: 12) CLI Specifications, SHELLEY_CLI_WALLETS, WALLETS_CREATE_08 - --address-pool-gap values, arbitraty string -> fail uncaught exception: IOException of type InvalidArgument fd:633: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/SHELLEY_CLI_WALLETS/WALLETS_CREATE_08 - --address-pool-gap values/arbitraty string -> fail/" src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:212:9: 13) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_04 - Can manage HW wallet the same way as others, Can update name uncaught exception: IOException of type InvalidArgument fd:675: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/SHELLEY_CLI_HW_WALLETS/HW_WALLETS_04 - Can manage HW wallet the same way as others/Can update name/" src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:379:46: 14) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_06 - Test parameters, Address pool gap invalid, Pool gap: 9 uncaught exception: IOException of type InvalidArgument fd:700: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/SHELLEY_CLI_HW_WALLETS/HW_WALLETS_06 - Test parameters/Address pool gap invalid/Pool gap: 9/" src/Test/Integration/Scenario/CLI/Port.hs:133:5: 15) CLI Specifications, COMMON_CLI_PORTS, PORT_03 - Cannot omit --port when server uses random port (wallet create) uncaught exception: IOException of type InvalidArgument fd:643: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/COMMON_CLI_PORTS/PORT_03 - Cannot omit --port when server uses random port (wallet create)/" src/Test/Integration/Scenario/CLI/Port.hs:184:5: 16) CLI Specifications, COMMON_CLI_PORTS, PORT_03 - Cannot omit --port when server uses random port (address list) uncaught exception: IOException of type InvalidArgument fd:694: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/COMMON_CLI_PORTS/PORT_03 - Cannot omit --port when server uses random port (address list)/" src/Test/Integration/Scenario/CLI/Port.hs:196:13: 17) CLI Specifications, COMMON_CLI_PORTS, PORT_04 - Fail nicely when port is out-of-bounds, serve --port 1010 uncaught exception: IOException of type InvalidArgument fd:642: hGetContents: invalid argument (invalid byte sequence) To rerun use: --match "/CLI Specifications/COMMON_CLI_PORTS/PORT_04 - Fail nicely when port is out-of-bounds/serve --port 1010/"

Anviking · 2020-11-04T13:48:09Z

bors r+
(yolo)

2191: Parallelise integration tests using ResourceT r=Anviking a=Anviking # Issue Number ADP-466 (follow-up from ADP-419 ) # Overview  - [x] Use `runResourceT` in each test scenario to ensure only the wallet created in it are deleted. - [x] Re-write all `listWallets` expectations to check that the expected wallets are a _subset_ of the actual wallets, instead of checking for equality. - [x] Enable `-j 8` in buildkite # Comments - Buildkite now takes ≈35 min instead of 50 min - On my machine `stack test cardano-wallet:integration --ta '-j 8'` takes 380s instead of ~1800s. - Haven't enabled parallel tests in hydra yet because of errors - I now pushed `anviking/ADP-466/parallel` (parallel _everything_) directly here.   Co-authored-by: Johannes Lund <[email protected]> Co-authored-by: IOHK <[email protected]>

iohk-bors · 2020-11-04T17:06:48Z

Build failed:

ci/hydra-build:required

Timeout:

    parallel puts replace values in _any_ order
      Checkpoint
        +++ OK, passed 100 tests (49% conflicting db entries).
      Wallet Metadata
        +++ OK, passed 100 tests (49% conflicting db entries).
      Tx History
building of '/nix/store/1g02d3ibj4sa37v4ppwj2hw7azqbw6df-cardano-wallet-core-test-unit-2020.11.3-check' timed out after 900 seconds of silence

Anviking · 2020-11-04T17:08:49Z

bors r+

iohk-bors · 2020-11-04T19:14:10Z

Build succeeded:

Anviking self-assigned this Sep 29, 2020

iohk-bors bot added a commit that referenced this pull request Sep 29, 2020

Try #2191:

8d78c7a

Anviking force-pushed the anviking/replace-tearDown branch from 6c1613f to ddab4d3 Compare September 29, 2020 16:26

iohk-bors bot added a commit that referenced this pull request Sep 29, 2020

Try #2191:

eb95dd5

Anviking force-pushed the anviking/replace-tearDown branch from 6dce2e8 to f00890c Compare October 1, 2020 13:30

iohk-bors bot added a commit that referenced this pull request Oct 1, 2020

Try #2191:

1f37417

iohk-bors bot added a commit that referenced this pull request Oct 1, 2020

Try #2191:

8ea0404

iohk-bors bot added a commit that referenced this pull request Oct 2, 2020

Try #2191:

f053b93

Anviking closed this Oct 8, 2020

Anviking reopened this Oct 19, 2020

Anviking force-pushed the anviking/replace-tearDown branch 3 times, most recently from 1fde64f to 9a85557 Compare October 20, 2020 09:31

This comment has been minimized.

Sign in to view

iohk-bors bot added a commit that referenced this pull request Oct 20, 2020

Try #2191:

5439e90

Anviking force-pushed the anviking/replace-tearDown branch from 88626c0 to 68d9089 Compare October 20, 2020 11:24

This comment has been minimized.

Sign in to view

Anviking and others added 20 commits November 4, 2020 14:46

Adapt listWallets tests to work in parallel

5217e7c

Launch node with -N4 in integration tests

4249056

Enable parallel in integration tests

edd7724

Fix simplePayload and WALLET_UPDATE tests

bdf22d3

Make OWASP_INJECTION_CREATE_WALLET_01 use a dynamic mnemonic

4034744

such that it works in parallel. Removing the `walletId` check shouldn't really matter.

Make selectCoins work with MonadIO

fd02a3e

Replace liftIO $ verify with verify

336a066

Fixups which are difficult to squash

7a3a31a

Regenerate nix

bd1e6ca

Fix potential race in CLI HW_WALLETS_04

a4a17a5

Add HasCallStack to counterexample

e9573da

Fix for flaky TRANS_EXTERNAL_02

07f5e90

By mistake I first added the fix to TRANS_EXTERNAL_01 instead of TRANS_EXTERNAL_02. But I'm not keeping it on both to be safe.

Debug log for TRANS_EXTERNAL_02

13db5a2

Fix flaky TRANS_DELETE_01

5327b5d

Co-Authored-By: KtorZ <[email protected]>

Add counterexamples to debug TRANS_TTL_{01,02}

1e4c82f

Anviking force-pushed the anviking/replace-tearDown branch from 0625c46 to 1e4c82f Compare November 4, 2020 13:46

Anviking mentioned this pull request Nov 4, 2020

Flaky TRANS_TTL_{01,02} - SlotNo 80 > SlotNo 50 #2295

Closed

iohk-bors bot merged commit 0bf98c9 into master Nov 4, 2020

iohk-bors bot deleted the anviking/replace-tearDown branch November 4, 2020 19:14

Anviking mentioned this pull request Nov 9, 2020

Flaky test - various DB properties causing timeout #2292

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelise integration tests using ResourceT #2191

Parallelise integration tests using ResourceT #2191

Anviking commented Sep 29, 2020 •

edited

Loading

Anviking commented Sep 29, 2020

iohk-bors bot commented Sep 29, 2020

Anviking commented Sep 29, 2020

iohk-bors bot commented Sep 29, 2020

rvl commented Sep 30, 2020

Anviking commented Sep 30, 2020

Anviking commented Oct 1, 2020

iohk-bors bot commented Oct 1, 2020 •

edited by Anviking

Loading

Anviking commented Oct 1, 2020

iohk-bors bot commented Oct 1, 2020 •

edited by Anviking

Loading

Anviking commented Oct 2, 2020

Anviking commented Oct 2, 2020

Anviking commented Oct 8, 2020

This comment has been minimized.

This comment has been minimized.

Anviking commented Nov 4, 2020 •

edited

Loading

iohk-bors bot commented Nov 4, 2020 •

edited by Anviking

Loading

Anviking commented Nov 4, 2020

iohk-bors bot commented Nov 4, 2020

Parallelise integration tests using ResourceT #2191

Parallelise integration tests using ResourceT #2191

Conversation

Anviking commented Sep 29, 2020 • edited Loading

Issue Number

Overview

Comments

Anviking commented Sep 29, 2020

iohk-bors bot commented Sep 29, 2020

try

Anviking commented Sep 29, 2020

iohk-bors bot commented Sep 29, 2020

try

rvl commented Sep 30, 2020

Anviking commented Sep 30, 2020

Anviking commented Oct 1, 2020

iohk-bors bot commented Oct 1, 2020 • edited by Anviking Loading

try

Anviking commented Oct 1, 2020

iohk-bors bot commented Oct 1, 2020 • edited by Anviking Loading

try

Anviking commented Oct 2, 2020

Anviking commented Oct 2, 2020

Anviking commented Oct 8, 2020

This comment has been minimized.

This comment has been minimized.

Anviking commented Nov 4, 2020 • edited Loading

iohk-bors bot commented Nov 4, 2020 • edited by Anviking Loading

Anviking commented Nov 4, 2020

iohk-bors bot commented Nov 4, 2020

Anviking commented Sep 29, 2020 •

edited

Loading

iohk-bors bot commented Oct 1, 2020 •

edited by Anviking

Loading

iohk-bors bot commented Oct 1, 2020 •

edited by Anviking

Loading

Anviking commented Nov 4, 2020 •

edited

Loading

iohk-bors bot commented Nov 4, 2020 •

edited by Anviking

Loading