Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: Various CLI tests failing due to high load #2855

Closed
1 task done
Anviking opened this issue Aug 26, 2021 · 3 comments
Closed
1 task done

Flaky test: Various CLI tests failing due to high load #2855

Anviking opened this issue Aug 26, 2021 · 3 comments
Labels
Test failure A flaky test or nightly CI failure

Comments

@Anviking
Copy link
Member

Anviking commented Aug 26, 2021 •
edited
Loading

Please ensure:

  • This is actually a flaky test already present in the code and not caused by your PR.

Context

From #2854:

There has been strange CI failures where the wallet seems overloaded
(balance is 0, and wallet is unsynced even after 90s). They seem CLI
specific.

There may be several reasons for the CLI specific failures. But one is:
It seems createWalletViaCLI does not automatically clean up wallets
using ResourceT. In the CLI.Shelley.Wallets module there are 15 calls to
createWalletViaCLI, but only 5 corresponding deleteWalletViaCLI ones.

The wallet server does not perform well with many wallets. So leaking
wallets might contribute to slowness and failures.

Group of:

https://github.com/input-output-hk/cardano-wallet/blob/de41cdeb78d8057b22db1689887dc3a9092b590e/lib/shelley/test/integration/Main.hs#L218-L221

Job name

Integration

Test case name(s)

No response

Error message

Various:

- `expected: Ready, but got: Syncing (Quantity (Percentage (4913 % 5000)))`
- `hFlush: resource vanished (Broken pipe)`
- `writev: resource vanished (Broken pipe)`
- `Waited longer than 90s to resolve action: "Wallet balance is as expected".`

Build link

#2721 (comment)
@Anviking Anviking added the Test failure A flaky test or nightly CI failure label Aug 26, 2021
@Anviking Anviking changed the title Flaky test: Various CLI tests failing from high load Flaky test: Various CLI tests failing due to high load Aug 26, 2021
This was referenced Aug 26, 2021
Closed
Closed
Closed
Closed
Merged
Closed
@rvl
Copy link
Contributor

rvl commented Aug 27, 2021

Good analysis, thanks @Anviking. 👍

This was referenced Aug 27, 2021
Merged
Merged
iohk-bors bot added a commit that referenced this issue Aug 29, 2021
@iohk-bors @Anviking
Merge #2857 #2858
e43dec2 
2857: Fix leaks of ≈21 more wallets in integration tests r=rvl a=Anviking

- [x] Replace 3 manual calls to `Link.postWallets` with the `ResourceT` `postByronWallet`
- [x] Also make `createWalletFromPublicKeyViaCLI` run in `ResourceT`

### Comments

- 21 wallets is a pretty significant number, so will be good to fix. Although the leaks were at the end of the tests, which lessens the impact.
- There seem to be _some_ concurrency related leaks.

#### Before second fix

- --match "WALLET" -j 1 ->18 leaks, all at the end

#### After fixes

- --match "WALLET" -j 1, no leaks
- --match "WALLET" -j 8, 10 leaks (3 BUSY, 6 others)

```
SQLITE_BUSY in logs:
ica.14828b804a1ebe804caddd1abb5d314d2be0a62a.sqlite
ica.25341683360a040721f587d069b95762b302f608.sqlite
rnd.5fc0960622bc4d3a5d366a0920dfd9b5ef578ced.sqlite

rnd.81273c47add77e98b3993e01cab8b2a836cd48b7.sqlite
she.95cff5e68f07b759c72d4da4f1aa96e0b101bba2.sqlite
she.a99218ce4bbc77f05617d642c0595246621b67b6.sqlite

Remaining DBs:
ica.14828b804a1ebe804caddd1abb5d314d2be0a62a.sqlite
ica.25341683360a040721f587d069b95762b302f608.sqlite
rnd.5fc0960622bc4d3a5d366a0920dfd9b5ef578ced.sqlite

ica.ed49f6e7723fa2c15575c147566981beab6aba3e.sqlite
rnd.693940686b48985ab6ec8035a56575e0473e96de.sqlite
rnd.81273c47add77e98b3993e01cab8b2a836cd48b7.sqlite
sha.9316fc445533f9e58becf0d367d3a9404585ea86.sqlite
she.3464b427c15341dab8aaf6d89c8f79e5f24f9eae.sqlite
she.a0bd7c6accac92d54fb667bf32bcba4db4cd552f.sqlite
stake-pools.sqlite
```


<!-- Additional comments, links, or screenshots to attach, if any. -->

### Issue Number

ADP-1090


2858: Add (Instance of #ISSUE) re-write rule for bors comments r=rvl a=Anviking

- [x] Add simple re-write rules for bors-failure tags using annotations in Github issue titles

### Comments

This gives us a fairly easy way to close specific tickets in favor of a
more general one, without having to edit the corresponding bors-comments.

This is done by writing e.g. "(Instance of #2)" for issue #1, if you
want re-annotate failures from #1 as really being from #2.

**Example:** #2855 

<!-- Additional comments, links, or screenshots to attach, if any. -->

### Issue Number

<!-- Reference the Jira/GitHub issue that this PR relates to, and which requirements it tackles.
  Note: Jira issues of the form ADP- will be auto-linked. -->


Co-authored-by: Johannes Lund <[email protected]>
This was referenced Sep 1, 2021
Merged
Merged
@jonathanknowles jonathanknowles mentioned this issue Sep 3, 2021
Merged
@Anviking Anviking mentioned this issue Dec 14, 2021
Merged
1 task
@Anviking
Copy link
Member Author

There has been a recent uptick of similar-looking failures:

./scripts/bors-stats.rb --after "13 nov" --tag "#2855" --details true:

16 Nov 09:14 #2855, buildkite https://github.com/input-output-hk/cardano-wallet/pull/3021#issuecomment-970071727
https://buildkite.com/input-output-hk/cardano-wallet/builds/17681
Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/CLI/Shelley/Transactions.hs:243:59:
--
  | 1) CLI Specifications, SHELLEY_CLI_TRANSACTIONS, TRANS_CREATE_06 - Invalid amount, string with diacritics
  | uncaught exception: IOException of type ResourceVanished
  | fd:108: hFlush: resource vanished (Broken pipe)

#2500
16 Nov 10:49 #2855, buildkite https://github.com/input-output-hk/cardano-wallet/pull/3021#issuecomment-970149115
https://buildkite.com/input-output-hk/cardano-wallet/builds/17685
Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/CLI/Shelley/Transactions.hs:243:59:
--
  | 1) CLI Specifications, SHELLEY_CLI_TRANSACTIONS, TRANS_CREATE_06 - Invalid amount, []
  | uncaught exception: IOException of type ResourceVanished
  | fd:109: hFlush: resource vanished (Broken pipe)

#2500
16 Nov 11:37 #2855, buildkite https://github.com/input-output-hk/cardano-wallet/pull/3018#issuecomment-970185877
https://buildkite.com/input-output-hk/cardano-wallet/builds/17686
Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/CLI/Shelley/Transactions.hs:243:59:
--
  | 1) CLI Specifications, SHELLEY_CLI_TRANSACTIONS, TRANS_CREATE_06 - Invalid amount, string with diacritics
  | uncaught exception: IOException of type ResourceVanished
  | fd:102: hFlush: resource vanished (Broken pipe)
  |

#2500
30 Nov 11:37 #2855, buildkite https://github.com/input-output-hk/cardano-wallet/pull/3039#issuecomment-982554208
https://buildkite.com/input-output-hk/cardano-wallet/builds/17806
Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/CLI/Shelley/Transactions.hs:244:59:
--
  | 1) CLI Specifications, SHELLEY_CLI_TRANSACTIONS, TRANS_CREATE_06 - Invalid amount, no amount
  | uncaught exception: IOException of type ResourceVanished
  | fd:121: hFlush: resource vanished (Broken pipe)

#2500
This problem was supposed to be fixed. Interesting.
14 Dec 22:41 #2855, buildkite https://github.com/input-output-hk/cardano-wallet/pull/3051#issuecomment-994109717
https://buildkite.com/input-output-hk/cardano-wallet/builds/17954
Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/CLI/Shelley/Transactions.hs:392:59:
--
  | 1) CLI Specifications, SHELLEY_CLI_TRANSACTIONS, TRANS_ESTIMATE_09 - Invalid amount, string with diacritics
  | uncaught exception: IOException of type ResourceVanished
  | fd:130: hFlush: resource vanished (Broken pipe)

#2855
15 Dec 15:02 #2855, buildkite https://github.com/input-output-hk/cardano-wallet/pull/3050#issuecomment-994878034
https://buildkite.com/input-output-hk/cardano-wallet/builds/17972
Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/CLI/Shelley/Transactions.hs:244:59:
--
  | 1) CLI Specifications, SHELLEY_CLI_TRANSACTIONS, TRANS_CREATE_06 - Invalid amount, 1.5
  | uncaught exception: IOException of type ResourceVanished
  | fd:144: hFlush: resource vanished (Broken pipe)

#2500

succeeded: 0 failed: 6 (100%) total: 6
Excluding 0 #expected or #duplicate failures

@Anviking Anviking mentioned this issue Dec 15, 2021
Closed
1 task
@rvl rvl mentioned this issue Dec 21, 2021
Merged
@Anviking Anviking mentioned this issue Jan 19, 2022
Merged
1 task
This was referenced Feb 10, 2022
Merged
Merged
Merged
Merged
@Anviking Anviking mentioned this issue Feb 24, 2022
Merged
@Anviking
Copy link
Member Author

I don't think there's any current problem specific to CLI tests. Closing. There was an issue of leaking wallets which was fixed some year ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Test failure A flaky test or nightly CI failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Anviking @rvl