Flaky test - Cardano.Wallet.Shelley.Transaction estimateMaxNumberOfInputs timeout - macOS

[rvl]

Context

A unit test is sometimes failing on macOS due to a timeout.

Test Case

• Package: cardano-wallet
• Module: Cardano.Wallet.Shelley.TransactionSpec
• Spec: estimateMaxNumberOfInputs for ByronKey on Mainnet
• Platform: macOS
• CI: Hydra

Failure / Counter-example

https://hydra.iohk.io/build/4162295

Failures:

  test/unit/Cardano/Wallet/Shelley/TransactionSpec.hs:222:9:
  1) Cardano.Wallet.Shelley.Transaction, estimateMaxNumberOfInputs for ByronKey on Mainnet, less outputs ==> more inputs
       Timeout (after 340 tests):
         Quantity {getQuantity = 1888}
         205

  To rerun use: --match "/Cardano.Wallet.Shelley.Transaction/estimateMaxNumberOfInputs for ByronKey on Mainnet/less outputs ==> more inputs/"

---

Resolution

---

QA

[KtorZ]

That test as a timeout on the execution time, which is maybe a bit too tight for the mac mini running this in CI. Since it passes most of the time, we're probably not that far (although I am afraid that we'll hit the same issue with Windows).

We could increase the timeout and still make sure that the function executes within a reasonable time.

[rvl]

Is the timeout to ensure that excessively slow shrinkers or generators can be stopped? How long is it?

For this failure, is the timeout too short or is the test case too slow?

[KtorZ]

The timeout is to ensure that the function being testing executes within an acceptable time. That was part of the debugging / slowness analysis of large wallets. That function was on the critical path. Now, the acceptable time was previously set to 100ms yet, it's taking longer in CI for some generated values.

For this failure, is the timeout too short or is the test case too slow?

The tests (there are 3 using a timeout) are very short actually, some assertion on the result of estimateMaxNumberOfInputs with different parameters, so the test themselves are calling this function twice and checking that it doesn't take too long. Even one second would be fine here. But 20 or 30 wouldn't.

[Anviking]

Haven't seen again since 22 Jan. Think we can close.