Monitor memory allocation and stress wallet primitives

[rvl]

Context

We need automated tests to ensure that our wallet meets its performance requirements.

In particular, we need these test cases to detect space leaks, or wrong algorithmic complexity.

Decision

• Come up with test scenarios. For example:
  • receiving blocks from the backend
  • wallet restoration
  • listing transactions
  • input selection
• Make sure benchmarks are run in CI, at least nightly.

Acceptance Criteria

1. The generators must be able to generate sufficiently large data sets along all significant dimensions.
2. Memory usage must be shown for different data set sizes.
3. The benchmarks must be run in CI. They may be run nightly rather than on every push.

---

Development Plan

Basic

• Use mainnet and testnet data
• The listen polling function (5s interval) will make executing the test more tricky (and testing when it has finished applying blocks) -- make some workaround.
• Time restore operations
• Add benchmark job to CI. Use ghc +RTS -h option to get memory usage statistics.

More interesting

• Change mkWalletLayer so that the address derivation scheme can be swapped out.
• Make an address derivation scheme which "owns" a given proportion of addresses.
• Add more benchmarks with varying address ownership (and hence number of transactions).

Better

Moved to https://github.com/input-output-hk/cardano-wallet/wiki/Blackboard#better-restoration-stress-benchmark

PR

Number	Branch	Base
#157	rvl/100/bench	master
#160	rvl/100/address-discovery	rvl/100/bench
#169	-	master
#190	-	master

QA

• We now run nightly build restoration benchmarks on buildkite against both mainnet and testnet.

• For each network, we run three benchmarks, playing with different version of 'isOurs' (address discovery)

  • The 'classic' one for sequential wallet. Downside is that we are restoring a wallet from a public arbitrary mnemonic and end up with an empty wallet. Still, we go through the whole chain and process every non-empty block

  • To step-up a bit the game, we also run a 10% isOurs, which recognize, at random, 10% of the address on the chain (using the crc on each address). This allows us to create artificially big(ger) wallets and observe the memory usage for those.

  • Even more, and to observe how the backend scale, we run a 50% isOurs heuristic where half of the chain gets recognized as "ours".

• Builds (with status visible on the README via the NIGHTLY base currently generate a single artifact, which is a heap memory visualization of the benchmark). We don't have any ways of automatically checking the benchmark results at the moment, so this is still something which requires a manual checks. A bullet point to the release checklist has been added to remind ourselves to do so.

[KtorZ]

It'd be nice if we could then set some thresholds that would make the CI fails if they are crossed. Also, the ticket only mentions space benchmarking, but would it make sense to also do some time benchmarking on some operations (like coin selection, wallet restoration against mainnet, address discovery for BIG set of addresses)?

[KtorZ]

• Figure out a way of generating semi-realistic transactions in blocks
• Use a mock network layer to feed generated blocks to wallet layer

@rvl What do you think about using mainnet or testnet data for this? (at least initially). One "fun" thing maybe, would be to provide an 'isOurs' implementation that pretty much recognize ALL addresses as ours, (or one every other) which will such out pretty much everything from mainnet / testnet. Not sure how many transactions is represents, but probably something like 21600 * 110 ~ 2M 🤷‍♂️

[rvl]

What do you think about using mainnet or testnet data for this?

That is something we should definitely do -- an acceptance test like we have for the old wallet. The nightly test also measures time taken and heap usage.

I am happy to use the existing chains initially. However we will only be testing historical data, not future possible inputs. And there are other drawbacks.

Adding special isOurs implementations is a good idea.

[KtorZ]

@rvl let's not tackle the Better section as part of this ticket but, keep that for another ticket later on.

[piotr-iohk]

lgtm 👍
Good that it is a badge on README and also that it's added as part of release checklist, so it is not something that just runs somewhere and noone checks the results or checks them very seldom.

For that matter I have added also the idea from the comment -> https://github.com/input-output-hk/cardano-wallet/wiki/Blackboard#benchmark-tests---automatically-check-the-results