coin-split stress test wallet database errors

[hansieodendaal]

Describe the bug

Note: Running with the #5906 fix for large transactions on top of the latest development commit 1195afb8c5668eef6bcde5703cfcad6a106f8740 on esmeralda.

Running a system-level coin-split stress test involving 4x wallets produced these intermittent errors on some of the wallets:

- CoinSplit error! Transaction service error `Transaction storage error: `Tried to insert an output that already exists in the database``
- CoinSplit error! Transaction service error `Transaction storage error: `Diesel error: `UNIQUE constraint failed: completed_transactions.tx_id
- CoinSplit error! Output manager error: `Output manager storage error: `Output has already been spent``

Command-line output wallet 1:

minotari_console_wallet.exe --base-path . --input-file ..\@coin_split.txt --config %cd%\config\config.toml --log-config %cd%\config\log4rs_console_wallet.yml --network esmeralda

...

4. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

5. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

CoinSplit error! Transaction service error `Transaction storage error: `Tried to insert an output that already exists in the database``

6. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

CoinSplit error! Transaction service error `Transaction storage error: `Diesel error: `UNIQUE constraint failed: completed_transactions.tx_id```

7. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

CoinSplit error! Output manager error: `Output manager storage error: `Output has already been spent``

8. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

9. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

...

Monitoring 18 sent transactions to Broadcast stage...
The configured timeout (300s) was reached before all transactions reached the Broadcast stage. See the logs for more info.
Minotari Console Wallet running... (Script mode completed)

Press Enter to continue to the wallet, or type q (or quit) followed by Enter.

Command-line output wallet 2:

minotari_console_wallet.exe --base-path . --input-file ..\@coin_split.txt --config %cd%\config\config.toml --log-config %cd%\config\log4rs_console_wallet.yml --network esmeralda

...

2. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

3. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

4. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

CoinSplit error! Transaction service error `Transaction storage error: `Diesel error: `UNIQUE constraint failed: completed_transactions.tx_id```

5. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

CoinSplit error! Transaction service error `Transaction storage error: `Diesel error: `UNIQUE constraint failed: completed_transactions.tx_id```

6. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

7. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

8. CoinSplit(CoinSplitArgs { amount_per_split: MicroMinotari(10000), num_splits: 499, fee_per_gram: MicroMinotari(1), message: "Coin split" })

Coin split succeeded

...

Monitoring 19 sent transactions to Broadcast stage...
The configured timeout (300s) was reached before all transactions reached the Broadcast stage. See the logs for more info.
Minotari Console Wallet running... (Script mode completed)

Press Enter to continue to the wallet, or type q (or quit) followed by Enter.

To Reproduce

Steps to reproduce the behaviour:
Perform a system-level coin-split stress test:

• Collect 20 UTXOs of at least 5T each in a wallet
• Exit the wallet and execute this script from the command line:

# Let us stress the network

get-balance

coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499

get-balance

coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499
coin-split 0.01T 499

get-balance

coin-split 0.01T 100

get-balance

Desktop:
Windows 10

[SWvheerden]

fixed in #5916