Thirdparty TTK Test-Collection is not repeatable

[mdebarros]

Summary:

Thirdparty TTK Test-Collection is not repeatable, the following result will occur if helm test command is run a second time:

--------------------FINAL REPORT--------------------
Test Suite:Thirdparty Tests
Environment:Development
┌───────────────────────────────────────────────────┐
│                      SUMMARY                      │
├───────────────────┬───────────────────────────────┤
│ Total assertions  │ 21                            │
├───────────────────┼───────────────────────────────┤
│ Passed assertions │ 16                            │
├───────────────────┼───────────────────────────────┤
│ Failed assertions │ 5                             │
├───────────────────┼───────────────────────────────┤
│ Total requests    │ 20                            │
├───────────────────┼───────────────────────────────┤
│ Total test cases  │ 8                             │
├───────────────────┼───────────────────────────────┤
│ Passed percentage │ 76.19%                        │
├───────────────────┼───────────────────────────────┤
│ Started time      │ Fri, 11 Mar 2022 09:53:29 GMT │
├───────────────────┼───────────────────────────────┤
│ Completed time    │ Fri, 11 Mar 2022 09:53:31 GMT │
├───────────────────┼───────────────────────────────┤
│ Runtime duration  │ 2809 ms                       │
└───────────────────┴───────────────────────────────┘

Workarounds:

1. The following manual steps can be followed to clear the necessary persistent data to re-run the test successfully again:

1.1 Account Linking - Happy Path OTP

1.1.1 PISP Cache

del c51ec534-ee48-4575-b6a9-ead2955b8069

1.1.2 Consent & Auth DB

delete from Consent where id='46876aac-5db8-4353-bb3c-a6a905843ce7';

1.2. Third Party Transaction Request - Happy Path

1.2.1 PISP Cache

del b5d6206c-4f06-497d-af15-ed866ea6958f
del c2470148-1be2-4c0b-aece-aa8dcb92a6cc

1.2.2 Consent & Auth DB

delete from Consent where id='2acf1dfa-ce45-486e-b19e-ae4ad9804a63';

1.3. All keys added to PISP Cache from running tests for completenes

1. "c51ec534-ee48-4575-b6a9-ead2955b8069"
2. "callback_329294234"
3. "b51ec534-ee48-4575-b6a9-ead2955b8069"
4. "b5d6206c-4f06-497d-af15-ed866ea6958f"
5. "d51ec534-ee48-4575-b6a9-ead2955b8069"
6. "c2470148-1be2-4c0b-aece-aa8dcb92a6cc"
7. "key-parties-MSISDN-329294234-undefined"

2. Alternatively, re-run helm test command by excluding the Thirdparty tests as follows

❯ helm test moja3 --filter !name=ml-ttk-test-val-tp,!name=ml-ttk-test-setup-tp

Severity:
Low

Priority:
Low

Expected Behavior

Thirdparty TTK Test-Collection should be repeatable.

Steps to Reproduce

1. Run Thirdparty Provisioning collection
2. Run Thirdparty Test collection
3. Run Thirdparty Test collection again and compare results

Specifications

• Component (if known): Thirdparty Test Collection
• Version: v13.1.0
• Platform: Testing-Toolkit
• Subsystem: n/a
• Type of testing: n/a
• Bug found/raised by: @mdebarros

Notes:

• Severity when opened: low
• Priority when opened: low

[kleyow]

Here is the script used to reset the docker test harness. This could be helpful in adapting it to helm.

https://github.com/mojaloop/pisp-project/blob/master/scripts/_reset_test_state.sh