reduce failure rate of 2FA admin functional tests #4591
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Status
Ready for review
Description of Changes
Fixes #4580
Fixes #4541
This PR adds a retry loop function for clicking Selenium alerts for the 2FA tests. My procedure for debugging these test issues is to:
Hop into the dev container and
pip3 install pytest-repeat
Add the
--count
arg thatpytest-repeat
provides so I can rerun the offending test, e.g. here 100 times:develop
to confirm I'm reducing the failure rate, my observations from local testing:With this PR: 100/100 success
On develop: 99/100 success
With this PR: 100/100 success
On develop: 97/100 success
Note that there's obviously going to be some variability and these are low number statistics so your results may be slightly different.
Testing
While one could redo the above, since these are test only changes I don't think it's super valuable redoing the above. Instead, I would just scan the diff and see if it looks reasonable (e.g.
_admin_visits_reset_2fa_totp
didn't have any retry logic previously so not surprising it had a higher failure rate). If we see these same issues pop up again, we can reopen the tickets.Deployment
Test only
Checklist
If you made changes to the server application code:
make lint
) and tests (make -C securedrop test
) pass in the development container