[functional testing] Resolve flaky logout in TB tests

[redshiftzero]

In the existing Tor Browser functional testing branch, the logout in the journalist interface is sometimes flaky - i.e. it fails to log the user out and the test fails. We need to investigate why this is happening and resolve it prior to merge. Alternatively if it cannot easily be resolved, we should add a retry such that there are not spurious test failures.

Epic: #3659

[conorsch]

Lately, I've been running the TBB functional test suite in the long-running feature branch tbb-0.9.0, and have yet to see this issue crop up. @redshiftzero do you have any more info to share here? (I assume if you had copy/pasteable error output, it'd be in here already 😇; just asking for completeness.)

@zenmonkeykstop Try running the functional tests frequently on tbb-0.9.0 (including with the changes for the as-yet-unmerged #3664). Report any errors you encounter! The tests have been stable for me lately, but with more runs, we're likely to catch at least a few failures. Perhaps we'll catch this one. It's particularly important to run the tests manually given that the tbb-0.9.0 branch doesn't have working CI yet (see #3660).

If we cannot find evidence of the flaky logout, then we should close this ticket, as changes elsewhere in the TBB push may have resolved. I also wonder how changes for #3486 will affect occurrence of the flakey logout.

[redshiftzero]

I don't have more details to report: this was reported by @kushaldas in discussion during a video chat we had. I filed this ticket (along with the other functional testing tickets) to break up the outstanding tasks into individual tickets. I can report that I have not seen this in some time (I ran the tests several times successfully during review of #3628), so if @zenmonkeykstop doesn't hit this when he runs against his instance then 👍 to close

[zenmonkeykstop]

I'm running the functional tests from tbb-0.9.0 against prod VMs and seeing a bunch of failures. User setup is as per the README, except that the READMe references a script create-demo-user.py which doesn't exist - I'm assuming that it means create-dev-data.py and running that instead?

The following tests fail:

Result	Test	Duration	Links
Failed	functional/test_admin_interface.py::TestAdminInterface::()::test_admin_interface	63.82
Failed	functional/test_admin_interface.py::TestAdminInterface::()::test_admin_edits_hotp_secret	71.59
Failed	functional/test_admin_interface.py::TestAdminInterface::()::test_admin_deletes_user	74.34
Failed	functional/test_source_session_timeout.py::TestSourceSessions::()::test_source_session_timeout	38.60
Failed	functional/test_submit_and_retrieve_file.py::TestSubmitAndRetrieveFile::()::test_submit_and_retrieve_happy_path	131.04
Failed	functional/test_submit_and_retrieve_message.py::TestSubmitAndRetrieveMessage::()::test_submit_and_retrieve_happy_path

Will drill down a bit and figure out why; but @conorsch and @kushaldas if the test instance should be set up differently, please advise.

[kushaldas]

@zenmonkeykstop Please test it against the local instance using the following command.

$ ./bin/dev-shell ./bin/run-test -v tests/functional/

[zenmonkeykstop]

Oh, my bad @kushaldas - running locally, all tests pass. Will rerun them some more to see if I can trigger the flaky logout failure.

[zenmonkeykstop]

@kushaldas I've run the tests repeatedly against the local dev instance and have not been able to replicate flaky logout. Everything passes or xfails. 👍 from me.

[redshiftzero]

thanks @zenmonkeykstop. Closing!