Use Tor for functional testing and pre-release QA

[eloquence]

In order to increase fidelity of our functional tests in CI, and to partially automate our pre-release QA, we want to:

• use Tor (via tbselenium) instead of Firefox for running our functional tests;
• make it possible to run those tests against an external server.

This work is happening in a feature branch, tbb-0.9.0. Changes against that branch should be tightly scoped and go through review. Issues, when closed, can be marked as "Done in feature branch" on the board. Once the above goals are met, we will prepare a merge into develop.

Individual issues related to this epic have [functional testing] in the title and the functional tests label. As an epic, this issue will remain in the near-term backlog; individual sub-tasks will be moved across the board. New sub-tasks may be added to the epic as the work progresses.

User Stories

• As a developer, I want the functional tests to catch issues that are specific to the Tor browser, which would not be discovered when testing only against Firefox, so that I can fix them prior to merge.
• As a developer, I want a failure to make necessary changes to AppArmor profiles to be caught by CI so that bugs don't creep in to the develop branch.
• As a release manager, I would like to be able to run my entire suite of functional tests against a release candidate running in VMs in order to reduce manual testing and speed up the release QA process.

Tasks in epic

For tasks flagged as investigation tasks, please open up new tasks once the investigation is complete.

Preparatory to CI integration

During this stage, it is acceptable if we see CI failures in the feature branch.

• Install latest Tor Browser, geckodriver in dev + test - [functional testing] Install latest Tor Browser, geckodriver in dev + test #3485
• Dismiss or supress Tor startup window - [functional testing] Determine how to handle TB initial startup window #3487
• Gracefully start the server only when needed - [functional testing] Gracefully start the server only when needed #3483
• Run local instance tests in container - Tests against a local instance in the container #3628
• pages-layout tests need update for tor based testing functional_tests - [functional testing] pages-layout tests need update for tor based testing #3656
• Remove time.sleeps where possible - [functional testing] Remove time.sleeps where possible #3486
• Investigation task: Determine how to handle users when testing external server [functional testing] Determine how to handle users when testing external server  #3488
  • create-dev-data.py should get one user for stage based testing - [functional testing] create-dev-data.py should get one user for stage based testing #3671
• Replace command based download with pure Python - [functional testing] Replace command based download with pure Python #3691
• Remove TBB install logic from app-test role - [functional testing] Remove TBB install logic from app-test role #3678
• Resolve flaky logout in TB tests - [functional testing] Resolve flaky logout in TB tests #3484
• Add VNC to container to make tests observable - [functional testing] Add VNC server to functional test runner container #3718

CI integration

After this stage, we must not see any CI failures in the feature branch.

• Fix container build issues for Tor tests in CI - [functional testing] Support containerized tor-browser-selenium tests in CI- #3660
• Remove DEBUG log level from functional test output [functional testing] Remove DEBUG log level output #3698
• Extract tor details from staging to instance_information.json - [functional testing] Extract tor details from staging to instance_information.json #3621
• Run Tor staging tests in CI against external server - [functional testing] Support tor-browser-selenium tests against app-staging in CI #3661

Merge

Please open up follow-up issues as appropriate during review.

• Final review and merge into develop

[redshiftzero]

Going forward, the pages-layout tests are not expected to pass against an external server, only the regular functional tests. This is something that we should document when we merge this feature branch into develop.

[redshiftzero]

The tbb-0.9.0 feature branch has been around for some time, and is consuming significant team time to keep maintaining. I propose we scope this down by:

• Merging Remove DEBUG output from functional tests, fix test requirements, add VNC server #3817 once all tests are passing
• Address [functional testing] Remove time.sleeps where possible #3486 (cleanup task prior to final merge)
• Determine if [functional testing] Functional tests using GPG fail when run against staging VMs #3687 is still an issue, if so fix (not an issue, thanks @zenmonkeykstop!)
• Final review/cleanup and merge into develop

This branch must be in a non-broken state. That is:

• tests must reliably pass locally,
• tests must reliably pass in CI (in the test container),
• tests (without pages-layout) must pass against an external server. Even if testing against an external server is flaky, I strongly suggest we merge this branch in and address the flakiness of testing against an external server in followups. In my opinion, it is acceptable in this case because testing against an external server is not in CI, nor is it in the path of developers continuing to work. The alternative is that we either rip out the functionality of the external server logic entirely, which would be significant backtracking, or we delay the merging of this work further, which introduces maintenance burden for the team continuing to work on this branch and keep it up to date with develop.

Finally, #3661 (previously part of this epic) would be kept in the backlog as a future ticket. This means that we would be removing the execution of the unit and functional tests in the staging environment. This is acceptable in my opinion as we already run the unit and functional tests in another CI job. That said, we should add back the running of the unit tests in the staging environment if we ever use grsec kernels for the staging CI machines.

What do y'all think?

[redshiftzero]

At this point, all these tasks are done except external server testing does not work (after Tor Browser 8, one gets a message about the proxy server refusing connections). In the upstream issue, it's recommended to use an external tor process. This does work for a subset of tests, but all tests won't pass due to other issues, i.e. the js disabling not working under Tor Browser 8. In order to get what we've done merged so we can make further progress in smaller steps, I think that we should merge the functionality as is, but add a commit that we can (ideally) later cleanly revert that adds the functionality for external server testing.

[eloquence]

All the essential work is done in a feature branch and now needs to be cherry-picked/squashed/rebased as appropriate and merged iteratively. Will open separate tickets for that, closing this epic.