[functional testing] pages-layout tests need update for tor based testing

[kushaldas]

Part of epic #3659.

Description

As we are moving towards Tor browser based functional tests, the pages-layout tests also need update to work properly.

Right now all of them happily failing on the local_tests branch.

[kushaldas]

In the pages-layout tests, the test_source_flagged requires deletion of the source key from the file system, means it can only run in the local instance. I propose we execute this only in the local container based tests but not against any onion address. @redshiftzero @conorsch @dachary

[kushaldas]

My suggestion would be to run the layout tests only against the local container as only running the test_journlist.py takes the following time.

102 passed in 3420.95 seconds.

[kushaldas]

102 passed in 2637.02 seconds

That is directly using Firefox ESR on the container. Still 2 tests are failing, related to HOTP.

[kushaldas]

Below are a few findings from today's test marathon.

Version mismatch between Firefox and geckodriver

We are using an older version of geckodriver due to dependency of the Tor Selenium and that means we have to the use the Firefox ESR version. This brings in problem like #3628 (comment)

Test timing

We can run the tests against both local and external onion address using the same code base, but that means Tor Browser will have to start everytime, and that increases time for the pages-layout tests to more than 1 hours (around 1 hour and 20 minutes).

If we use a direct Firefox ESR browser and directly connect to localhost for the container based tests, the things are much better and all pages-layout tests are passing properly.

[conorsch]

We are using an older version of geckodriver due to dependency of the Tor Selenium

For those playing along at home, the tor-browser-selenium setup instructions state:

You also need to install geckodriver v0.17.0 from the geckodriver releases page. Make sure you install the v0.17.0 version; newer or older versions will not be compatible with the current Tor Browser series.

Regarding potential problems with the older geckodriver

This brings in problem like #3628 (comment)

In the develop branch, we're still running FF 46, rather than 52 ESR, which is is being introduced via the TBB Selenium work. In a nutshell, is the problem you're experiencing @kushaldas strictly that we cannot force window sizes? I agree with @redshiftzero that we should indeed do so where possible, but if that's the only blocker, then it sounds like we can indeed use 52 ESR for page-layout tests and Tor Browser for the functional nav tests.

that means we have to the use the Firefox ESR version

I would entertain solutions that allow us to use TBB everywhere, although the long-running test times is indeed a problem. If we can use FF 52 ESR for page-layout (for faster tests) and keep the functional tests over Tor for use in QA, that'd be sufficient for closing out the current TBB Selenium push.

[kushaldas]

This is work in progress on top of #3628.

layouts_wip.txt

I working on top of @msheiny's work 1c4750943ade3e5ae661e84d8e751f76b674636e. After applying the patch, you can run all the pages-layout tests with the following command.

$ ./bin/dev-shell ./bin/run-test -v tests/pages-layout

Remember that 153 passed in 3724.53 seconds :)

[kushaldas]

While running the whole test suite together, I have:

404 passed, 2 xfailed in 5612.60 seconds

[kushaldas]

Now I have all the tests passing locally, some good news finally.

[redshiftzero]

👍 to continuing to use Firefox for the page-layout tests. It's a nice-to-have to run them using Tor Browser because then the autogenerated screenshots in the user guides would be rendered using Tor Browser instead of Firefox (there are slight differences but nothing major or confusing). That said, this is not worth significantly lengthening the CI staging job.

[conorsch]

Resolved via #3664.