Suggestion: start and manage GUI tests from within the Qt application

[devos50]

Currently, we use pytest to start, run, and manage the GUI tests. pytest starts the Qt application and closes it when the tests are done. At the moment, we see that our GUI tests are unstable. I suspect that this is partly caused by the combination of Qt and pytest when running the tests. Therefore, I believe it is worthwhile to explore an alternative approach: run the GUI tests from within the Qt application itself. The main motivation for this approach is to remove an element that could be interfering with the GUI tests, namely the pytest infrastructure.

This would work as follows: we can add an addition flag, e.g., --gui-tests to the run_tribler.py script. This flag indicates that Tribler should start in GUI test mode and execute the GUI tests. Tribler will read the methods in the test_gui.py file and extract the methods which name start with test_. When the tests are done, Tribler will be closed automatically and exit with either process exit code 0 (all tests passed) or exit code 1 (one of the tests have failed). This approach would require some logic to identify and run the tests. We should also explore support for code coverage, and printing tracebacks when one of the tests have failed.

Advantages of this approach:

• We sidestep the need for pytest to run the GUI tests, removing an element that might be interfering with the GUI tests. In the long term, this should lead to more stability when executing the GUI tests.
• If we modify our build pipeline, we can also run these tests in frozen mode. This can help us to debug GUI glitches that only occur in a frozen environment and/or on specific platforms.

Disadvantages of this approach:

• We need to write some additional logic to execute GUI tests ourselves (since we cannot use the pytest infrastructure anymore).
• It is more work to execute a single test or a few select tests. This situation, however, does not occur often in my experience and we can always implement support for this later.

I have implemented a prototype of this approach in a branch, see here.

[kozlovsky]

I was able to fix the stability of the GUI tests; the PR will be ready in two hours. Probably, it is good to keep the ability to run GUI tests from pytest.

On the other side, I like the idea of the possibility of running tests in frozen mode. It may be really useful in some situations!

[qstokkink]

You could consider bundling your tests in a unittest.TestCase to avoid writing your own test_ name matching. The unittest module has all sorts of functions available to automatically load or find tests for TestCases (e.g., unittest.TestLoader.loadTestsFromTestCase). This is fully compatible with pytest.

[qstokkink]

Tribler now has a web-based front end and this issue is no longer relevant.