[Tests] GUI tests on Mac randomly fails with Fatal Python error: Segmentation fault

[drew2a]

The full log: https://jenkins-ci.tribler.org/job/GH_Tribler_PR_Tests/job/PR_GUI_macOS/3519/console

Fatal Python error: Segmentation fault

Current thread 0x00000001086b95c0 (most recent call first):
<no Python frame>
/var/folders/9h/rc1ccw796x7dsjy2_nxjrxw40000gn/T/jenkins7492326316941345297.sh: line 11: 20855 Segmentation fault: 11  pytest src/tribler-gui/tribler_gui --guitests -p no:randomly -v --cov=tribler_gui --cov-report=xml
Build step 'Execute shell' marked build as failure

Error occurred during in #6563

[devos50]

I did see this one earlier on my machine but it's very hard to reproduce and seems to happen randomly. A screenshot of the crash details as reported by MacOS:

So this seems to happen during some cleanup procedures when exiting the Python interpreter. Searching for sip_api_visit_wrappers brings me to this issue, reporting a similar crash. Some more information:

When the Python interpreter leaves a scope (for example when it returns from a function) it will potentially garbage collect all objects local to that scope. The order in which it is done is, in effect, random. Theoretically this can cause problems because it may mean that the C++ destructors of any wrapped Qt instances are called in an order that Qt isn’t expecting and may result in a crash. However, in practice, this is only likely to be a problem when the application is terminating.

As a way of mitigating this possiblity PyQt5 ensures that the C++ destructors of any QObject instances owned by Python are invoked before the destructor of any QCoreApplication instance is invoked. Note however that the order in which the QObject destructors are invoked is still random.

Since the stack trace does not provide any further useful information, this one can be pretty nasty to debug. It could be that we are mis-managing a particular QObject but given the sheer amount of objects in our application, we probably need to attach a debugger to the crashing process to determine which object is causing the crash.

Updating the version of Python/Qt/sip might also helpful.

[kozlovsky]

Good finding about the sip!

Seems related to:
https://gitanswer.com/pyqtgraph-segmentation-fault-during-garbage-collection-python-826318977
mantidproject/mantidimaging#875
mantidproject/mantidimaging#894

If I understand correctly, it is possible that the error happened not when the Python interpreter exits, but when pyqtgraph closes the ImaveView object. Then we probably can avoid segfault by changing the way we work with pyqtgraph.

[drew2a]

The error still occurs in #6702
Console output: https://jenkins-ci.tribler.org/job/GH_Tribler_PR_Tests/job/PR_GUI_macOS/3881/console

[drew2a]

Didn't see any segmentation fault errors for a long time.