Flaky core tests

[drew2a]

I'm trying to find flaky tests in the Tribler test suite by using this tool: https://github.com/dropbox/pytest-flakefinder

Here is the description of the experiment: Each test was run 50 times (no parallel), using macOS as the operating system and Python version 3.9.6.

The founded flaky tests are the following:

• core/components/metadata_store (test_channel_update_and_download)
• core/components/libtorrent (test_move_to_non_existing_dir, test_seeding) FREEZE
• core/components/restapi (no running event loop)
• core/components/restapi (test_write_data)
• core/components/restapi (test_maybe_add)
• core/components/restapi (test_aiohttp_assertion_patched)
• core/components/tunnel (test_payouts_e2e)
• core/tests (test_enable_fault_handler) false positive
• core/utilities (test_show_system_popup_win) false positive
• core/utilities (test_create_torrent_two_files) FREEZE

From real PRs:

• test_consolidate_channel_torrent https://github.com/Tribler/tribler/actions/runs/5543081002/jobs/10118622017?pr=7542#step:6:1236 (FREEZE), can not reproduce
• test_commit_channel_torrent https://github.com/Tribler/tribler/actions/runs/5543923045/jobs/10121257517?pr=7543#step:6:1272 (FREEZE) can not reproduce

metadata_store

➜ pytest src/tribler/core/components/metadata_store --flake-finder

==================================== short test summary info =====================================
FAILED src/tribler/core/components/metadata_store/tests/test_channel_download.py::test_channel_update_and_download[2] - Failed: Timeout >60.0s
================== 2 failed, 10848 passed, 26001 warnings in 1952.47s (0:32:32) ==================

Another issue with the same test:

src/tribler/core/components/metadata_store/tests/test_channel_download.py:58 (test_channel_update_and_download[26])
1565621688018 != 1565621688015

Expected :1565621688015
Actual   :1565621688018

libtorrent

➜ pytest src/tribler/core/components/libtorrent --flake-finder

A freeze was detected during the execution of test_move_to_non_existing_dir in test_downloads_endpoint.py. The test appears to freeze indefinitely:

restapi

➜ pytest src/tribler/core/components/restapi --flake-finder

======================================================== short test summary info ========================================================
FAILED src/tribler/core/components/restapi/rest/tests/test_events_endpoint.py::test_write_data[43] - assert 40 == 30
FAILED src/tribler/core/components/restapi/rest/tests/test_rest_manager.py::test_aiohttp_assertion_patched[17] - aiohttp.client_exceptions.ClientOSError: [Errno 54] Connection reset by peer
================================ 31 failed, 2869 passed, 50 skipped, 9957 warnings in 140.79s (0:02:20) =================================

tunnel

➜ pytest src/tribler/core/components/tunnel --flake-finder

======================================================== short test summary info ========================================================
FAILED src/tribler/core/components/tunnel/tests/test_triblertunnel_community.py::TestTriblerTunnelCommunity::test_payouts_e2e - AssertionError: False is not true
================================ 34 failed, 1966 passed, 100 skipped, 4202 warnings in 425.59s (0:07:05) ================================

core/tests

➜ pytest src/tribler/core/tests --flake-finder

======================================================== short test summary info ========================================================
FAILED src/tribler/core/tests/test_check_os.py::test_enable_fault_handler[49] - AssertionError: Expected 'enable' to have been called once. Called 2 times.
============================================ 49 failed, 1001 passed, 2107 warnings in 12.25s ============================================

core/utilities

➜ pytest src/tribler/core/utilities --flake-finder

======================================================== short test summary info ========================================================
FAILED src/tribler/core/utilities/tests/test_utilities.py::test_show_system_popup_win[8] - AssertionError: Expected 'MessageBox' to be called once. Called 2 times.
================================ 49 failed, 8901 passed, 50 skipped, 18000 warnings in 222.73s (0:03:42) ================================

This issue is related to #7132 and #7134

[drew2a]

The flakiness of test_payouts_e2e can only be observed when the tests are run repeatedly, around 15-20 times, within the same process.

[drew2a]

FAILED src/tribler/core/utilities/tests/test_pony_utils.py::test_patched_db_session 
AssertionError: Expected '_format_warning' to have been called.

Failed test_patched_db_session on win machine (cc @kozlovsky) in #7622

https://github.com/Tribler/tribler/actions/runs/6530502005/job/17729909354?pr=7622#step:6:2735

[drew2a]

FAILED src/tribler/core/components/libtorrent/restapi/tests/test_downloads_endpoint.py::test_change_hops
asyncio.exceptions.TimeoutError

FAILED test_change_hops on macOS machine in #7622
https://github.com/Tribler/tribler/actions/runs/6535191519/job/17744992894?pr=7622#step:8:2797

[drew2a]

Most of the flaky tests were fixed. The frozen tests detected during real PR checks are listed here: #7706