test_runner: automatically rerun flaky tests

[bayandin]

Describe your changes

This PR adds a plugin that automatically reruns (up to 3 times) flaky tests. Internally, it uses data from TEST_RESULT_CONNSTR database and pytest-rerunfailures plugin.

We think the test is flaky if it has different statuses (passed and failed) for the same revision for test runs on main branch.

Flaky tests are fetched by scripts/flaky_tests.py script (it's possible to use it in a standalone mode to learn which tests are flaky), stored to a JSON file and then the file is passed to the pytest plugin.

$ poetry run scripts/flaky_tests.py postgres://soft-base-394976.cloud.neon.tech:5432/main
connecting to the database...
fetching flaky tests...
	test_runner/performance/test_branch_creation.py::test_branch_creation_heavy_write[20]
	test_runner/regress/test_compatibility.py::test_forward_compatibility
	test_runner/regress/test_ondemand_download.py::test_ondemand_download_large_rel[real_s3]
	test_runner/regress/test_ondemand_download.py::test_ondemand_download_timetravel[real_s3]
	test_runner/regress/test_remote_storage.py::test_remote_storage_backup_and_restore[real_s3]
	test_runner/regress/test_tenant_detach.py::test_detach_while_attaching[real_s3]
	test_runner/regress/test_tenant_detach.py::test_tenant_reattach[real_s3]
	test_runner/regress/test_tenant_size.py::test_get_tenant_size_with_multiple_branches
	test_runner/regress/test_tenants.py::test_pageserver_with_empty_tenants[real_s3]
	test_runner/regress/test_wal_acceptor.py::test_wal_backup[real_s3]
saving results to flaky.json

Likely enough, the PR test run encountered such a flaky test, and it was run thrice: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-3880/debug/4542694447/index.html#categories/f08a7e35e427fe455a88f4ee0deda892/20612f8b60a53a90/retries

Issue ticket number and link

N/A

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

[arssher]

Hm, I'm not aware of test_wal_backup being flaky. Seems like this queries allure db runs on main? Is there a quick way to see those runs in allure?

[bayandin]

Hm, I'm not aware of test_wal_backup being flaky.

Taking into account that the most of flaky tests related to real_s3 it could've caused by the network or S3 itself

Seems like this queries allure db runs on main?

Yes, just to avoid experiments affecting the statistics

Is there a quick way to see those runs in allure?

https://neon-github-public-dev.s3.amazonaws.com/reports/main/debug/4539605341/index.html#suites/82004ab4e3720b47bf78f312dabe7c55/dad923cb652c5a2c/history

[koivunej]

Re: #3880 (comment)

Hm, I'm not aware of test_wal_backup being flaky.

Taking into account that the most of flaky tests related to real_s3 it could've caused by the network or S3 itself

I think we have a problem with many tests running into the 10s shutdown timeout with real_s3 while at the same time having a preceding wait_for_upload or similar with a larger timeout because there's a race that makes wait_for_upload (or similar) complete instantly, even though we will soon compact and need to upload a bunch of stuff. Context: #3697 (comment) but I could be wrong here just as well, had to context switch and have just been linking back to the comment :)

Yeah kind of makes me worried that we will miss out on issues like this, assuming it's a real issue. We would still see the retried tests in some dashboard?

[bayandin]

We would still see the retried tests in some dashboard?

Tests that we automatically reran will be posted to the PR comment (after #3907)
Later I'll add them to Grafana Dashboard