unittest: execute tests in parallel

[donpellegrino]

BPO	37873
Nosy	@terryjreedy, @giampaolo, @ezio-melotti, @voidspace, @bharel, @tirkarthi, @donpellegrino

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-08-16.14:15:27.971>
labels = ['type-feature', 'library', '3.9']
title = 'unittest: execute tests in parallel'
updated_at = <Date 2020-09-02.09:57:36.866>
user = 'https://github.com/donpellegrino'

bugs.python.org fields:

activity = <Date 2020-09-02.09:57:36.866>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-08-16.14:15:27.971>
creator = 'user93448'
dependencies = []
files = []
hgrepos = []
issue_num = 37873
keywords = []
message_count = 5.0
messages = ['349864', '349938', '350080', '375126', '375128']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'giampaolo.rodola', 'ezio.melotti', 'michael.foord', 'bar.harel', 'xtreak', 'user93448']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37873'
versions = ['Python 3.9']

Linked PRs

• gh-82054: Implements test.regrtest unittest sharding to pull in long tails #99637
• gh-82054: allow test runner to split test_asyncio #103859
• gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. #103927
• gh-82054: Parallel test sharding of test_asyncio and test_multiprocessing #103934

[donpellegrino]

The unittest documentation makes reference to a potential parallelization feature:

"Note that shared fixtures do not play well with [potential] features like test parallelization and they break test isolation. They should be used with care." (https://docs.python.org/3/library/unittest.html)

However, it seems that executing tests in parallel is not yet a feature of unittest. This enhancement request is to add parallel execution of tests to unittest.

A command line option may be a good interface. Ideally, it would be compatible with test discovery. Outside of the Python ecosystem, a common practice is to define test cases in a Makefile and then execute GNU Make with the '-j' flag (https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel). Adding such an option to unittest would be a convenience and may save the effort of bringing in additional libraries or tools for parallel unit test execution.

[terryjreedy]

test.regrtest has a -j option. Perhaps some of the Python coding for that could be used for unitest also.

[tirkarthi]

See also https://mail.python.org/pipermail/python-ideas/2017-September/047100.html . One of the ideas in the thread was to move test.regrtest parallel execution functionality into unittest. I think this would be good to have it in unittest like support in pytest for -j.

[donpellegrino]

Leveraging GNU Parallel (https://www.gnu.org/software/parallel/) might help simplify implementation. Perhaps that could be used as a subprocess call?

[vstinner]

Leveraging GNU Parallel (https://www.gnu.org/software/parallel/) might help simplify implementation. Perhaps that could be used as a subprocess call?

In general, we attempt to avoid depending on the availability of external tool. For example, I don't expect this tool to be available on Windows, whereas it would be better to support parallel execution on Windows as well.

[gpshead]

An example of implementing sharding within a single unittest so they can run in parallel across whatever number of processes and machines you want is what we use at work: https://github.com/abseil/abseil-py/blob/v1.3.0/absl/testing/absltest.py#L2359 using the Bazel sharding protocol.

If test.regrtest supported using such a sharding protocol and unittest adopted that TestCase.getTestCaseNames implementation logic to compute the shards for each process, we'd really pull in our long tail test time in CPython's own unittest suite.

One of the ideas in the thread was to move test.regrtest parallel execution functionality into unittest. I think this would be good to have it in unittest like support in pytest for -j.

Yes. Though wider Python community wise you'll just find many people saying "just use pytest-xdist" in your project for -j support. I can't disagree with that.

I still want parallelization for CPython's own unittest suite (launched via python -m test) regardless of that. It'd help save core dev & contributor time as well as increasing buildbot and CI thruput.

[gpshead]

I'm leaving this open as there's plenty more opportunity to do things here (up to and including potentially making my full parallelism PR work reliably). The just merged PR at least splits our longest test, test_asyncio, up into parallel sub-tests given it was already defined as a package of tests in separate files.

Refactoring test_concurrent_futures.py and the entire _test_multiprocessing.py suite and its trio of start-method runners into similar package of sub-test.py files would allow it to use the already committed pieces upon make test or python -m test when no list of tests is specified to further pull in the long tail on default run-everything CI, buildbot, and developer runs.

[gpshead]

With the refactorings @vstinner has done recently, we're close enough that I'm going to just close this issue. See my now closed draft PR #99637 for the full parallelism implementation I started with and reasoning for why it likely isn't worth us chasing at this point.

Python users who want that parallelism in their own test suites can use absltest from https://pypi.org/project/absl-py/ or other frameworks (pytest-xdist?) that offer it.