Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-108388: Convert test_concurrent_futures to package #108401

Merged
merged 4 commits into from
Aug 24, 2023

Conversation

vstinner
Copy link
Member

@vstinner vstinner commented Aug 24, 2023

Convert test_concurrent_futures to a package of 7 sub-tests.

Add remote_globals to create_executor_tests()

Convert test_concurrent_futures to a package of 7 sub-tests.

Add remote_globals to create_executor_tests()
@vstinner
Copy link
Member Author

Timing of running tests: ./python -m test test_concurrent_futures -j0 --slowest on a machine with 12 threads / 6 CPU cores.

  • reference: 2 min 43 sec
  • PR: 1 min 13 sec

With the PR:

10 slowest tests:
- test_concurrent_futures.test_wait: 1 min 13 sec
- test_concurrent_futures.test_process_pool: 42.6 sec
- test_concurrent_futures.test_shutdown: 20.1 sec
- test_concurrent_futures.test_as_completed: 12.0 sec
- test_concurrent_futures.test_deadlock: 11.1 sec
- test_concurrent_futures.test_future: 3.2 sec
- test_concurrent_futures.test_init: 2.3 sec

Copy link
Member

@sobolevn sobolevn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To fix the CI

Lib/test/test_concurrent_futures/test_shutdown.py Outdated Show resolved Hide resolved
@corona10 corona10 self-requested a review August 24, 2023 14:05
@vstinner
Copy link
Member Author

GHA job timings:

  • Windows x86: 24 min 37 sec
  • Windows x64: 19 min 27 sec
  • macOS: 18 min 22 sec
  • Ubuntu: 10 min
  • Address Sanitizer: 14 min 49 sec

Details.

Windows x86:

10 slowest tests:
- test_importlib: 3 min 39 sec
- test.test_multiprocessing_spawn.test_manager: 2 min 49 sec
- test_compileall: 2 min 47 sec
- test_venv: 2 min 33 sec
- test_tarfile: 2 min 28 sec
- test.test_multiprocessing_spawn.test_processes: 2 min 10 sec
- test_threading: 2 min 2 sec
- test_fstring: 1 min 56 sec
- test_queue: 1 min 52 sec
- test_socket: 1 min 43 sec

Windows x64:

10 slowest tests:
- test.test_multiprocessing_spawn.test_processes: 3 min 18 sec
- test.test_multiprocessing_spawn.test_manager: 2 min 36 sec
- test_regrtest: 2 min 27 sec
- test_importlib: 2 min 6 sec
- test_compileall: 2 min 5 sec
- test_venv: 2 min 2 sec
- test_zipfile: 1 min 57 sec
- test_io: 1 min 47 sec
- test_tarfile: 1 min 45 sec
- test_largefile: 1 min 38 sec

macOS:

10 slowest tests:
- test.test_multiprocessing_spawn.test_processes: 3 min 20 sec
- test_tarfile: 2 min 48 sec
- test.test_multiprocessing_forkserver.test_processes: 2 min 19 sec
- test_pickle: 1 min 58 sec
- test_cppext: 1 min 40 sec
- test_statistics: 1 min 39 sec
- test_compileall: 1 min 25 sec
- test.test_concurrent_futures.test_wait: 1 min 24 sec
- test_subprocess: 1 min 22 sec
- test.test_multiprocessing_spawn.test_misc: 1 min 21 sec

Ubuntu:

10 slowest tests:
- test_gdb: 2 min 23 sec
- test.test_multiprocessing_spawn.test_processes: 1 min 31 sec
- test.test_concurrent_futures.test_wait: 1 min 22 sec
- test_signal: 1 min 2 sec
- test.test_multiprocessing_forkserver.test_processes: 1 min 1 sec
- test_subprocess: 1 min
- test_weakref: 44.6 sec
- test_venv: 44.2 sec
- test_socket: 42.1 sec
- test.test_concurrent_futures.test_process_pool: 41.4 sec

Address Sanitizer:

10 slowest tests:
- test_subprocess: 2 min 11 sec
- test_compileall: 2 min 9 sec
- test_venv: 2 min
- test_gdb: 1 min 39 sec
- test_weakref: 1 min 32 sec
- test_tarfile: 1 min 26 sec
- test_regrtest: 1 min 12 sec
- test_source_encoding: 1 min 11 sec
- test_multiprocessing_main_handling: 1 min 1 sec
- test_decimal: 1 min

Copy link
Member

@gpshead gpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you unintentionally got rid of most of the Threading tests.

(do not merge this even after fixing up this review round's comments, this requires further very close side by side review that Github's UI is incapable of providing as it lacks the ability to track file splits and moves as a diff of what's changed -- I'll need to spend time on the command line in a client looking things over)

Otherwise: Thank you for taking this on! This test suite has needed refactoring into something manageable forever.

@bedevere-bot
Copy link

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

@gpshead gpshead self-assigned this Aug 24, 2023
@gpshead gpshead marked this pull request as draft August 24, 2023 15:34
@gpshead
Copy link
Member

gpshead commented Aug 24, 2023

I moved this PR to Draft state to signal that I don't want anyone to think it is ready to merge even if someone approves it until after we've taken a close look to verify that we haven't lost anything in the refactoring.

@vstinner
Copy link
Member Author

I believe you unintentionally got rid of most of the Threading tests.

I compared ./python -m test test_concurrent_futures --list-cases|wc -l with and without my change: I get 248 test cases in both cases.

@vstinner
Copy link
Member Author

For your comments like "ThreadPoolExecutorTest is defined above but missing here" on test_process_pool.py, I suggest you looking at which tests are executed rather than only looking at the code. For example:

$ ./python -m test test_concurrent_futures --list-cases|grep ThreadPoolExecutorTest
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_default_workers
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_executor_map_current_future_cancel
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_free_reference
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_hang_global_shutdown_lock
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_idle_thread_reuse
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_exception
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_submits_without_iteration
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_timeout
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_max_workers_negative
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_no_stale_references
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_saturation
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_shutdown_race_issue12456
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_submit
test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_submit_keyword

You can compare between the main branch and my PR: I get the same 15 tests, but with my PR, test names also get an additional test_process_pool. in their name.

@vstinner
Copy link
Member Author

I splitted test_process_pool in two testrs:

  • test_thread_pool: 15 tests, takes 10.6 sec
  • test_process_pool: 60 tests, takes 32.0 sec

Before, test_process_poll was the sum (75 tests, ~42 sec) 🙂.

@gpshead
Copy link
Member

gpshead commented Aug 24, 2023

Thanks. confirmed locally as well, all tests are present and accounted for!

@gpshead gpshead marked this pull request as ready for review August 24, 2023 17:08
@vstinner vstinner merged commit aa6f787 into python:main Aug 24, 2023
20 checks passed
@vstinner vstinner deleted the test_concurrent_package branch August 24, 2023 17:21
@vstinner
Copy link
Member Author

Thanks for the review @gpshead! For sure, there is room for improvement for these tests. I hope that this split will ease future maintenance.

@gpshead gpshead added the needs backport to 3.12 bug and security fixes label Aug 24, 2023
@miss-islington
Copy link
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

@miss-islington
Copy link
Contributor

Sorry, @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker aa6f787faa4bc45006da4dc2f942fb9b82c98836 3.12

@miss-islington miss-islington assigned vstinner and unassigned gpshead Aug 24, 2023
@bedevere-bot
Copy link

GH-108443 is a backport of this pull request to the 3.12 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.12 bug and security fixes label Aug 24, 2023
@gpshead
Copy link
Member

gpshead commented Aug 24, 2023

the merge conflict on this is probably just the list in regrtest runtests.py. retry that automation after the test_multiprocessing one is in 3.12.

vstinner added a commit to vstinner/cpython that referenced this pull request Aug 24, 2023
…08401)

Convert test_concurrent_futures to a package of sub-tests.

(cherry picked from commit aa6f787)
@vstinner
Copy link
Member Author

the merge conflict on this is probably just the list in regrtest runtests.py. retry that automation after the test_multiprocessing one is in 3.12.

Right. I backported the change to Python 3.12. There is a minor conflict in libregrtest on:

 SPLITTESTDIRS = {
     "test_asyncio",
+    "test_concurrent_futures",
 }

The backport is in conflict with PR #108401 (split multiprocessing tests) which may be merged first.

Yhg1s pushed a commit that referenced this pull request Aug 26, 2023
#108443)

gh-108388: Convert test_concurrent_futures to package (#108401)

Convert test_concurrent_futures to a package of sub-tests.

(cherry picked from commit aa6f787)
@bedevere-app
Copy link

bedevere-app bot commented Sep 22, 2023

GH-109704 is a backport of this pull request to the 3.11 branch.

@vstinner
Copy link
Member Author

vstinner commented Sep 22, 2023

Since I was asked to backport all test changes (including refactoring) to stable branches, I backport this change to Python 3.11: GH-109704.

vstinner added a commit that referenced this pull request Sep 22, 2023
#109704)

* gh-108388: Convert test_concurrent_futures to package (#108401)

Convert test_concurrent_futures to a package of sub-tests.

(cherry picked from commit aa6f787)

Notes on backport to 3.11:

* AsCompletedTests: Revert test_future_times_out() => test_zero_timeout()
* Restore TODO comment
* ThreadPoolExecutorTest.test_hang_global_shutdown_lock():
  add @support.requires_resource('cpu').
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants