GH-84559: Deprecate fork being the multiprocessing default. #100618

gpshead · 2022-12-30T09:57:19Z

This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start().

Issue: multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' #84559

This starts the process. Users who don't specify their own start method and use the default on platforms where it is 'fork' will see a DeprecationWarning upon multiprocessing.Pool() construction or upon multiprocessing.Process.start().

gpshead · 2023-01-01T00:39:29Z

This code could be a lot cleaner if #39615 were implemented...

Doc/library/concurrent.futures.rst

gvanrossum

Changes to test_asyncio LGTM. I hope I won't have to review the rest, I know basically nothing about multiprocessing. :-(

kumaraditya303

I approve the asyncio change.

Lib/multiprocessing/context.py

hugovk · 2023-02-02T15:12:28Z

Please could you add this to What's New, under Pending Removal in Python 3.14?

gpshead · 2023-02-02T20:26:38Z

Please could you add this to What's New, under Pending Removal in Python 3.14?

Ahha, I knew we had a list of upcoming things somewhere. Good idea, done! What's New text added.

bedevere-bot · 2023-02-03T00:18:46Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (dynamic linking) 3.x has failed when building commit 0ca67e6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1056/builds/1286) and take a look at the build logs.
Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1056/builds/1286

Failed tests:

test_multiprocessing_defaults

Failed subtests:

test_no_mp_warning_when_using_explicit_fork_context - test.test_multiprocessing_defaults.DefaultWarningsTest.test_no_mp_warning_when_using_explicit_fork_context
test_default_fork_start_method_warning_pool - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_pool
test_default_fork_start_method_warning_manager - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_manager
test_default_fork_start_method_warning_process - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_process

Summary of the results of the build (if available):

== Tests result: FAILURE ==

335 tests OK.

10 slowest tests:

test_math: 1 min 57 sec
test_tokenize: 1 min 9 sec
test_unparse: 48.0 sec
test_lib2to3: 45.6 sec
test_capi: 27.1 sec
test_argparse: 20.3 sec
test_unicodedata: 18.7 sec
test_pickle: 17.7 sec
test_zipfile: 16.2 sec
test_decimal: 12.9 sec

1 test failed:
test_multiprocessing_defaults

98 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_poll
test_poplib test_pty test_pwd test_queue test_readline
test_regrtest test_repl test_resource test_select test_selectors
test_smtplib test_smtpnet test_socket test_socketserver test_spwd
test_ssl test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
test_winreg test_winsound test_wmi test_wsgiref test_xmlrpc
test_xmlrpc_net test_zipfile64 test_zipimport_support
test_zoneinfo
0:21:59 load avg: 11.21
0:21:59 load avg: 11.21 Re-running failed tests is not supported with --python host runner option.

Total duration: 21 min 59 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 34, in test_default_fork_start_method_warning_process
    process.start()  # warning should point here.
    ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 227, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 302, in _Popen
    return super()._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 285, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
               ^^^^^^^^^
OSError: [Errno 52] Function not implemented


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 47, in test_default_fork_start_method_warning_pool
    pool = multiprocessing.Pool(1)  # warning should point here.
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 75, in test_no_mp_warning_when_using_explicit_fork_context
    pool = fork_mp.Pool(1)
           ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/test/test_multiprocessing_defaults.py", line 61, in test_default_fork_start_method_warning_manager
    manager = multiprocessing.Manager()  # warning should point here.
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/context.py", line 58, in Manager
    from .managers import SyncManager
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/managers.py", line 29, in <module>
    from . import connection
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-dl/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'

bedevere-bot · 2023-02-03T00:23:45Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-emscripten node (pthreads) 3.x has failed when building commit 0ca67e6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1050/builds/1283) and take a look at the build logs.
Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1050/builds/1283

Failed tests:

test_multiprocessing_defaults

Failed subtests:

test_no_mp_warning_when_using_explicit_fork_context - test.test_multiprocessing_defaults.DefaultWarningsTest.test_no_mp_warning_when_using_explicit_fork_context
test_default_fork_start_method_warning_pool - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_pool
test_default_fork_start_method_warning_manager - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_manager
test_default_fork_start_method_warning_process - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_process

Summary of the results of the build (if available):

== Tests result: FAILURE ==

339 tests OK.

10 slowest tests:

test_math: 2 min 1 sec
test_tokenize: 1 min 11 sec
test_unparse: 47.9 sec
test_lib2to3: 43.0 sec
test_io: 39.3 sec
test_capi: 36.4 sec
test_unicodedata: 27.4 sec
test_zipfile: 25.3 sec
test_pickle: 22.0 sec
test_decimal: 19.5 sec

1 test failed:
test_multiprocessing_defaults

94 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_httplib test_httpservers test_idle test_imaplib
test_interpreters test_ioctl test_kqueue test_launcher test_lzma
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_poll
test_poplib test_pty test_pwd test_readline test_regrtest
test_repl test_resource test_select test_selectors test_smtplib
test_smtpnet test_socket test_socketserver test_spwd test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
test_winreg test_winsound test_wmi test_wsgiref test_xmlrpc
test_xmlrpc_net test_xxlimited test_zipfile64
test_zipimport_support test_zoneinfo
0:22:12 load avg: 7.44
0:22:12 load avg: 7.44 Re-running failed tests is not supported with --python host runner option.

Total duration: 22 min 12 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 75, in test_no_mp_warning_when_using_explicit_fork_context
    pool = fork_mp.Pool(1)
           ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 47, in test_default_fork_start_method_warning_pool
    pool = multiprocessing.Pool(1)  # warning should point here.
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 61, in test_default_fork_start_method_warning_manager
    manager = multiprocessing.Manager()  # warning should point here.
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 58, in Manager
    from .managers import SyncManager
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/managers.py", line 29, in <module>
    from . import connection
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/test/test_multiprocessing_defaults.py", line 34, in test_default_fork_start_method_warning_process
    process.start()  # warning should point here.
    ^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 227, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 302, in _Popen
    return super()._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/context.py", line 285, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/Lib/multiprocessing/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
               ^^^^^^^^^
OSError: [Errno 52] Function not implemented

bedevere-bot · 2023-02-03T00:26:12Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 0ca67e6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/1328) and take a look at the build logs.
Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/1328

Failed tests:

test_multiprocessing_defaults

Failed subtests:

test_no_mp_warning_when_using_explicit_fork_context - test.test_multiprocessing_defaults.DefaultWarningsTest.test_no_mp_warning_when_using_explicit_fork_context
test_default_fork_start_method_warning_pool - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_pool
test_default_fork_start_method_warning_manager - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_manager
test_default_fork_start_method_warning_process - test.test_multiprocessing_defaults.DefaultWarningsTest.test_default_fork_start_method_warning_process

Summary of the results of the build (if available):

== Tests result: FAILURE ==

326 tests OK.

10 slowest tests:

test_tokenize: 1 min 14 sec
test_math: 1 min 14 sec
test_lib2to3: 31.5 sec
test_capi: 31.4 sec
test_unparse: 29.1 sec
test_unicodedata: 22.7 sec
test_decimal: 19.1 sec
test_statistics: 18.5 sec
test_pickle: 13.4 sec
test_zipfile: 10.9 sec

1 test failed:
test_multiprocessing_defaults

107 tests skipped:
test__xxsubinterpreters test_asyncgen test_asyncio test_bz2
test_check_c_globals test_clinic test_cmd_line
test_concurrent_futures test_contextlib_async test_ctypes
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
test_docxmlrpc test_dtrace test_embed test_epoll test_faulthandler
test_fcntl test_file_eintr test_fork1 test_ftplib test_gdb
test_grp test_gzip test_httplib test_httpservers test_idle
test_imaplib test_interpreters test_ioctl test_kqueue
test_launcher test_lzma test_mailbox test_mmap test_msilib
test_multiprocessing_fork test_multiprocessing_forkserver
test_multiprocessing_main_handling test_multiprocessing_spawn
test_nis test_openpty test_ossaudiodev test_pdb test_peg_generator
test_perf_profiler test_pipes test_poll test_poplib test_pty
test_pwd test_queue test_readline test_regrtest test_repl
test_resource test_select test_selectors test_smtplib test_smtpnet
test_socket test_socketserver test_spwd test_sqlite3 test_ssl
test_stable_abi_ctypes test_startfile test_subprocess
test_sys_settrace test_syslog test_tcl test_telnetlib test_thread
test_threadedtempfile test_threading test_threading_local test_tix
test_tkinter test_tools test_ttk test_ttk_textonly test_turtle
test_urllib test_urllib2 test_urllib2_localnet test_urllib2net
test_urllib_response test_urllibnet test_venv test_wait3
test_wait4 test_webbrowser test_winconsoleio test_winreg
test_winsound test_wmi test_wsgiref test_xmlrpc test_xmlrpc_net
test_xxlimited test_zipfile64 test_zipimport_support test_zlib
test_zoneinfo
0:05:45 load avg: 3.51
0:05:45 load avg: 3.51 Re-running failed tests is not supported with --python host runner option.

Total duration: 5 min 45 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 47, in test_default_fork_start_method_warning_pool
    pool = multiprocessing.Pool(1)  # warning should point here.
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 34, in test_default_fork_start_method_warning_process
    process.start()  # warning should point here.
    ^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 227, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 302, in _Popen
    return super()._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 284, in _Popen
    from .popen_fork import Popen
  File "/Lib/multiprocessing/popen_fork.py", line 12, in <module>
    class Popen(object):
  File "/Lib/multiprocessing/popen_fork.py", line 24, in Popen
    def poll(self, flag=os.WNOHANG):
                        ^^^^^^^^^^
AttributeError: module 'os' has no attribute 'WNOHANG'


Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 61, in test_default_fork_start_method_warning_manager
    manager = multiprocessing.Manager()  # warning should point here.
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 58, in Manager
    from .managers import SyncManager
  File "/Lib/multiprocessing/managers.py", line 29, in <module>
    from . import connection
  File "/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'


Traceback (most recent call last):
  File "/Lib/test/test_multiprocessing_defaults.py", line 75, in test_no_mp_warning_when_using_explicit_fork_context
    pool = fork_mp.Pool(1)
           ^^^^^^^^^^^^^^^
  File "/Lib/multiprocessing/context.py", line 121, in Pool
    from .pool import Pool
  File "/Lib/multiprocessing/pool.py", line 30, in <module>
    from .connection import wait
  File "/Lib/multiprocessing/connection.py", line 21, in <module>
    import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'

bedevere-bot · 2023-02-03T00:26:36Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 FIPS Only Blake2 Builtin Hash 3.x has failed when building commit 0ca67e6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/469/builds/4156) and take a look at the build logs.
Check if the failure is related to this commit (0ca67e6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/469/builds/4156

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

417 tests OK.

10 slowest tests:

test_gdb: 7 min
test_concurrent_futures: 2 min 49 sec
test_asyncio: 2 min 28 sec
test_capi: 2 min 18 sec
test_multiprocessing_spawn: 2 min 11 sec
test_math: 1 min 49 sec
test_tokenize: 1 min 18 sec
test_multiprocessing_forkserver: 1 min 16 sec
test_multiprocessing_fork: 1 min 11 sec
test_subprocess: 50.6 sec

1 test altered the execution environment:
test_multiprocessing_defaults

16 tests skipped:
test_check_c_globals test_devpoll test_ioctl test_kqueue
test_launcher test_msilib test_peg_generator test_startfile
test_tix test_tkinter test_ttk test_winconsoleio test_winreg
test_winsound test_wmi test_zipfile64

Total duration: 13 min 33 sec

Click to see traceback logs

remote: Enumerating objects: 64, done.        
remote: Counting objects:   1% (1/64)        
remote: Counting objects:   3% (2/64)        
remote: Counting objects:   4% (3/64)        
remote: Counting objects:   6% (4/64)        
remote: Counting objects:   7% (5/64)        
remote: Counting objects:   9% (6/64)        
remote: Counting objects:  10% (7/64)        
remote: Counting objects:  12% (8/64)        
remote: Counting objects:  14% (9/64)        
remote: Counting objects:  15% (10/64)        
remote: Counting objects:  17% (11/64)        
remote: Counting objects:  18% (12/64)        
remote: Counting objects:  20% (13/64)        
remote: Counting objects:  21% (14/64)        
remote: Counting objects:  23% (15/64)        
remote: Counting objects:  25% (16/64)        
remote: Counting objects:  26% (17/64)        
remote: Counting objects:  28% (18/64)        
remote: Counting objects:  29% (19/64)        
remote: Counting objects:  31% (20/64)        
remote: Counting objects:  32% (21/64)        
remote: Counting objects:  34% (22/64)        
remote: Counting objects:  35% (23/64)        
remote: Counting objects:  37% (24/64)        
remote: Counting objects:  39% (25/64)        
remote: Counting objects:  40% (26/64)        
remote: Counting objects:  42% (27/64)        
remote: Counting objects:  43% (28/64)        
remote: Counting objects:  45% (29/64)        
remote: Counting objects:  46% (30/64)        
remote: Counting objects:  48% (31/64)        
remote: Counting objects:  50% (32/64)        
remote: Counting objects:  51% (33/64)        
remote: Counting objects:  53% (34/64)        
remote: Counting objects:  54% (35/64)        
remote: Counting objects:  56% (36/64)        
remote: Counting objects:  57% (37/64)        
remote: Counting objects:  59% (38/64)        
remote: Counting objects:  60% (39/64)        
remote: Counting objects:  62% (40/64)        
remote: Counting objects:  64% (41/64)        
remote: Counting objects:  65% (42/64)        
remote: Counting objects:  67% (43/64)        
remote: Counting objects:  68% (44/64)        
remote: Counting objects:  70% (45/64)        
remote: Counting objects:  71% (46/64)        
remote: Counting objects:  73% (47/64)        
remote: Counting objects:  75% (48/64)        
remote: Counting objects:  76% (49/64)        
remote: Counting objects:  78% (50/64)        
remote: Counting objects:  79% (51/64)        
remote: Counting objects:  81% (52/64)        
remote: Counting objects:  82% (53/64)        
remote: Counting objects:  84% (54/64)        
remote: Counting objects:  85% (55/64)        
remote: Counting objects:  87% (56/64)        
remote: Counting objects:  89% (57/64)        
remote: Counting objects:  90% (58/64)        
remote: Counting objects:  92% (59/64)        
remote: Counting objects:  93% (60/64)        
remote: Counting objects:  95% (61/64)        
remote: Counting objects:  96% (62/64)        
remote: Counting objects:  98% (63/64)        
remote: Counting objects: 100% (64/64)        
remote: Counting objects: 100% (64/64), done.        
remote: Compressing objects:   3% (1/32)        
remote: Compressing objects:   6% (2/32)        
remote: Compressing objects:   9% (3/32)        
remote: Compressing objects:  12% (4/32)        
remote: Compressing objects:  15% (5/32)        
remote: Compressing objects:  18% (6/32)        
remote: Compressing objects:  21% (7/32)        
remote: Compressing objects:  25% (8/32)        
remote: Compressing objects:  28% (9/32)        
remote: Compressing objects:  31% (10/32)        
remote: Compressing objects:  34% (11/32)        
remote: Compressing objects:  37% (12/32)        
remote: Compressing objects:  40% (13/32)        
remote: Compressing objects:  43% (14/32)        
remote: Compressing objects:  46% (15/32)        
remote: Compressing objects:  50% (16/32)        
remote: Compressing objects:  53% (17/32)        
remote: Compressing objects:  56% (18/32)        
remote: Compressing objects:  59% (19/32)        
remote: Compressing objects:  62% (20/32)        
remote: Compressing objects:  65% (21/32)        
remote: Compressing objects:  68% (22/32)        
remote: Compressing objects:  71% (23/32)        
remote: Compressing objects:  75% (24/32)        
remote: Compressing objects:  78% (25/32)        
remote: Compressing objects:  81% (26/32)        
remote: Compressing objects:  84% (27/32)        
remote: Compressing objects:  87% (28/32)        
remote: Compressing objects:  90% (29/32)        
remote: Compressing objects:  93% (30/32)        
remote: Compressing objects:  96% (31/32)        
remote: Compressing objects: 100% (32/32)        
remote: Compressing objects: 100% (32/32), done.        
remote: Total 36 (delta 30), reused 7 (delta 4), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '0ca67e6313c11263ecaef7ce182308eeb5aa6814'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 0ca67e6313 GH-84559: Deprecate fork being the multiprocessing default. (#100618)
Switched to and reset branch 'main'

make: *** [Makefile:1912: buildbottest] Error 3

This reverts the core of python#100618 while leaving relevant documentation improvements and minor refactorings in place.

…01551) This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.

The default of `fork` is known to be problematic. Python itself is changing the default to `spawn`. The new default is expected to be in place for Python 3.14. Python references for the change to the default: * python/cpython#84559 * python/cpython#100618 We also have several places where this option had to be set to `spawn` to make tests work. The AMD code even checks and overrides the value if it's not set to `spawn`. Simplify things for everyone and just default to `spawn`, but leave the option in place just in case, at least for now. Signed-off-by: Russell Bryant <[email protected]>

gpshead added the topic-multiprocessing label Dec 30, 2022

bedevere-bot mentioned this pull request Dec 30, 2022

multiprocessing's default posix start method of 'fork' is broken: change to `'forkserver' || 'spawn' #84559

Closed

bedevere-bot added the awaiting core review label Dec 30, 2022

gpshead self-assigned this Dec 30, 2022

gpshead added 13 commits December 31, 2022 04:10

address CI test failures?

df83fba

the new warning needs no pickle name mapping

ec1e868

avoid DeprecationWarnings vs -Werror

a1064de

prefer forkserver over fork for concurrent.futures

223b748

explicitly use spawn for simple process launchers.

fc64626

warn from concurrent.futures, reword things.

e45fab6

avoid double warning in futures

29764a2

Cleanup messages, test futures warning.

c52b9db

cleanup warn stacklevel logic

4c8cc34

Document all the things!

7179684

Have all doctests use 'spawn'.

f7e3161

Correct warning stacklevel for Manager.

9947860

more doctest cleanup.

5834f2d

NEWS entry.

7d48e86

FFY00 reviewed Jan 9, 2023

View reviewed changes

Doc/library/concurrent.futures.rst Outdated Show resolved Hide resolved

docs: add the implicit i

28c0cfc

gpshead mentioned this pull request Jan 21, 2023

gh-39615: Add warnings.warn() skip_file_prefixes support #100840

Merged

gpshead added 4 commits January 27, 2023 18:36

Merge branch 'main' into multiprocessing/deprecate-fork-default

2820bb8

Use skip_file_prefixes.

4c16205

use the modern warnings API, doc updates.

7d84a72

documentation cleanup

8b2029e

gpshead added the stdlib Python modules in the Lib dir label Jan 28, 2023

gpshead requested a review from Yhg1s January 28, 2023 04:28

gpshead marked this pull request as ready for review January 28, 2023 04:28

gpshead requested review from vsajip, 1st1, asvetlov, gvanrossum and kumaraditya303 as code owners January 28, 2023 04:28

gvanrossum reviewed Jan 28, 2023

View reviewed changes

kumaraditya303 removed their request for review January 31, 2023 16:53

kumaraditya303 reviewed Jan 31, 2023

View reviewed changes

Yhg1s approved these changes Feb 1, 2023

View reviewed changes

Lib/multiprocessing/context.py Outdated Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting core review labels Feb 1, 2023

gpshead added 2 commits February 1, 2023 14:40

Remove an unused variable.

846e9a0

doc typo fix, unnecessary method change revert.

7078986

Add What's New entries.

e3e3b3c

gpshead merged commit 0ca67e6 into python:main Feb 2, 2023

bedevere-bot removed the awaiting merge label Feb 2, 2023

gpshead deleted the multiprocessing/deprecate-fork-default branch February 2, 2023 23:50

gpshead added a commit to gpshead/cpython that referenced this pull request Feb 3, 2023

pythongh-84559: Remove the new mp warning, too disruptive.

20cec00

This reverts the core of python#100618 while leaving relevant documentation improvements and minor refactorings in place.

gpshead mentioned this pull request Feb 3, 2023

gh-84559: Remove the new multiprocessing warning, too disruptive. #101551

Merged

gpshead added a commit that referenced this pull request Feb 3, 2023

gh-84559: Remove the new multiprocessing warning, too disruptive. (#1…

d4c410f

…01551) This reverts the core of #100618 while leaving relevant documentation improvements and minor refactorings in place.

russellb mentioned this pull request Sep 18, 2024

[Core] Change multiproc method default to spawn vllm-project/vllm#8576

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-84559: Deprecate fork being the multiprocessing default. #100618

GH-84559: Deprecate fork being the multiprocessing default. #100618

gpshead commented Dec 30, 2022 •

edited by bedevere-bot

Loading

gpshead commented Jan 1, 2023

gvanrossum left a comment

kumaraditya303 left a comment

hugovk commented Feb 2, 2023

gpshead commented Feb 2, 2023

bedevere-bot commented Feb 3, 2023

bedevere-bot commented Feb 3, 2023

bedevere-bot commented Feb 3, 2023

bedevere-bot commented Feb 3, 2023

GH-84559: Deprecate fork being the multiprocessing default. #100618

GH-84559: Deprecate fork being the multiprocessing default. #100618

Conversation

gpshead commented Dec 30, 2022 • edited by bedevere-bot Loading

gpshead commented Jan 1, 2023

gvanrossum left a comment

Choose a reason for hiding this comment

kumaraditya303 left a comment

Choose a reason for hiding this comment

hugovk commented Feb 2, 2023

gpshead commented Feb 2, 2023

bedevere-bot commented Feb 3, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Feb 3, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Feb 3, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Feb 3, 2023

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gpshead commented Dec 30, 2022 •

edited by bedevere-bot

Loading