-
-
Notifications
You must be signed in to change notification settings - Fork 30
Caveat Usage with multiprocessing
Compare the broken code with the fixed code. You must manage the sys.path
when using multiprocessing
classes.
-
multiprocessing
allows Python to perform tasks faster by segregating work between multiple processes - Because any
multiprocessing
work occurs in a new process, the bootstrapping code for Python needs to copy over prerequisite data - As part of the bootstrapping process,
multiprocessing
will copy over thesys.path
at the time of instantiation - If you imported
bpy
before instantiating yourmultiprocessing
class (i.emultiprocessing.Pool
) you will get an error - The error is easily fixable by preserving the
sys.path
beforebpy
was imported (we will call thisORIG_SYS_PATH
) as well as thesys.path
afterbpy
was imported (we will call thisBPY_SYS_PATH
) and reverting it toORIG_SYS_PATH
just before the multiprocessing class instantiation, then reverting it back toBPY_SYS_PATH
before continuing more work in the main process
New processes spawned by a multiprocessing
class in code utilizing the bpy
Blender runtime are going to be flawed because of the design of the Blender runtime, and its reliance on the sys.path
being modified as part of its startup process. The flaw is, you are going to get an error when the instance of the multiprocessing
class tries to startup.
#!/usr/bin/python
# -*- coding: utf-8 -*-
"""A basic example program that exhibits the `bpy` and `multiprocessing` bug
Compatibility between these two libraries is strained by `bpy`'s own
requirement to append its user and addon modules directories to the `sys.path`
as well as the presence of a module in the addons directory called `bpy`,
resulting in an ambiguity between the legitimate `bpy` Blender runtime module
and the addon module the Blender 3d application requires
"""
# STD LIB imports
import multiprocessing
# EXTERNAL LIB IMPORTS
import bpy
def multiFunction(data):
print(data)
print(bpy)
if __name__ == '__main__' :
print(bpy) # do some `bpy` stuff here when in main process
data = ['Uno','Deux','Three']
p = multiprocessing.Pool()
p.map(multiFunction,data)
p.close()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
pkg_name=pkg_name, script_name=fname)
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
from _bpy import (
ModuleNotFoundError: No module named '_bpy'
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
exitcode = _main(fd)
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
exec(code, run_globals)
prepare(preparation_data)
File "c:\Users\TGubs\Code\Python\blender_test\tests\test_multiprocessing.py", line 6, in <module>
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
import bpy # Here, the sys.path is severely messed with, screws up the import
File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
_fixup_main_from_path(data['init_main_from_path'])
from _bpy import (
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
ModuleNotFoundError: No module named '_bpy'
run_name="__mp_main__")
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "c:\Users\TGubs\Code\Python\blender_test\tests\test_multiprocessing.py", line 6, in <module>
import bpy # Here, the sys.path is severely messed with, screws up the import
File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
from _bpy import (
ModuleNotFoundError: No module named '_bpy'
It can be difficult to debug this program with all of the processes concurrently writing to the same output, but luckily there is one giveaway that leads us to the root cause of the issue.
As part of the traceback we get something very telling:
File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
from _bpy import (
ModuleNotFoundError: No module named '_bpy'
What this tells us is that Python thinks that we should import modules from C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules
rather than the default of site-packages
. This is a giveaway that there was some messing around with the sys.path
prior to the instantiation of the multiprocessing.Pool
.
From observation we can find that bpy
does in fact prepend its paths to the sys.path
.
Microsoft Windows [Version 10.0.18362.239] (c) 2019 Microsoft Corporation. All rights reserved. C:\Users\TGubs>
cd Code/Python/blender_test
C:\Users\TGubs\Code\Python\blender_test>"venvs/3.6.8-32/Scripts/activate"
(3.6.8-32) C:\Users\TGubs\Code\Python\blender_test>py
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print("\n".join(sys.path))
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\python36.zip C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\DLLs C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\lib C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\lib\site-packages
>>> import bpy
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
>>> print("\n".join(sys.path))
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons_contrib C:\Users\TGubs\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\startup C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\python36.zip C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\DLLs C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\lib C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\lib\site-packages C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\freestyle\modules C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons\modules C:\Users\TGubs\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\modules
- We
import bpy
in the main process - This brings in the
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.832\Scripts\2.79\scripts\modules
to the top ofsys.path
because the Blender runtime needs it - The
multiprocessing.Pool
instantiation grabs our badsys.path
- Since modules aren't portable across the
multiprocessing.Queue
our new processes are going to try to import the modules again - Search along
sys.path
forbpy
- "I found one at
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules
!" - Error: can't locate or import its dependency
_bpy
(mostly because that is a Blender runtime convention that does not apply in Python) - Process setup fails repeatedly because of this
To fix this we will hold the ORIG_SYS_PATH
in a variable prior to bpy
getting imported. We will hold sys.path
prior to bpy
being imported in a variable called BPY_SYS_PATH
. We will revert the sys.path
to ORIG_SYS_PATH
prior to creating the multiprocessing.Pool
and revert sys.path
back to BPY_SYS_PATH
after the multiprocessing.Pool
has been instantiated, so we can continue to work with bpy
unimpeded in the main process.
#!/usr/bin/python
# -*- coding: utf-8 -*-
"""A basic example program that shows the `bpy` and `multiprocessing` fix
Compatibility between these two libraries is strained by `bpy`'s own
requirement to append its user and addon modules directories to the `sys.path`
as well as the presence of a module in the addons directory called `bpy`,
resulting in an ambiguity between the legitimate `bpy` Blender runtime module
and the addon module the Blender 3d application requires
"""
# STD LIB imports
import multiprocessing
import sys
ORIG_SYS_PATH = list(sys.path) # Make a new instance of sys.path
# EXTERNAL LIB IMPORTS
import bpy # Here, the sys.path is severely messed with, screws up the import
# in the new process that is created in multiprocessing.Pool()
BPY_SYS_PATH = list(sys.path) # Make instance of `bpy`'s modified sys.path
def multiFunction(data):
print(data)
print(bpy)
if __name__ == '__main__' :
print(bpy) # do some bpy stuff here when in main process
data = ['Uno','Deux','Three']
sys.path = ORIG_SYS_PATH # This way, the new process can import `bpy`
p = multiprocessing.Pool()
sys.path = BPY_SYS_PATH # this way, we can continue to use `bpy` in
# the main process
# do more stuff with bpy in main process
p.map(multiFunction,data)
p.close()
Please see #23