Skip to content

Caveat Usage with multiprocessing

Jump to bottom
Tyler Alden Gubala edited this page Aug 10, 2019 · 3 revisions

Too Long; Didn't Read

Compare the broken code with the fixed code. You must manage the sys.path when using multiprocessing classes.

Overview

  • multiprocessing allows Python to perform tasks faster by segregating work between multiple processes
  • Because any multiprocessing work occurs in a new process, the bootstrapping code for Python needs to copy over prerequisite data
  • As part of the bootstrapping process, multiprocessing will copy over the sys.path at the time of instantiation
  • If you imported bpy before instantiating your multiprocessing class (i.e multiprocessing.Pool) you will get an error
  • The error is easily fixable by preserving the sys.path before bpy was imported (we will call this ORIG_SYS_PATH) as well as the sys.path after bpy was imported (we will call this BPY_SYS_PATH) and reverting it to ORIG_SYS_PATH just before the multiprocessing class instantiation, then reverting it back to BPY_SYS_PATH before continuing more work in the main process

The caveat

New processes spawned by a multiprocessing class in code utilizing the bpy Blender runtime are going to be flawed because of the design of the Blender runtime, and its reliance on the sys.path being modified as part of its startup process. The flaw is, you are going to get an error when the instance of the multiprocessing class tries to startup.

Code without the fix

#!/usr/bin/python
# -*- coding: utf-8 -*-
"""A basic example program that exhibits the `bpy` and `multiprocessing` bug

Compatibility between these two libraries is strained by `bpy`'s own
requirement to append its user and addon modules directories to the `sys.path`
as well as the presence of a module in the addons directory called `bpy`,
resulting in an ambiguity between the legitimate `bpy` Blender runtime module
and the addon module the Blender 3d application requires
"""
# STD LIB imports
import multiprocessing
# EXTERNAL LIB IMPORTS
import bpy

def multiFunction(data):
    print(data)
    print(bpy)

if __name__ == '__main__' :
    print(bpy) # do some `bpy` stuff here when in main process

    data = ['Uno','Deux','Three']

    p = multiprocessing.Pool()

    p.map(multiFunction,data)
    p.close()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
    from _bpy import (
ModuleNotFoundError: No module named '_bpy'
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code       
    exitcode = _main(fd)
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 114, in _main
    exec(code, run_globals)
    prepare(preparation_data)
  File "c:\Users\TGubs\Code\Python\blender_test\tests\test_multiprocessing.py", line 6, in <module>      
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 225, in prepare
    import bpy # Here, the sys.path is severely messed with, screws up the import
  File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
    _fixup_main_from_path(data['init_main_from_path'])
    from _bpy import (
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
ModuleNotFoundError: No module named '_bpy'
    run_name="__mp_main__")
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 263, in run_path       
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\TGubs\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code       
    exec(code, run_globals)
  File "c:\Users\TGubs\Code\Python\blender_test\tests\test_multiprocessing.py", line 6, in <module>
    import bpy # Here, the sys.path is severely messed with, screws up the import
  File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
    from _bpy import (
ModuleNotFoundError: No module named '_bpy'

It can be difficult to debug this program with all of the processes concurrently writing to the same output, but luckily there is one giveaway that leads us to the root cause of the issue.

As part of the traceback we get something very telling:

File "C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules\bpy\__init__.py", line 38, in <module>
    from _bpy import (
ModuleNotFoundError: No module named '_bpy'

What this tells us is that Python thinks that we should import modules from C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\Scripts\2.79\scripts\modules rather than the default of site-packages. This is a giveaway that there was some messing around with the sys.path prior to the instantiation of the multiprocessing.Pool.

Debugging the code

From observation we can find that bpy does in fact prepend its paths to the sys.path.

Microsoft Windows [Version 10.0.18362.239] (c) 2019 Microsoft Corporation. All rights reserved. C:\Users\TGubs>cd Code/Python/blender_test C:\Users\TGubs\Code\Python\blender_test>"venvs/3.6.8-32/Scripts/activate" (3.6.8-32) C:\Users\TGubs\Code\Python\blender_test>py Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> print("\n".join(sys.path))

C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\python36.zip C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\DLLs C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\lib C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\lib\site-packages

>>> import bpy

AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead

>>> print("\n".join(sys.path))

C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons_contrib C:\Users\TGubs\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\startup C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\python36.zip C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\DLLs C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\lib C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32 C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\lib\site-packages C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\freestyle\modules C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons\modules C:\Users\TGubs\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\modules

Steps in the unfixed code

  1. We import bpy in the main process
  2. This brings in the C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.832\Scripts\2.79\scripts\modules to the top of sys.path because the Blender runtime needs it
  3. The multiprocessing.Pool instantiation grabs our bad sys.path
  4. Since modules aren't portable across the multiprocessing.Queue our new processes are going to try to import the modules again
  5. Search along sys.path for bpy
  6. "I found one at C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules!"
  7. Error: can't locate or import its dependency _bpy (mostly because that is a Blender runtime convention that does not apply in Python)
  8. Process setup fails repeatedly because of this

What is the fix?

To fix this we will hold the ORIG_SYS_PATH in a variable prior to bpy getting imported. We will hold sys.path prior to bpy being imported in a variable called BPY_SYS_PATH. We will revert the sys.path to ORIG_SYS_PATH prior to creating the multiprocessing.Pool and revert sys.path back to BPY_SYS_PATH after the multiprocessing.Pool has been instantiated, so we can continue to work with bpy unimpeded in the main process.

Fixed Code

#!/usr/bin/python
# -*- coding: utf-8 -*-
"""A basic example program that shows the `bpy` and `multiprocessing` fix

Compatibility between these two libraries is strained by `bpy`'s own
requirement to append its user and addon modules directories to the `sys.path`
as well as the presence of a module in the addons directory called `bpy`,
resulting in an ambiguity between the legitimate `bpy` Blender runtime module
and the addon module the Blender 3d application requires
"""
# STD LIB imports
import multiprocessing
import sys

ORIG_SYS_PATH = list(sys.path) # Make a new instance of sys.path
# EXTERNAL LIB IMPORTS
import bpy # Here, the sys.path is severely messed with, screws up the import 
           # in the new process that is created in multiprocessing.Pool()

BPY_SYS_PATH = list(sys.path) # Make instance of `bpy`'s modified sys.path

def multiFunction(data):
    print(data)
    print(bpy)

if __name__ == '__main__' :
    print(bpy) # do some bpy stuff here when in main process

    data = ['Uno','Deux','Three']

    sys.path = ORIG_SYS_PATH # This way, the new process can import `bpy`

    p = multiprocessing.Pool()

    sys.path = BPY_SYS_PATH # this way, we can continue to use `bpy` in 
                            # the main process

    # do more stuff with bpy in main process

    p.map(multiFunction,data)
    p.close()

Related reading

Please see #23

Clone this wiki locally