ImportError: No module named '_bpy' When Pickling #23

HCAWN · 2019-07-23T09:05:05Z

Describe the bug
I have a script that splits in to a multiprocess.pool(). I require the bpy module both before the pool and inside each thread in the pool. I can get bpy to function outside or inside but not both.

There are three things main methods I've tried:

import bpy at the start of the script outside of if __name__ == '__main__' :
this causes ImportError: No module named '_bpy' - I presume because it's trying to import bpy multiple times
import bpy inside of if __name__ == '__main__' :
this works for outside the pool but if I want to parse the module into the pool, I have to pickle it via dill and then unpickle it inside each thread which also results in ImportError: No module named '_bpy'
import bpy inside of if __name__ == '__main__' : and then inside each thread:
same error comes up as above, once again I think this is because I'm trying to import it more than once.

To Reproduce
Steps to reproduce the behaviour:
example when pickling and unpickling:

import bpy
import dill as pickle	

#pickle bpy module manually
with open('bpydill', 'wb') as file:
	pickle.dump(bpy, file)
	
#unpickle bpy module when inside thread
with open('bpydill', 'rb') as file:
	bpy = pickle.load(file)

Expected behavior
dill module has the ability to pickle modules and so should be able to parse the module into each thread. I understand that _bpy is a special part of the module that is compiled into blender and you've done some great trick to even get it to work.

Desktop (please complete the following information):
Windows 10 Pro 64
python 3.6.6

Additional context
Would be great if you know of another way I can get bpy to work inside and outside of the pool aside from pickling or importing multiple times.l

The text was updated successfully, but these errors were encountered:

TylerGubala · 2019-07-23T14:34:01Z

Is the "to reproduce" code block the code you are actually trying to run? I am not confident that bpy is safe for pickling. It is a strange solution that breaks a lot of established Python conventions.

HCAWN · 2019-07-23T15:40:47Z

Thanks for the quick reply. That snippet was just a way to show the problem. Here are some examples of more appropriate code:
Doesn't work

import multiprocessing
import bpy #doing import up here really breaks things

def multiFunction(data):
    print(data)
    print(bpy)

if __name__ == '__main__' :
    print(bpy)
    # do some bpy stuff here when in single thread

    p = multiprocessing.Pool()
    data = ['Uno','Deux','Three']
    p.map(multiFunction,data)
    p.close()

Also doesn't work

import multiprocessing

def multiFunction(data):
    print(data)
    import bpy # importing here again to get it into the threads causes the '_bpy' problem
    # pickling and unpickling here leads to the same issue
    print(bpy)

if __name__ == '__main__' :
    import bpy
    print(bpy)
    # do some bpy stuff here when in single thread


    p = multiprocessing.Pool()
    data = ['Uno','Deux','Three']
    p.map(multiFunction,data)
    p.close()

works - access outside of multi only

import multiprocessing

def multiFunction(data):
    print(data)
    #sad lack of bpy stuff possible in here

if __name__ == '__main__' :

    import bpy
    print(bpy)
    # do some bpy stuff here when in single thread


    p = multiprocessing.Pool()
    data = ['Uno','Deux','Three']
    p.map(multiFunction,data)
    p.close()

works - access inside of multi only

import multiprocessing

def multiFunction(data):
    print(data)
    import bpy # importing here as only way to get into threads
    print(bpy)

if __name__ == '__main__' :
    # no bpy stuff possible in here

    p = multiprocessing.Pool()
    data = ['Uno','Deux','Three']
    p.map(multiFunction,data)
    p.close()

For some reason, I can't import the module at the start of the file as per norm (as per the top example) the multiprocessing pool really does not like that.

TylerGubala · 2019-07-23T16:28:30Z

Okay, thanks a bunch for the awesome example case. I will have to look at it tonight, however I'm wondering about the validity of this "no module named _bpy" error.

_bpy if I'm not mistaken (I'm on mobile on work lunch right now so checking would be a pain) is a module that is appended to the sys.modules when the Blender runtime is instantiated. I'm thinking that maybe (and this is just a small theory) multiprocessing migrates bpy (thinking it took everything with it from the sys.modules cache, but it really didn't) and then the bpy runtime tries to import its (internal) _bpy module that lives in the .pyd

One thing to note; this project really doesn't do anything special or fancy. There aren't any real "tricks" involved. In fact, it's probably inefficient by a fair bit. The project just takes the works of a Blender guru called "Ideasman42" and expands upon it; it takes the Blender runtime, and just compiles it like it was a Python module.

So really bpy is not just exposing the related libraries like you might come to expect in a typical Python module; it actually just starts the Blender runtime and exposes the hooks to Python after the import is successful... which leads to all sorts of hijinks....

Sorry if that was just rambling but I will take your examples and see what I can do with them. I am thinking, maybe if you always in your multiprocessing function, as the first step clear the sys.modules cache and re-import maybe something good will happen? But I will have to test that working theory out.

Thanks for your patience.

HCAWN · 2019-07-24T09:19:45Z

Great, many thanks.

Ah yes that's interesting, it's certainly something to do with how the bpy module is constructed. I tried this and it results in the same error:

import bpy
import sys
print(bpy)
sys.modules.clear()
del bpy
try:
    print(bpy)
except: 
    print('bpy not printable')
import bpy #_bpy error here

clearly it's not able to completely remove bpy from the session as if I were to jsut run:

import bpy
import bpy

Then it doesn't struggle obviously.

H

TylerGubala · 2019-07-24T15:22:03Z

Bpy brings a bunch of paths into the sys.path area. The weird thing is that in my tests using your code, it is importing bpy from the 2.79/scripts/addons/modules area or something like that (can't remember right now, on mobile tested late at night) I'm wondering if that is the culprit, as the bpy that exists as a module in blender is different than the bpy runtime that exists in my built module. I'm going to try copying bpy.pyd, named as _bpy.pyd and see if the error persists tonight, as this is a very bizzare error.

…

On Wed, Jul 24, 2019, 5:19 AM Henry Neilson ***@***.***> wrote: Great, many thanks. Ah yes that's interesting, it's certainly something to do with how the bpy module is constructed. I tried this and it results in the same error: import bpy import sys print(bpy) sys.modules.clear() del bpy try: print(bpy) except: print('bpy not printable') import bpy #_bpy error here clearly it's not able to completely remove bpy from the session as if I were to jsut run: import bpy import bpy Then it doesn't struggle obviously. H — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#23?email_source=notifications&email_token=AFLEYR3N7GWEA3VTZ4TGWV3QBANDLA5CNFSM4IGB2JS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VXBYI#issuecomment-514552033>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLEYR2YGQHQM7BPN2BP6EDQBANDLANCNFSM4IGB2JSQ> .

TylerGubala · 2019-07-24T22:44:24Z

As I figured, this seems to be alleviated by preserving the sys.path. The following illustrates the way that bpy and the Blender runtime in general monkey with the sys.path to achieve the (desirable, in Blender's case) functionality of being able to import addon and contributed modules from the installation directory.

Microsoft Windows [Version 10.0.18362.239]
(c) 2019 Microsoft Corporation. All rights reserved.
C:\Users\TGubs>cd Code/Python/blender_test
C:\Users\TGubs\Code\Python\blender_test>"venvs/3.6.8-32/Scripts/activate"
(3.6.8-32) C:\Users\TGubs\Code\Python\blender_test>py
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import sys
>>> print("\n".join(sys.path))

C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\python36.zip
C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\DLLs
C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\lib
C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\lib\site-packages

>>> import bpy

AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead

>>> print("\n".join(sys.path))

C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons_contrib
C:\Users\TGubs\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\startup
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\python36.zip
C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\DLLs
C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32\lib
C:\Users\TGubs\AppData\Local\Programs\Python\Python36-32
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\lib\site-packages
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\freestyle\modules
C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\addons\modules
C:\Users\TGubs\AppData\Roaming\Blender Foundation\Blender\2.79\scripts\addons\modules

This, unfortunately interferes with Python's ability to import bpy in the new process that is created on multiprocessing.Pool. See, bpy, and any module for that matter is not pickleable... as in it's not going to be pickled and communicated through the Queue that is created when a new process is instantiated.

Sources:

So, multiprocessing's new Pool process is going to have to re-import the module. Unfortunately, something that happens at the creation of multiprocessing.Pool is that the sys.path gets copied and sent across the Queue to the new process. This is really bad in the case of bpy (but remember it's necessary for the Blender standalone executable) because the Blender runtime places its import paths at the front of the line.

Notice that in my above example, that C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules is close to the top; higher up even than the regular Python import path.

Well, that is really, really bad for us, because inside of that folder is a Python package named bpy! (which again, exposes some stuff for the Blender standalone executable, and so is somewhat necessary)

So what does it all mean? Well, here's the order:

We import bpy in the main process
This brings in the C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.832\Scripts\2.79\scripts\modules to the top of sys.path because the Blender runtime needs it
The multiprocessing.Pool instantiation grabs our bad sys.path
Since modules aren't portable across the multiprocessing.Queue our new processes are going to try to import the modules again
Search along sys.path for bpy
"I found one at C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-32\Scripts\2.79\scripts\modules!"
Error: can't locate or import its dependency _bpy (mostly because that is a Blender runtime convention that does not apply in Python)
Process setup fails repeatedly because of this

How can this be alleviated? Well... seems like something that would break Blender if we changed this to work the way that Python supports packaging. Adding paths to sys.path is kind of bad, especially when it introduces ambiguities in the package names like this.

But there is a workaround that is kind of hacky to get bpy working as it should in the other processes. You just have to temporarily switch out the sys.path so that it doesn't find that ambiguous bpy python package.

import multiprocessing
import sys

ORIG_SYS_PATH = list(sys.path) # Make a new instance of sys.path

import bpy # Here, the sys.path is severely messed with, screws up the import 
           # in the new process that is created in multiprocessing.Pool()

BPY_SYS_PATH = list(sys.path) # Make instance of `bpy`'s modified sys.path

def multiFunction(data):
    print(data)
    print(bpy)

if __name__ == '__main__' :
    print(bpy) # do some bpy stuff here when in main process

    data = ['Uno','Deux','Three']

    sys.path = ORIG_SYS_PATH

    p = multiprocessing.Pool()

    sys.path = BPY_SYS_PATH

    # do more stuff with bpy in main process

    p.map(multiFunction,data)
    p.close()

Running this gets me:

AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
<module 'bpy' from 'C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\lib\site-packages\bpy.cp36-win_amd64.pyd'>
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Uno
<module 'bpy' from 'C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\lib\site-packages\bpy.cp36-win_amd64.pyd'>
Deux
<module 'bpy' from 'C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\lib\site-packages\bpy.cp36-win_amd64.pyd'>
Three
<module 'bpy' from 'C:\Users\TGubs\Code\Python\blender_test\venvs\3.6.8-64\lib\site-packages\bpy.cp36-win_amd64.pyd'>
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
AL lib: (EE) UpdateDeviceParams: Failed to set 44100hz, got 48000hz instead
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB

That should be able to get you working for now. Not sure what to do here, or what the right approach might be. It really is an anti-pattern (in my opinion) that Blender does this, but the workaround doesn't seem too painful.

Let me know what you think @HCAWN .

HCAWN · 2019-07-26T15:45:53Z

@TylerGubala, amazing! Very good analysis. I've implemented your fix and it works great! Many thanks. I'm glad you're still maintaining this repo, it is beyond useful :)

TylerGubala · 2019-07-26T15:51:27Z

Thanks. I do have some better support for this repo coming out soon. I know that is the eternal promise, that improvements are always "coming soon" but I really have been trying to nail down people's issues (like this one) because the automated building that I envision for bpy is an entirely different headache.

Thanks for the high praise, though, and I hope this repo continues to be useful for you.

Closed.

3d-illusions · 2021-04-04T13:38:50Z

thannks @TylerGubala , but how would you get this to work from within an addon? It seems that bpy has already been imported prior to my addon initialising, so I don't seem able to get the original state of sys.path.

TylerGubala · 2021-04-04T17:48:22Z

@3d-illusions I really doubt that you would be able to use the aforementioned workaround in your addon. You might, but that is highly dependent on your use case. See below:

https://docs.blender.org/api/current/info_gotcha.html#strange-errors-when-using-the-threading-module

It would be more helpful to bring this up on Blender Dev talk since the operability between addon and the Blender runtime (within bpy as a Python module or otherwise) is maintained in the Blender source code.

3d-illusions · 2021-04-05T07:07:19Z

@3d-illusions I really doubt that you would be able to use the aforementioned workaround in your addon. You might, but that is highly dependent on your use case. See below:

https://docs.blender.org/api/current/info_gotcha.html#strange-errors-when-using-the-threading-module

It would be more helpful to bring this up on Blender Dev talk since the operability between addon and the Blender runtime (within bpy as a Python module or otherwise) is maintained in the Blender source code.

Thanks. It does say that subrocess and multiprocess modules can be used with Blender, so there must be a way without the workaround perhaps. I've posted on devtalk:

https://devtalk.blender.org/t/no-module-named-bpy-when-using-python-multiprocessing/18259/3

fingers crossed!

TylerGubala self-assigned this Jul 23, 2019

TylerGubala closed this as completed Jul 26, 2019

TylerGubala added documentation Something in the readme, comments or docstrings could be improved question Further information is requested wontfix This will not be worked on labels Jul 26, 2019

josuemtzmo mentioned this issue Jul 24, 2021

Implement dask client. blendernc/blendernc#120

Merged

pearbender mentioned this issue May 14, 2024

fix: wrong arguments key in train_s1.py & smpl render exception on Windows fudan-generative-vision/champ#118

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImportError: No module named '_bpy' When Pickling #23

ImportError: No module named '_bpy' When Pickling #23

HCAWN commented Jul 23, 2019 •

edited by TylerGubala

Loading

TylerGubala commented Jul 23, 2019

HCAWN commented Jul 23, 2019 •

edited by TylerGubala

Loading

TylerGubala commented Jul 23, 2019 •

edited

Loading

HCAWN commented Jul 24, 2019 •

edited by TylerGubala

Loading

TylerGubala commented Jul 24, 2019 via email

TylerGubala commented Jul 24, 2019 •

edited

Loading

HCAWN commented Jul 26, 2019

TylerGubala commented Jul 26, 2019

3d-illusions commented Apr 4, 2021

TylerGubala commented Apr 4, 2021

3d-illusions commented Apr 5, 2021

ImportError: No module named '_bpy' When Pickling #23

ImportError: No module named '_bpy' When Pickling #23

Comments

HCAWN commented Jul 23, 2019 • edited by TylerGubala Loading

TylerGubala commented Jul 23, 2019

HCAWN commented Jul 23, 2019 • edited by TylerGubala Loading

TylerGubala commented Jul 23, 2019 • edited Loading

HCAWN commented Jul 24, 2019 • edited by TylerGubala Loading

TylerGubala commented Jul 24, 2019 via email

TylerGubala commented Jul 24, 2019 • edited Loading

HCAWN commented Jul 26, 2019

TylerGubala commented Jul 26, 2019

3d-illusions commented Apr 4, 2021

TylerGubala commented Apr 4, 2021

3d-illusions commented Apr 5, 2021

HCAWN commented Jul 23, 2019 •

edited by TylerGubala

Loading

HCAWN commented Jul 23, 2019 •

edited by TylerGubala

Loading

TylerGubala commented Jul 23, 2019 •

edited

Loading

HCAWN commented Jul 24, 2019 •

edited by TylerGubala

Loading

TylerGubala commented Jul 24, 2019 •

edited

Loading