-
-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ImportError: No module named '_bpy' When Pickling #23
Comments
Is the "to reproduce" code block the code you are actually trying to run? I am not confident that bpy is safe for pickling. It is a strange solution that breaks a lot of established Python conventions. |
Thanks for the quick reply. That snippet was just a way to show the problem. Here are some examples of more appropriate code: import multiprocessing
import bpy #doing import up here really breaks things
def multiFunction(data):
print(data)
print(bpy)
if __name__ == '__main__' :
print(bpy)
# do some bpy stuff here when in single thread
p = multiprocessing.Pool()
data = ['Uno','Deux','Three']
p.map(multiFunction,data)
p.close() Also doesn't work import multiprocessing
def multiFunction(data):
print(data)
import bpy # importing here again to get it into the threads causes the '_bpy' problem
# pickling and unpickling here leads to the same issue
print(bpy)
if __name__ == '__main__' :
import bpy
print(bpy)
# do some bpy stuff here when in single thread
p = multiprocessing.Pool()
data = ['Uno','Deux','Three']
p.map(multiFunction,data)
p.close() works - access outside of multi only import multiprocessing
def multiFunction(data):
print(data)
#sad lack of bpy stuff possible in here
if __name__ == '__main__' :
import bpy
print(bpy)
# do some bpy stuff here when in single thread
p = multiprocessing.Pool()
data = ['Uno','Deux','Three']
p.map(multiFunction,data)
p.close() works - access inside of multi only import multiprocessing
def multiFunction(data):
print(data)
import bpy # importing here as only way to get into threads
print(bpy)
if __name__ == '__main__' :
# no bpy stuff possible in here
p = multiprocessing.Pool()
data = ['Uno','Deux','Three']
p.map(multiFunction,data)
p.close() For some reason, I can't import the module at the start of the file as per norm (as per the top example) the multiprocessing pool really does not like that. |
Okay, thanks a bunch for the awesome example case. I will have to look at it tonight, however I'm wondering about the validity of this "no module named
One thing to note; this project really doesn't do anything special or fancy. There aren't any real "tricks" involved. In fact, it's probably inefficient by a fair bit. The project just takes the works of a Blender guru called "Ideasman42" and expands upon it; it takes the Blender runtime, and just compiles it like it was a Python module. So really Sorry if that was just rambling but I will take your examples and see what I can do with them. I am thinking, maybe if you always in your multiprocessing function, as the first step clear the sys.modules cache and re-import maybe something good will happen? But I will have to test that working theory out. Thanks for your patience. |
Great, many thanks. Ah yes that's interesting, it's certainly something to do with how the bpy module is constructed. I tried this and it results in the same error: import bpy
import sys
print(bpy)
sys.modules.clear()
del bpy
try:
print(bpy)
except:
print('bpy not printable')
import bpy #_bpy error here clearly it's not able to completely remove import bpy
import bpy Then it doesn't struggle obviously. H |
Bpy brings a bunch of paths into the sys.path area.
The weird thing is that in my tests using your code, it is importing bpy
from the 2.79/scripts/addons/modules area or something like that (can't
remember right now, on mobile tested late at night)
I'm wondering if that is the culprit, as the bpy that exists as a module in
blender is different than the bpy runtime that exists in my built module.
I'm going to try copying bpy.pyd, named as _bpy.pyd and see if the error
persists tonight, as this is a very bizzare error.
…On Wed, Jul 24, 2019, 5:19 AM Henry Neilson ***@***.***> wrote:
Great, many thanks.
Ah yes that's interesting, it's certainly something to do with how the bpy
module is constructed. I tried this and it results in the same error:
import bpy
import sys
print(bpy)
sys.modules.clear()
del bpy
try:
print(bpy)
except:
print('bpy not printable')
import bpy #_bpy error here
clearly it's not able to completely remove bpy from the session as if I
were to jsut run:
import bpy
import bpy
Then it doesn't struggle obviously.
H
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#23?email_source=notifications&email_token=AFLEYR3N7GWEA3VTZ4TGWV3QBANDLA5CNFSM4IGB2JS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VXBYI#issuecomment-514552033>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFLEYR2YGQHQM7BPN2BP6EDQBANDLANCNFSM4IGB2JSQ>
.
|
As I figured, this seems to be alleviated by preserving the
This, unfortunately interferes with Python's ability to import Sources:
So, Notice that in my above example, that Well, that is really, really bad for us, because inside of that folder is a Python package named So what does it all mean? Well, here's the order:
How can this be alleviated? Well... seems like something that would break Blender if we changed this to work the way that Python supports packaging. Adding paths to But there is a workaround that is kind of hacky to get import multiprocessing
import sys
ORIG_SYS_PATH = list(sys.path) # Make a new instance of sys.path
import bpy # Here, the sys.path is severely messed with, screws up the import
# in the new process that is created in multiprocessing.Pool()
BPY_SYS_PATH = list(sys.path) # Make instance of `bpy`'s modified sys.path
def multiFunction(data):
print(data)
print(bpy)
if __name__ == '__main__' :
print(bpy) # do some bpy stuff here when in main process
data = ['Uno','Deux','Three']
sys.path = ORIG_SYS_PATH
p = multiprocessing.Pool()
sys.path = BPY_SYS_PATH
# do more stuff with bpy in main process
p.map(multiFunction,data)
p.close() Running this gets me:
That should be able to get you working for now. Not sure what to do here, or what the right approach might be. It really is an anti-pattern (in my opinion) that Blender does this, but the workaround doesn't seem too painful. Let me know what you think @HCAWN . |
@TylerGubala, amazing! Very good analysis. I've implemented your fix and it works great! Many thanks. I'm glad you're still maintaining this repo, it is beyond useful :) |
Thanks. I do have some better support for this repo coming out soon. I know that is the eternal promise, that improvements are always "coming soon" but I really have been trying to nail down people's issues (like this one) because the automated building that I envision for Thanks for the high praise, though, and I hope this repo continues to be useful for you. Closed. |
thannks @TylerGubala , but how would you get this to work from within an addon? It seems that bpy has already been imported prior to my addon initialising, so I don't seem able to get the original state of sys.path. |
@3d-illusions I really doubt that you would be able to use the aforementioned workaround in your addon. You might, but that is highly dependent on your use case. See below: https://docs.blender.org/api/current/info_gotcha.html#strange-errors-when-using-the-threading-module It would be more helpful to bring this up on Blender Dev talk since the operability between addon and the Blender runtime (within |
Thanks. It does say that subrocess and multiprocess modules can be used with Blender, so there must be a way without the workaround perhaps. I've posted on devtalk: https://devtalk.blender.org/t/no-module-named-bpy-when-using-python-multiprocessing/18259/3 fingers crossed! |
Describe the bug
I have a script that splits in to a multiprocess.pool(). I require the bpy module both before the pool and inside each thread in the pool. I can get bpy to function outside or inside but not both.
There are three things main methods I've tried:
import bpy
at the start of the script outside ofif __name__ == '__main__' :
this causes
ImportError: No module named '_bpy'
- I presume because it's trying to import bpy multiple timesimport bpy
inside ofif __name__ == '__main__' :
this works for outside the
pool
but if I want to parse the module into the pool, I have to pickle it viadill
and then unpickle it inside each thread which also results inImportError: No module named '_bpy'
import bpy
inside ofif __name__ == '__main__' :
and then inside each thread:same error comes up as above, once again I think this is because I'm trying to import it more than once.
To Reproduce
Steps to reproduce the behaviour:
example when pickling and unpickling:
Expected behavior
dill
module has the ability to pickle modules and so should be able to parse the module into each thread. I understand that_bpy
is a special part of the module that is compiled into blender and you've done some great trick to even get it to work.Desktop (please complete the following information):
Windows 10 Pro 64
python 3.6.6
Additional context
Would be great if you know of another way I can get bpy to work inside and outside of the
pool
aside from pickling or importing multiple times.lThe text was updated successfully, but these errors were encountered: