Jep integration with scala spark

[yunboouyang]

Is your feature request related to a problem? Please describe.
I'm trying jep 3.9.0 in spark scala and doing experiments on Yarn clusters. However I always got the following error

jep.JepException: <class 'ModuleNotFoundError'>: No module named 'jep'
	at <string>.<module>(<string>:1)
	at jep.Jep.eval(Native Method)
	at jep.Jep.eval(Jep.java:451)
	at jep.Jep.configureInterpreter(Jep.java:302)
	at jep.SharedInterpreter.configureInterpreter(SharedInterpreter.java:64)
	at jep.Jep.<init>(Jep.java:282)
	at jep.SharedInterpreter.<init>(SharedInterpreter.java:56)

This issue is related with #18, which indicates I need to set up PYTHONHOME environment variable so that I can find the virtual env directory. This is not easy in Spark. Could you provide some advice on how Jep integrates with Spark? Thank you!

[yunboouyang]

I also tried using addIncludePaths to include jep package in sys.path in python and set config for SharedInterpreter. However, a new error occurs:

jep.JepException: <class 'ImportError'>: dynamic module does not define module export function (PyInit_jep)
	at <string>.<module>(<string>:1)
	at jep.Jep.eval(Native Method)
	at jep.Jep.eval(Jep.java:451)
	at jep.Jep.configureInterpreter(Jep.java:302)
	at jep.SharedInterpreter.configureInterpreter(SharedInterpreter.java:64)
	at jep.Jep.<init>(Jep.java:282)
	at jep.SharedInterpreter.<init>(SharedInterpreter.java:56)

[yunboouyang]

Hi! I used objdump to print out the symbol table for jep.cpython-36m-x86_64-linux-gnu.so but cannot find anything related with PyInit_. Is this the reason that the function PyInit_jep cannot be found?

[bsteffensmeier]

There is no PyInit_Jep in the .so because it is not loaded by python but instead is loaded by java/jni. The error probably means your python path is incorrect, it should point to the directory that contains the jep directory and not the jep directory itself. Python should load the jep module using the init.py in the jep directory, not using the .so.

[yunboouyang]

Thanks! The cause is that I set python path to the jep directory. Close this issue now.