jep.JepException: <type 'exceptions.ImportError'>: No module named python.framework

[xvolks]

• OS Platform, Distribution, and Version:
  macOS 10.14.1
• Python Distribution and Version:
  macOS's 2.7.10
• Java Distribution and Version:
  Oracle 1.8.0_172
• Jep Version:
  3.8.2 installed with pip2
• Python packages used (e.g. numpy, pandas, tensorflow):
  keras and tensorflow.

When I run a simple python script it seems to work great but if I try to import Keras, it throws an exception:
jep.JepException: <type 'exceptions.ImportError'>: No module named python.framework

I've tried to install jep on 3 different macs with the same result.

I was also unable to install JEP with Python 3.7.1 with following error TypeError: '_NamespacePath' object does not support indexing

I've joined a Kotlin source code demonstrating this behaviour.
Its output is :

Hello, World from kotlin
Hello world from python
2.7.10 (default, Aug 17 2018, 17:41:52) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
Using TensorFlow backend.
Exception in thread "main" jep.JepException: <type 'exceptions.ImportError'>: No module named python.framework
	at /Users/xvolks/transfer_learning/lib/python2.7/site-packages/keras/backend/tensorflow_backend.<module>(tensorflow_backend.py:6)
	at /Users/xvolks/transfer_learning/lib/python2.7/site-packages/keras/backend/__init__.<module>(__init__.py:89)
	at /Users/xvolks/transfer_learning/lib/python2.7/site-packages/keras/utils/conv_utils.<module>(conv_utils.py:9)
	at /Users/xvolks/transfer_learning/lib/python2.7/site-packages/keras/utils/__init__.<module>(__init__.py:6)
	at /Users/xvolks/transfer_learning/lib/python2.7/site-packages/keras/__init__.<module>(__init__.py:3)
	at jep.Jep.eval(Native Method)
	at jep.Jep.eval(Jep.java:507)
	at org.xvolks.research.Hello$Companion.evalMultiline(Hello.kt:31)
	at org.xvolks.research.Hello$Companion.access$evalMultiline(Hello.kt:26)
	at org.xvolks.research.Hello.eval(Hello.kt:22)
	at org.xvolks.research.HelloKt.main(Hello.kt:10)

Hello.kt.gz

[bsteffensmeier]

It looks like keras is just failing because part of tensorflow is failing. Can you get rid of the import keras and instead do

import tensorflow
print(tensorflow.__path__)
import tensorflow.python.framework

I expect the third line will fail since that is what keras is doing when it fails but I am hoping the print statement will give you some indication what is wrong with tensorflow.

Also have you verified that you can import keras from python, without using jep?

[xvolks]

As asked, I have modified the code of Hello.kt this way:

package org.xvolks.research

import jep.Jep
import jep.JepConfig
import java.util.*


fun main(args: Array<String>) {
    println("Hello, World from kotlin")
    val hello = Hello()

    hello.eval("""
import sys
print('Hello world from python')
print (sys.version)
import tensorflow
print(tensorflow.__path__)
import tensorflow.python.framework
""")

}

The output is

Hello, World from kotlin
Hello world from python
2.7.10 (default, Aug 17 2018, 17:41:52) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
[]
Exception in thread "main" jep.JepException: <type 'exceptions.ImportError'>: No module named python.framework
	at jep.Jep.eval(Native Method)
	at jep.Jep.eval(Jep.java:507)
	at org.xvolks.research.Hello$Companion.evalMultiline(Hello.kt:45)
	at org.xvolks.research.Hello$Companion.access$evalMultiline(Hello.kt:40)
	at org.xvolks.research.Hello.eval(Hello.kt:36)
	at org.xvolks.research.HelloKt.main(Hello.kt:22)

I wonder why tensorflow is not really found, but does not fail on import statement.

EDIT: After removing dependencies on deeplearning4j in my pom.xml and removing all code using DL4j, JEP is now able to load tensorflow fine.
This is a bit disappointing since I was planning to use both DL4j and Keras in the same project.

Is there any common / incompatible dependency between JEP and DL4j ?

Here is the pom part with the problem :

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <kotlin.version>1.3.0</kotlin.version>
        <kotlin.code.style>official</kotlin.code.style>
        <junit.version>4.12</junit.version>
        <nd4j.backend>nd4j-native-platform</nd4j.backend>
        <deeplearning.version>1.0.0-beta3</deeplearning.version>
        <nd4j.version>${deeplearning.version}</nd4j.version>
        <dl4j.version>${deeplearning.version}</dl4j.version>
        <datavec.version>${deeplearning.version}</datavec.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>${nd4j.backend}</artifactId>
            <version>${nd4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-nlp</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui_2.11</artifactId>
            <version>${dl4j.version}</version>
        </dependency>

        <dependency>
            <groupId>black.ninia</groupId>
            <artifactId>jep</artifactId>
            <version>3.8.2</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-stdlib</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-test-junit</artifactId>
            <version>${kotlin.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

[bsteffensmeier]

dl4j must be pulling in a Java package named tensorflow. By default Jep scans the classpath and allows you to import all your Java packages into Python, this takes precedence over the Python packages if there is a naming conflict. To resolve this I recommend passing a custom ClassEnquirer into you JepConfig. You want to make sure that isJavaPackage() returns false for tensorflow. You should be able to simply extend the default ClassList and overriding that one method to check for tensorflow.

[xvolks]

Thank you very much for your advice. It worked.
For the record I wrote a singleton :

package org.xvolks.research

import jep.ClassEnquirer
import jep.ClassList

object JepClassEnquirer : ClassEnquirer {
    override fun getClassNames(pkgName: String?): Array<String> {
        return ClassList.getInstance().getClassNames(pkgName)
    }

    override fun getSubPackages(pkgName: String?): Array<String> {
        return ClassList.getInstance().getSubPackages(pkgName)
    }

    override fun isJavaPackage(name: String?): Boolean {
        return if (name == "tensorflow") {
            false
        } else {
            ClassList.getInstance().isJavaPackage(name)
        }
    }
}

and modified my Jep loading code with :

class Hello {

    fun eval(string: String) {
        evalMultiline(string, jeps.get()
        )
    }

    companion object {
        val jeps = ThreadLocal.withInitial { createJep() }

        private fun evalMultiline(script: String, jep:Jep) {
            val statement = "exec(\"\"\"$script\"\"\")"
            jep.eval(statement)
        }

        private fun createJep(): Jep {
            return JepConfig()
                .setInteractive(true)
                .setClassEnquirer(JepClassEnquirer) // ensure loading tensorflow from Python side
                .setRedirectOutputStreams(false)
                .createJep()
        }
    }
}

Now Jep, tensorflow on Python and Dl4j are working together.
Thanks.