Workarounds for CPython Extensions

Background

Jep doesn't work correctly with some CPython extensions due to how those extensions were coded. Oftentimes projects were not designed towards the idea of running in embedded sub-interpreters instead of in a standard, single Python interpreter. When running Jep, these types of problems have been known to appear as a JVM crash, an attribute inexplicably set to None, or a thread deadlock. Please see the Package Compatibility page for information on packages others have used and tried, and feel free to contribute and make the list more informative.

Most of these errors can be traced to the CPython extension using global static variables that are not sandboxed to a sub-interpreter, and therefore behave erroneously when running in a multi-threaded or multi-interpreter environment. In these cases, it's good practice to try and track down the error and identify it to the project maintainers so they are at least aware of the problem. There's not much Jep can do to ensure that these other modules work correctly in a multi-threaded, embedded, sub-interpreter environment.

Workarounds

In the meantime, there are workarounds to avoid some of these problems when using Jep:

1. Use a singleton Jep instance. This prevents other sub-interpreters in the same process from messing up the C variables. You can potentially use the singleton instance for that one module and have multiple sub-interpreters for other Python code.
2. Use thread pools to reuse Jep instances and never close those Jep instances. This prevents errors where the module's methods or attributes may be garbage collected or reset when a sub-interpreter shuts down.
3. Synchronize on Jep calls into Python that use the module in particular. If the module is not thread safe, you can still attempt multiple sub-interpreters and synchronize when using Python that uses that module.