pyimport("scipy.fft") fails for scipy v1.8.1

[sethaxen]

If we install scipy v1.8.1 in the Julia-specific conda environment and call pyimport("scipy.fft"), we get the following error:

julia> using PyCall

julia> pyimport("scipy.fft")
ERROR: PyError (PyImport_ImportModule

The Python package scipy.fft could not be imported by pyimport. Usually this means
that you did not install scipy.fft in the Python version being used by PyCall.

PyCall is currently configured to use the Julia-specific Python distribution
installed by the Conda.jl package.  To install the scipy.fft module, you can
use `pyimport_conda("scipy.fft", PKG)`, where PKG is the Anaconda
package that contains the module scipy.fft, or alternatively you can use the
Conda package directly (via `using Conda` followed by `Conda.add` etcetera).

Alternatively, if you want to use a different Python distribution on your
system, such as a system-wide Python (as opposed to the Julia-specific Python),
you can re-configure PyCall with that Python.   As explained in the PyCall
documentation, set ENV["PYTHON"] to the path/name of the python executable
you want to use, run Pkg.build("PyCall"), and re-launch Julia.

) <class 'ImportError'>
ImportError("/home/sethaxen/.julia/juliaup/julia-1.7.3+0~x64/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)")
  File "/home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/__init__.py", line 91, in <module>
    from ._helper import next_fast_len
  File "/home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/_helper.py", line 3, in <module>
    from ._pocketfft import helper as _helper
  File "/home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/_pocketfft/__init__.py", line 3, in <module>
    from .basic import *
  File "/home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/_pocketfft/basic.py", line 6, in <module>
    from . import pypocketfft as pfft

Stacktrace:
 [1] pyimport(name::String)
   @ PyCall ~/.julia/packages/PyCall/7a7w0/src/PyCall.jl:550
 [2] top-level scope
   @ REPL[3]:1

If we manually launch the Julia-specific conda's Python executable and call import scipy.fft, it loads with no problem. If we instead install scipy v1.8.0, then pyimport("scipy.fft") also runs with no problem.

[stevengj]

This is the underlying error:

ImportError("/home/sethaxen/.julia/juliaup/julia-1.7.3+0~x64/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-39-x86_64-linux-gnu.so)")

i.e. it is an incompatibility with the C++ standard library — Julia links one version, pypocketfft wants another.

Not much we can do about this in PyCall. You'll need either a different version of Julia or a different version of pypocketfft, built with compatible versions of libstdc++.

[sethaxen]

Ah okay, so this has the same solution as #722 (comment), namely

$ LD_PRELOAD=$HOME/.julia/conda/3/lib/libstdc++.so julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.3 (2022-05-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using PyCall

julia> pyimport("scipy.fft")
PyObject <module 'scipy.fft' from '/home/sethaxen/.julia/conda/3/lib/python3.9/site-packages/scipy/fft/__init__.py'>

Is this the best advice I can give to users of my package? It seems that this incompatibility of Julia's libstdc++ with some Python package's can happen at any time. In this case, a new Julia version has even been released (v1.8.0-rc1) since scipy v1.8.1 was released, and has the same error.

[bencardoen]

@sethaxen Thank you for your workaround, this fixes (for now) my setup (both with Julia 1.7 and 1.8), I had a conda package "kneed" that dependend on scipy.fft, with the above error. I can confirm that the PRELOAD trick works.

[StefanPofahl]

I can see a difference between MS Windows and Linux, on Windows, I do not observe an issue.

[StefanPofahl]

Here my code snippet that fails:

using PyCall
py"""
from scipy import signal
"""

and here the error output:

ERROR: PyError ($(Expr(:escape, :(ccall(#= /home/stefan/.julia/packages/PyCall/ygXW2/src/pyeval.jl:38 =# @pysym(:PyEval_EvalCode), PyPtr, (PyPtr, PyPtr, PyPtr), o, globals, locals))))) <class 'ModuleNotFoundError'>
ModuleNotFoundError("No module named 'scipy'")
  File "/home/stefan/.julia/packages/PyCall/ygXW2/src/pyeval.jl", line 1, in <module>
    const Py_single_input = 256  # from Python.h

Stacktrace:
 [1] pyerr_check
   @ ~/.julia/packages/PyCall/ygXW2/src/exception.jl:62 [inlined]
 [2] pyerr_check
   @ ~/.julia/packages/PyCall/ygXW2/src/exception.jl:66 [inlined]
 [3] _handle_error(msg::String)
   @ PyCall ~/.julia/packages/PyCall/ygXW2/src/exception.jl:83
 [4] macro expansion
   @ ~/.julia/packages/PyCall/ygXW2/src/exception.jl:97 [inlined]
 [5] #117
   @ ~/.julia/packages/PyCall/ygXW2/src/pyeval.jl:38 [inlined]
 [6] disable_sigint
   @ ./c.jl:458 [inlined]
 [7] pyeval_(s::String, globals::PyDict{String, PyObject, true}, locals::PyDict{String, PyObject, true}, input_type::Int64, fname::String)
   @ PyCall ~/.julia/packages/PyCall/ygXW2/src/pyeval.jl:37
 [8] top-level scope
   @ ~/.julia/packages/PyCall/ygXW2/src/pyeval.jl:230

and if I issue the last script the output is:

include("/home/stefan/.julia/packages/PyCall/ygXW2/src/pyeval.jl")
ERROR: LoadError: UndefVarError: @pycheckn not defined
in expression starting at /home/stefan/.julia/packages/PyCall/ygXW2/src/pyeval.jl:10

[bencardoen]

@StefanPofahl
This works for me
using PyCall; p=pyimport("scipy.signal")

Note that to get around the fft GLIBC issue I use:

using PyCall
using Conda
Conda.add("gcc=12.1.0"; channel="conda-forge")
Conda.add("scikit-image")
# Pin this version, to avoid clashes with libgcc.34
Conda.add("scipy=1.8.0")

[bencardoen]

@StefanPofahl for completeness, in my packages where I use PyCall/scipy, I have
./deps/build.jl

with

#./deps/build.jl
using Pkg;
using Logging;
@info "Initiating build"
ENV["PYTHON"] = ""
Pkg.add("Conda")
Pkg.add("PyCall")
Pkg.build("PyCall")
Pkg.build("Conda")
using PyCall
using Conda
## Add the two packages we need
Conda.add("gcc=12.1.0"; channel="conda-forge")
Conda.add("scikit-image")
# Pin this version, to avoid clashes with libgcc.34
Conda.add("scipy=1.8.0")

Note that this is failing due to an unrelated issue with Conda on MacOs (I'm told the latest Mac updates fix that), other than that this works for me across systems.

[StefanPofahl]

Hi bencardoen
this worked fine, thanks :-)
Regards,
Stefan

[bencardoen]

glad it was of use :-)
@StefanPofahl the build.jl approach also (for me) means that the pyimport is precompiled, so when it's actually called, e.g. in a tight loop, there's no latency on first call. Though in general, using PackageCompiler.jl allows for even faster precompilation in my experience.

[StefanPofahl]

Hi @bencardoen , I will have a look on PackageCompiler.jl. Thanks!