Installing Slycot (openmp) breaks numpy #230

chnce · 2024-01-26T09:32:06Z

After installing slycot in a conda environment, a kernel crash occurs when trying to solve a linear system with a singular matrix.

I narrowed the issue down to the following:

slycot from conda-forge depends on libflang >=5.0.0,<6.0.0.a0
libflang 5.0.0 depends on openmp 5.0.0
openmp 5.0.0 overwrites "libiomp5md.dll" that was installed by numpy. The two dlls are very different in size (2mb for the numpy one, 600kb for the openmp one)

Openmp places its "libiomp5md.dll" in Library\bin.
The version from numpy is either in Library\bin (channel main) or directly in bin (conda-forge).

Actual Behavior

Using numpy.linalg.solve on singular matrix with dimension over 149 crashes the Kernel.

Expected Behavior

Expected a "LinAlgError: Singular matrix" as it is shown with matrices up to the dimension of 149.

Steps to Reproduce

Create a new conda environment with python 3.11, numpy (and ipython for convenience)
- mamba create -n slycot_crash python=3.11 numpy ipython -c conda-forge
activate the environment
- mamba activate slycot_crash
enter ipython
- ipython
paste and run the code below to confirm it works as expected (raises LinAlgError)
exit ipython
- exit
install openmp=5
- mamba install openmp=5 -c conda-forge
enter ipython and run the code below again to confirm kernel crash
(optional) change n=149 and it only fails if the size is 150 or larger

import numpy as np

n = 150
A = np.random.rand(n,n)
A[-1,:] = np.zeros((1,n)) # make matrix singular

b = np.random.rand(n)

x = np.linalg.solve(A, b)

The text was updated successfully, but these errors were encountered:

moorepants · 2024-01-26T10:13:19Z

How are you installing the various packages? All with conda? Or other ways?

chnce · 2024-01-26T12:00:23Z

Thanks for the quick response. I detailed the steps in the original post above.

moorepants · 2024-01-26T12:18:04Z

Given that you are reporting a clash with numpy and openmp binaries from conda forge, then I think you should open the issue at one or both of these locations:

https://github.com/conda-forge/openmp-feedstock

https://github.com/conda-forge/numpy-feedstock

I don't see a slycot specific issue (yet).

chnce · 2024-01-26T12:48:16Z

I agree that it's not a true issue of slycot. But maybe slycot could move to a more modern version of libflang. I have little hope, that anyone at openmp is going to fix a 6 year old version.

I could frankenstein together a working environment by ignoring the dependencies and installing:

slycot --no-deps
libflang=5 --no-deps
openmp=8.0.1 llvm-openmp=8.0.1

moorepants · 2024-01-26T13:25:44Z

Ah, ok I understand now. slycot forces an old libflang and thus forces you to install an old openmp, which has an incompatibility with numpy. So yes, it would be helpful to move slycot to a newer libflang.

It doesn't look like this is something we explicitly set:

https://github.com/conda-forge/slycot-feedstock/blob/main/recipe/meta.yaml

It may mean that conda forge requires libfang<6 from the {{ compiler('fortran') }} call. If so, then this would be a bug to report to the numpy and openmp feedstocks.

moorepants · 2024-01-26T13:27:37Z

It looks locked to libflang 5 on windows across all of conda-forge: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml#L30

moorepants · 2024-01-26T13:28:35Z

Related: conda-forge/conda-forge-pinning-feedstock#5081

bnavigator · 2024-01-26T19:39:40Z

Yes we try to follow the conda-forge pinning for flang. See also conda-forge/slycot-feedstock#65 and conda-forge/blas-feedstock#106

chnce · 2024-01-29T07:54:35Z

Thank you for your insights. I filed the issue with numpy and openmp feedstock.
I'm genuinely surprised that no one else seems to have the same issue. openmp 5 was downloaded half a million times from conda-forge.

robertobucher · 2024-03-09T16:23:13Z

There is a similar problem under Debian. Using the installed python3-numpy package (numpy version 1.24.2 in trixie)

pip install slycot

doesn't work correctly (building stops). The only way to install slycot ist to force the removing of python3-numpy

sudo dpkg -P --force-all python3-numpy)

and install numpy using pip (in this case the installed version is numpy-1.26.4).After this step

pip install spycot

works.

bnavigator · 2024-03-10T11:05:59Z

Could you post a build log with the error message, please?

robertobucher · 2024-03-10T12:18:29Z

Tested on a docker after clonng Slycot in git, but it correclty works after updating numpy. In the pyproject.toml the version of rerquired numpy is numpy >= 1.23.1 an in Debian we have 1.24.2.

root@debian:/psc/Slycot# pip install --break-system-packages .
Processing /psc/Slycot
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.23.1 in /usr/lib/python3/dist-packages (from slycot==0.5.5.dev60+gfea37c7) (1.24.2)
Building wheels for collected packages: slycot
Building wheel for slycot (pyproject.toml) ... error
error: subprocess-exited-with-error

�� Building wheel for slycot (pyproject.toml) did not run successfully.
�� exit code: 1
��> [166 lines of output]

  --------------------------------------------------------------------------------
  -- Trying 'Ninja' generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 3.5 will be removed from a future version of
    CMake.
  
    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.
  
  Not searching for unused variables given on the command line.
  
  CMake Error: CMake was unable to find a build program corresponding to "Ninja".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  -- Configuring incomplete, errors occurred!
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying 'Ninja' generator - failure
  --------------------------------------------------------------------------------
  
  
  
  --------------------------------------------------------------------------------
  -- Trying 'Unix Makefiles' generator
  --------------------------------
  ---------------------------
  ----------------------
  -----------------
  ------------
  -------
  --
  CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
    Compatibility with CMake < 3.5 will be removed from a future version of
    CMake.
  
    Update the VERSION argument <min> value or use a ...<max> suffix to tell
    CMake that the project does not need compatibility with older versions.
  
  Not searching for unused variables given on the command line.
  
  -- The C compiler identification is GNU 13.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The Fortran compiler identification is GNU 13.2.0
  -- Detecting Fortran compiler ABI info
  -- Detecting Fortran compiler ABI info - done
  -- Check for working Fortran compiler: /usr/bin/gfortran - skipped
  -- Configuring done (0.2s)
  -- Generating done (0.0s)
  -- Build files have been written to: /psc/Slycot/_cmake_test_compile/build
  --
  -------
  ------------
  -----------------
  ----------------------
  ---------------------------
  --------------------------------
  -- Trying 'Unix Makefiles' generator - success
  --------------------------------------------------------------------------------
  
  Configuring Project
    Working directory:
      /psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-build
    Command:
      /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/cmake/data/bin/cmake /psc/Slycot -G 'Unix Makefiles' --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.8 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.11.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DCMAKE_BUILD_TYPE:STRING=Release
  
  Not searching for unused variables given on the command line.
  -- The C compiler identification is GNU 13.2.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- The Fortran compiler identification is GNU 13.2.0
  -- Detecting Fortran compiler ABI info
  -- Detecting Fortran compiler ABI info - done
  -- Check for working Fortran compiler: /usr/bin/gfortran - skipped
  -- Found Python: /usr/bin/python3 (found version "3.11.8") found components: Interpreter Development NumPy Development.Module Development.Embed
  CMake Warning (dev) at /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:245 (find_package):
    Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
    are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
    the cmake_policy command to set the policy and suppress this warning.
  
  Call Stack (most recent call first):
    CMakeLists.txt:13 (find_package)
  This warning is for project developers.  Use -Wno-dev to suppress it.
  
  -- Found PythonInterp: /usr/bin/python3 (found version "3.11.8")
  CMake Warning (dev) at /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake/FindPythonExtensions.cmake:252 (find_package):
    Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
    are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
    the cmake_policy command to set the policy and suppress this warning.
  
  Call Stack (most recent call first):
    CMakeLists.txt:13 (find_package)
  This warning is for project developers.  Use -Wno-dev to suppress it.
  
  -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.11.so (found version "3.11.8")
  -- Found NumPy: /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include (found version "1.26.4")
  Traceback (most recent call last):
    File "/usr/bin/f2py3", line 33, in <module>
      sys.exit(load_entry_point('numpy==1.24.2', 'console_scripts', 'f2py3.11')())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/bin/f2py3", line 25, in importlib_load_entry_point
      return next(matches).load()
             ^^^^^^^^^^^^^
  StopIteration
  -- Found F2PY: /usr/bin/f2py3 (found version "")
  -- Looking for Fortran sgemm
  -- Looking for Fortran sgemm - not found
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE
  -- Looking for Fortran sgemm
  -- Looking for Fortran sgemm - found
  -- Found BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so
  -- Looking for Fortran cheev
  -- Looking for Fortran cheev - found
  -- Found LAPACK: /usr/lib/x86_64-linux-gnu/libopenblas.so;-lm;-ldl
  -- Python headers included from: /usr/include/python3.11
  -- NumPy headers included from: /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include
  -- F2PY headers included from: /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/f2py/src;/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include
  -- LAPACK: /usr/lib/x86_64-linux-gnu/libopenblas.so;-lm;-ldl
  -- BLAS: /usr/lib/x86_64-linux-gnu/libopenblas.so
  CMake Error at CMakeLists.txt:29 (message):
    Python include directory has a numpy sub-directory,
  
          /usr/include/python3.11/numpy,
      which is different from Numpy include directory
        /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include.
      You're probably building in a virtual environment, in which case
      uninstall numpy from the base environment and try again.
  
  
  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/setuptools_wrap.py", line 666, in setup
      env = cmkr.configure(
            ^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/cmaker.py", line 357, in configure
      raise SKBuildError(msg)
  
  An error occurred while configuring with CMake.
    Command:
      /tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/cmake/data/bin/cmake /psc/Slycot -G 'Unix Makefiles' --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.8 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.11.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-vj8gsmqn/overlay/local/lib/python3.11/dist-packages/numpy/core/include -DCMAKE_BUILD_TYPE:STRING=Release
    Source directory:
      /psc/Slycot
    Working directory:
      /psc/Slycot/_skbuild/linux-x86_64-3.11/cmake-build
  Please see CMake's output for more information.
  
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for slycot
Failed to build slycot
ERROR: Could not build wheels for slycot, which is required to install pyproject.toml-based projects
root@debian:/psc/Slycot#

This was referenced Jan 29, 2024

numpy is not binary compatible with libflang/openmp 5 conda-forge/numpy-feedstock#308

Open

numpy is not binary compatible with libflang/openmp 5 conda-forge/openmp-feedstock#112

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installing Slycot (openmp) breaks numpy #230

Installing Slycot (openmp) breaks numpy #230

chnce commented Jan 26, 2024 •

edited

Loading

moorepants commented Jan 26, 2024

chnce commented Jan 26, 2024

moorepants commented Jan 26, 2024

chnce commented Jan 26, 2024 •

edited

Loading

moorepants commented Jan 26, 2024

moorepants commented Jan 26, 2024

moorepants commented Jan 26, 2024

bnavigator commented Jan 26, 2024

chnce commented Jan 29, 2024

robertobucher commented Mar 9, 2024

bnavigator commented Mar 10, 2024

robertobucher commented Mar 10, 2024

Installing Slycot (openmp) breaks numpy #230

Installing Slycot (openmp) breaks numpy #230

Comments

chnce commented Jan 26, 2024 • edited Loading

Actual Behavior

Expected Behavior

Steps to Reproduce

moorepants commented Jan 26, 2024

chnce commented Jan 26, 2024

moorepants commented Jan 26, 2024

chnce commented Jan 26, 2024 • edited Loading

moorepants commented Jan 26, 2024

moorepants commented Jan 26, 2024

moorepants commented Jan 26, 2024

bnavigator commented Jan 26, 2024

chnce commented Jan 29, 2024

robertobucher commented Mar 9, 2024

bnavigator commented Mar 10, 2024

robertobucher commented Mar 10, 2024

chnce commented Jan 26, 2024 •

edited

Loading

chnce commented Jan 26, 2024 •

edited

Loading