CMake can't find Python since python 3.11 was added

[dhdaines]

Some of my manylinux builds that use this image are failing with a maddening and incomprehensible CMake error, seemingly no matter how I invoke my build or set up my virtual environment or lack thereof:

-- Could NOT find Python3 (missing: Python3_INCLUDE_DIRS Python3_LIBRARIES Development Development.Module 
Development.Embed) (found version "3.11.0")

A minimal CMakeLists.txt that triggers the problem:

cmake_minimum_required(VERSION 3.14)
project(test VERSION 0.1)
find_package(Python3 COMPONENTS Interpreter Development)
add_library(bindings bindings.c)
target_link_libraries(bindings PRIVATE Python3::Python)

Steps to replicate:

cp /dev/null bindings.c
mkdir build
cd build
cmake ..

Curiously, skbuild seems not to trigger this problem.

[dhdaines]

I think the problem is not just manylinux2014 but generalized to all the images where Python 3.11 was added. I have it with quay.io/pypa/manylinux2014_x86_64 as well.

[mayeut]

You shall not depend on Development / Python3::Python when building an extension module.
Instead, you should use Development.Module / Python3::Module:

cmake_minimum_required(VERSION 3.18)
project(test VERSION 0.1)
find_package(Python3 COMPONENTS Interpreter Development.Module)
add_library(bindings bindings.c)
target_link_libraries(bindings PRIVATE Python3::Module)
message(STATUS "Python3_FOUND: ${Python3_FOUND}")
message(STATUS "Python3_Interpreter_FOUND: ${Python3_Interpreter_FOUND}")
message(STATUS "Python3_EXECUTABLE: ${Python3_EXECUTABLE}")
message(STATUS "Python3_INTERPRETER_ID: ${Python3_INTERPRETER_ID}")
message(STATUS "Python3_STDLIB: ${Python3_STDLIB}")
message(STATUS "Python3_STDARCH: ${Python3_STDARCH}")
message(STATUS "Python3_SITELIB: ${Python3_SITELIB}")
message(STATUS "Python3_SITEARCH: ${Python3_SITEARCH}")
message(STATUS "Python3_SOABI: ${Python3_SOABI}")
message(STATUS "Python3_INCLUDE_DIRS: ${Python3_INCLUDE_DIRS}")
message(STATUS "Python3_Development.Module_FOUND: ${Python3_Development.Module_FOUND}")
message(STATUS "Python3_LINK_OPTIONS: ${Python3_LINK_OPTIONS}")
message(STATUS "Python3_VERSION: ${Python3_VERSION}")

[root@cab768644fe5 ~]# cmake -B cp36 -DPython3_EXECUTABLE=$(which python3.6)
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/local/bin/python3.6 (found version "3.6.15") found components: Interpreter Development.Module 
-- Python3_FOUND: TRUE
-- Python3_Interpreter_FOUND: TRUE
-- Python3_EXECUTABLE: /usr/local/bin/python3.6
-- Python3_INTERPRETER_ID: Python
-- Python3_STDLIB: /opt/_internal/cpython-3.6.15/lib/python3.6
-- Python3_STDARCH: /opt/_internal/cpython-3.6.15/lib/python3.6
-- Python3_SITELIB: /opt/_internal/cpython-3.6.15/lib/python3.6/site-packages
-- Python3_SITEARCH: /opt/_internal/cpython-3.6.15/lib/python3.6/site-packages
-- Python3_SOABI: cpython-36m-x86_64-linux-gnu
-- Python3_INCLUDE_DIRS: /opt/_internal/cpython-3.6.15/include/python3.6m
-- Python3_Development.Module_FOUND: TRUE
-- Python3_LINK_OPTIONS: 
-- Python3_VERSION: 3.6.15
-- Configuring done
-- Generating done
-- Build files have been written to: /root/cp36

[root@cab768644fe5 ~]# cmake -B cp310 -DPython3_EXECUTABLE=$(which python3.10)
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/local/bin/python3.10 (found version "3.10.5") found components: Interpreter Development.Module 
-- Python3_FOUND: TRUE
-- Python3_Interpreter_FOUND: TRUE
-- Python3_EXECUTABLE: /usr/local/bin/python3.10
-- Python3_INTERPRETER_ID: Python
-- Python3_STDLIB: /opt/_internal/cpython-3.10.5/lib/python3.10
-- Python3_STDARCH: /opt/_internal/cpython-3.10.5/lib/python3.10
-- Python3_SITELIB: /opt/_internal/cpython-3.10.5/lib/python3.10/site-packages
-- Python3_SITEARCH: /opt/_internal/cpython-3.10.5/lib/python3.10/site-packages
-- Python3_SOABI: cpython-310-x86_64-linux-gnu
-- Python3_INCLUDE_DIRS: /opt/_internal/cpython-3.10.5/include/python3.10
-- Python3_Development.Module_FOUND: TRUE
-- Python3_LINK_OPTIONS: 
-- Python3_VERSION: 3.10.5
-- Configuring done
-- Generating done
-- Build files have been written to: /root/cp310

[root@cab768644fe5 ~]# cmake -B cp311 -DPython3_EXECUTABLE=$(which python3.11)
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/local/bin/python3.11 (found version "3.11.0") found components: Interpreter Development.Module 
-- Python3_FOUND: TRUE
-- Python3_Interpreter_FOUND: TRUE
-- Python3_EXECUTABLE: /usr/local/bin/python3.11
-- Python3_INTERPRETER_ID: Python
-- Python3_STDLIB: /opt/_internal/cpython-3.11.0b3/lib/python3.11
-- Python3_STDARCH: /opt/_internal/cpython-3.11.0b3/lib/python3.11
-- Python3_SITELIB: /opt/_internal/cpython-3.11.0b3/lib/python3.11/site-packages
-- Python3_SITEARCH: /opt/_internal/cpython-3.11.0b3/lib/python3.11/site-packages
-- Python3_SOABI: cpython-311-x86_64-linux-gnu
-- Python3_INCLUDE_DIRS: /opt/_internal/cpython-3.11.0b3/include/python3.11
-- Python3_Development.Module_FOUND: TRUE
-- Python3_LINK_OPTIONS: 
-- Python3_VERSION: 3.11.0
-- Configuring done
-- Generating done
-- Build files have been written to: /root/cp311

[dhdaines]

Thank you, this fixes the problem, closing the bug (but I will make sure this is documented somewhere...)

[dhdaines]

Ahhhh! Thank you! Knowing CMake I presume this is undocumented or hidden in one not so obvious location in the docs. Sorry for the misplaced bug report… Le dim. 10 juill. 2022 à 14:28, Matthieu Darbois ***@***.***> a écrit :

…

You shall not depend on Development / Python3::Python when building an extension module. Instead, you should use Development.Module / Python3::Module: