Installation path for Python bindings is not searched by the interpreter if CMAKE_INSTALL_PREFIX is /usr/local

[PeterBowman]

Describe the bug
YARP selects an installation path for Python bindings which the Python interpreter is not able to find out-of-the-box.

To Reproduce
Configure YARP via cmake .. -DYARP_COMPILE_BINDINGS=ON -DCREATE_PYTHON=ON. CMake populates the following variables with no further action on my part (inspected via ccmake):

Python3_LIBRARY_RELEASE=/usr/lib/x86_64-linux-gnu/libpython3.8.so
_Python3_EXECUTABLE=/usr/bin/python3.8
_Python3_INCLUDE_DIR=/usr/include/python3.8
_Python3_LIBRARY_RELEASE=/usr/lib/x86_64-linux-gnu/libpython3.8.so

However, binaries (yarp.py and _yarp.so) are installed in the following path: CMAKE_INSTALL_PYTHON3DIR=lib/python3/dist-packages. This will not work for me (i.e. python3 -c 'import yarp' fails) because Python expects to find packages at /usr/local/lib/python3.8/dist-packages/ (among other paths), but note the currently configured value is lib/python3/ instead.

Expected behavior
Perhaps I could just tweak PYTHONPATH in my .bashrc, but it would be nice to have YARP select the best path for Python packages automatically. I believe this workaround wasn't necessary in the past.

Configuration

• OS: Ubuntu 20.04.2 LTS (focal), 5.8.0-44-generic
• YARP: 3.4.2+30-20210218.5+git5726d2dd4
• Compiler: g++ (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
• Python: Python 3.8.5 (default, Jul 28 2020, 12:59:40) installed via libpython3-dev from Canonical's PPA

Additional context
I believe the lib/python3/ path in CMAKE_INSTALL_PYTHON3DIR originates from:

yarp/bindings/python/CMakeLists.txt

Lines 32 to 35 in d02f3fd

	# installation path is determined reliably on most platforms using distutils
	execute_process(COMMAND ${Python3_EXECUTABLE} -c "from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix=''))"
	OUTPUT_VARIABLE Python3_INSTDIR
	OUTPUT_STRIP_TRAILING_WHITESPACE )

I can tweak CMAKE_INSTALL_PYTHON3DIR via ccmake, but the final installation path remains unchanged. The workaround for me consists of creating symlinks from lib/python3.8/dist-packages/ to lib/python3/dist-packages/, as suggested here.

Possible solution (also workaround, not ideal)
Use CMAKE_INSTALL_PYTHON3DIR instead of Python3_INSTDIR here:

yarp/bindings/python/CMakeLists.txt

Lines 46 to 47 in d02f3fd

	install(FILES ${CMAKE_BINARY_DIR}/lib/python3/yarp.py
	DESTINATION ${Python3_INSTDIR})

yarp/bindings/python/CMakeLists.txt

Lines 62 to 63 in d02f3fd

	install(TARGETS ${SWIG_MODULE_yarp_python_REAL_NAME}
	DESTINATION ${Python3_INSTDIR})

See also
Perhaps related to some extent: #2509 (Support a user defined version of Python for YARP bindings).

[traversaro]

Related to robotology/robotology-superbuild#428 . Apparently for python2 the existing logic (i.e. from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='') ) installed the bindings in lib/pythonMAJ.MIN/dist-packages/, but that changed when we started using Python 3. I am definitely in favor to switch to any logic such that if you install in /usr/local or /usr on Ubuntu/Debian the bindings can be searched without tweaking the PYTHONPATH.

cc @diegoferigo @GiulioRomualdi

[traversaro]

I am definitely in favor to switch to any logic such that if you install in /usr/local or /usr on Ubuntu/Debian the bindings can be searched without tweaking the PYTHONPATH.

The problem is that /usr/lib/python3 seems to be searched, and that is the reason why sysconfig.get_python_lib() returns it. Any idea why instead /usr/local/lib/python3 is not searched?

[traversaro]

I am definitely in favor to switch to any logic such that if you install in /usr/local or /usr on Ubuntu/Debian the bindings can be searched without tweaking the PYTHONPATH.

The problem is that /usr/lib/python3 seems to be searched, and that is the reason why sysconfig.get_python_lib() returns it. Any idea why instead /usr/local/lib/python3 is not searched?

Given this comment I updated the title to reflect more specifically the problem, @PeterBowman feel free to change it again if you think it make sense.

[traversaro]

I divided my comment in different sections to simplify its readability.

Why we need to use distutils.sysconfig.get_python_lib : for being platform indipendent

As for robotology-superbuild we are interesting in support both apt and conda-forge installation of Python bindings out of the box, I looked into this. On Ubuntu 20.04 apt we have:

>>> sys.path
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
>>> distutils.sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='')
'lib/python3/dist-packages'

while on Windows/conda-forge we have:

>>> sys.path
['', 'C:\\Users\\STraversaro\\Miniforge3\\envs\\python-tests\\python39.zip', 'C:\\Users\\STraversaro\\Miniforge3\\envs\\python-tests\\DLLs', 'C:\\Users\\STraversaro\\Miniforge3\\envs\\python-tests\\lib', 'C:\\Users\\STraversaro\\Miniforge3\\envs\\python-tests', 'C:\\Users\\STraversaro\\Miniforge3\\envs\\python-tests\\lib\\site-packages']
>>> distutils.sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='')
'Lib\\site-packages'

From this two cases, we can see that hardcoding the Python install path to lib/python<MAJOR>.<MINOR>/dist-packages/ is not a viable solution.

The actual source of the problem

I investigate a bit, and this seems to be more a Debian/Ubuntu problem rather then a downstream YARP problem, see https://bugs.launchpad.net/ubuntu/+source/python3-stdlib-extensions/+bug/1832215 . If the issue gets fixed as suggested in the issue (but the issue seems to have been like that for more than an year), then probably we would need to modify our logic from:

execute_process(COMMAND ${Python3_EXECUTABLE} -c "from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix=''))"
                OUTPUT_VARIABLE Python3_INSTDIR
                OUTPUT_STRIP_TRAILING_WHITESPACE )

to

execute_process(COMMAND ${Python3_EXECUTABLE} -c "from distutils import sysconfig; print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='${CMAKE_INSTALL_PREFIX}'))"
                OUTPUT_VARIABLE Python3_INSTDIR
                OUTPUT_STRIP_TRAILING_WHITESPACE )

as the actual install prefix would depend to the value passed to the prefix argument.

However, until the issue https://bugs.launchpad.net/ubuntu/+source/python3-stdlib-extensions/+bug/1832215 is solved, I am afraid that implementing Ubuntu-specific workarounds in the CMake may bring more problems then benefits, and so just specifying =DCMAKE_INSTALL_PYTHON3DIR=lib/python3.8/dist-packages/ seems the right strategy.

The bug we can actually quickly solve

However:

I can tweak CMAKE_INSTALL_PYTHON3DIR via ccmake, but the final installation path remains unchanged.

This is an unrelated bug, for fixing that we just need to modify the line:

yarp/bindings/python/CMakeLists.txt

Line 47 in d02f3fd

DESTINATION ${Python3_INSTDIR})

to

        DESTINATION ${CMAKE_INSTALL_PYTHON3DIR})

[PeterBowman]

+1 from me for fixing the destination path (${Python3_INSTDIR} -> ${CMAKE_INSTALL_PYTHON3DIR}) so that we can at least have a working -DCMAKE_INSTALL_PYTHON3DIR=whatever option.

By the way, I came across this SO answer and the command python -c import site; site.getsitepackages()[0] looks promising. On both Ubuntu 18.04 and 20.04 (Python installed via Canonical's PPA) it returns /usr/local/lib/pythonx.y/dist-packages. Note this is an absolute path.

[traversaro]

By the way, I came across this SO answer and the command python -c import site; site.getsitepackages()[0] looks promising. On both Ubuntu 18.04 and 20.04 (Python installed via Canonical's PPA) it returns /usr/local/lib/pythonx.y/dist-packages. Note this is an absolute path.

How can this information be combined with CMAKE_INSTALL_PREFIX ? Just fyi, on conda/Windows the site.getsitepackages() returns:

>>> import site
>>> site.getsitepackages()
['<conda_env_prefix>', '<conda_env_prefix>\\lib\\site-packages']

and the path returned by if print(sysconfig.get_python_lib(plat_specific=True,standard_lib=False,prefix='')) is Lib\\site-packages'.

[PeterBowman]

You are right, I had not tested this with Conda, so I'm going to back off my words :). It seems difficult to find a unique solution that works well for all scenarios.

[traversaro]

Yes, it may be worth mentioning that CMake itself uses distutils.sysconfig.get_python_lib to compute some of its result variables https://cmake.org/cmake/help/latest/module/FindPython3.html#result-variables .

[drdanz]

@traversaro I think this was fixed by #2523, right?

[traversaro]

Technically speaking the problem is in Ubuntu and it is tracked in https://bugs.launchpad.net/ubuntu/+source/python3-stdlib-extensions/+bug/1832215 . #2523 implement a way for user to workaround the problem, and at this point I think there is not much left to do at the YARP level. If @PeterBowman agrees, for me we can close this issue.

[PeterBowman]

@traversaro I'm sorry that I missed it, the #2523 patch should be followed by #2539 in order to fix the custom path issue. The latter PR will close this ticket 👍.

Note for travellers: when passing CMAKE_INSTALL_PYTHON3_DIR via command-line, it could be necessary to specify the variable type. Such as: cmake .. -DCMAKE_INSTALL_PYTHON3_DIR:PATH=path/to/dist-packages (note the PATH). Otherwise CMake might resolve it to an absolute path pointing at the current build directory. Apparently this is not an issue if using ccmake. Kudos to @jgvictores for testing and reporting it.

PS quoting https://cmake.org/cmake/help/latest/manual/cmake.1.html#options:

If the :<type> portion is given it must be one of the types specified by the set() command documentation for its CACHE signature. If the :<type> portion is omitted the entry will be created with no type if it does not exist with a type already. If a command in the project sets the type to PATH or FILEPATH then the <value> will be converted to an absolute path.

Emphasis mine. This variable is in fact interpreted as PATH in:

yarp/bindings/python/CMakeLists.txt

Line 32 in f593fec

set(CMAKE_INSTALL_PYTHON3DIR ${_CMAKE_INSTALL_PYTHON3DIR} CACHE PATH "python3 bindings (${_CMAKE_INSTALL_PYTHON3DIR})")

[traversaro]

@traversaro I'm sorry that I missed it, the #2523 patch should be followed by #2539 in order to fix the custom path issue. The latter PR will close this ticket 👍.

Good catch!

Note for travellers: when passing CMAKE_INSTALL_PYTHON3_DIR via command-line, it could be necessary to specify the variable type. Such as: cmake .. -DCMAKE_INSTALL_PYTHON3_DIR:PATH=path/to/dist-packages (note the PATH). Otherwise CMake might resolve it to an absolute path pointing at the current build directory. Apparently this is not an issue if using ccmake. Kudos to @jgvictores for reporting it.

Yes, I can add that specifying PATH will also avoid use strange backslash vs slash issue on Windows.