Several issues when building PyImath

[waebbl]

I've encountered a few issues, when building Imath with Python support enabled.

• The imathnumpy.so module has no executable bit set, although in the build directory the bit is set:

$ ls -l /var/tmp/portage/dev-libs/imath-9999/work/imath-9999_build/python3_8/
total 5836
-rwxr-xr-x 1 waebbl waebbl 5916200 Feb 21 21:15 imath.so
-rwxr-xr-x 1 waebbl waebbl   56872 Feb 21 21:15 imathnumpy.so

vs.

$ ls -l /var/tmp/portage/dev-libs/imath-9999/image/usr/lib/python3.8/site-packages/
total 4092
-rwxr-xr-x 1 waebbl waebbl 4148352 Feb 21 21:23 imath.so
-rw-r--r-- 1 waebbl waebbl   39920 Feb 21 21:23 imathnumpy.so

• The RPATH for imathnumpy.so isn't cleared during install stage (see the QA notice at the end of build log).
• The PyImath library has no proper API version string attached, it is named libPyImath_Python3_8-3.0.so.VERSION for Python 3.8 (see install stage in build log)
• The PyImath.pc pkg-config file seems not correct. First, the ${{,exec_}prefix} is missing on several variables. See the difference in the functions at https://github.com/AcademySoftwareFoundation/Imath/blob/master/config/CMakeLists.txt#L45 and https://github.com/AcademySoftwareFoundation/Imath/blob/master/src/python/config/CMakeLists.txt#L16. See also Wrong paths in PyIlmBase pkg-config for 2.5.5 openexr#922 for an identical issue in OpenEXR-2.5.5 and compare the installed Imath.pc and PyImath.pc files. Second, in the Libs: section, the PyImath library is missing a Python suffix. It only has the ${libsuffix} appended, while the acutal library has an _Python${PYTHON_MAJOR}_${PYTHON_MINOR}${libsuffix} suffix scheme. The file I got looks like:

##
## SPDX-License-Identifier: BSD-3-Clause
## Copyright Contributors to the OpenEXR Project.
##

prefix=/usr
exec_prefix=bin
libdir=lib64
includedir=include
libsuffix=

Name: PyImath
Description: Python bindings for the Imath libraries
Version: 3.0.0
Libs: -L${libdir} -lImath${libsuffix} -lPyImath${libsuffix}
Cflags: -I${includedir} -I${includedir}/Imath

For details, please see the attached build log. System is Gentoo Linux, gcc-10.2.0, Python-3.8.8, boost-1.75.0, numpy-1.19.5. If you need any more information, don't hesitate to ping me.

build.log

[cary-ilm]

Thanks, we'll investigate.

…

On Sun, Feb 21, 2021 at 12:59 PM Bernd ***@***.***> wrote: I've encountered a few issues, when building Imath with Python support enabled. - The imathnumpy.so module has no executable bit set, although in the build directory the bit is set: $ ls -l /var/tmp/portage/dev-libs/imath-9999/work/imath-9999_build/python3_8/ total 5836 -rwxr-xr-x 1 waebbl waebbl 5916200 Feb 21 21:15 imath.so -rwxr-xr-x 1 waebbl waebbl 56872 Feb 21 21:15 imathnumpy.so vs. $ ls -l /var/tmp/portage/dev-libs/imath-9999/image/usr/lib/python3.8/site-packages/ total 4092 -rwxr-xr-x 1 waebbl waebbl 4148352 Feb 21 21:23 imath.so -rw-r--r-- 1 waebbl waebbl 39920 Feb 21 21:23 imathnumpy.so - The RPATH for imathnumpy.so isn't cleared during install stage (see the QA notice at the end of build log). - The PyImath library has no proper API version string attached, it is named libPyImath_Python3_8-3.0.so.VERSION for Python 3.8 (see install stage in build log) - The PyImath.pc pkg-config file seems not correct. First, the ${{,exec_}prefix} is missing on several variables. See the difference in the functions at https://github.com/AcademySoftwareFoundation/Imath/blob/master/config/CMakeLists.txt#L45 and https://github.com/AcademySoftwareFoundation/Imath/blob/master/src/python/config/CMakeLists.txt#L16. See also AcademySoftwareFoundation/openexr#922 <AcademySoftwareFoundation/openexr#922> for an identical issue in OpenEXR-2.5.5 and compare the installed Imath.pc and PyImath.pc files. Second, in the Libs: section, the PyImath library is missing a Python suffix. It only has the ${libsuffix} appended, while the acutal library has an _Python${PYTHON_MAJOR}_${PYTHON_MINOR}${libsuffix} suffix scheme. The file I got looks like: ## ## SPDX-License-Identifier: BSD-3-Clause ## Copyright Contributors to the OpenEXR Project. ## prefix=/usr exec_prefix=bin libdir=lib64 includedir=include libsuffix= Name: PyImath Description: Python bindings for the Imath libraries Version: 3.0.0 Libs: -L${libdir} -lImath${libsuffix} -lPyImath${libsuffix} Cflags: -I${includedir} -I${includedir}/Imath For details, please see the attached build log. System is Gentoo Linux, gcc-10.2.0, Python-3.8.8, boost-1.75.0, numpy-1.19.5. If you need any more information, don't hesitate to ping me. build.log <https://github.com/AcademySoftwareFoundation/Imath/files/6018457/build.log> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#101>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFC3DGNB54HEQJAVEJP3CFLTAFX3HANCNFSM4X7LPV2Q> .

-- Cary Phillips | R&D Supervisor | ILM | San Francisco

[waebbl]

I'm sorry, I was probably wrong with the first bullet above. Removal of the executable bit is more likely to be done by the package manager routine I use to install the package from the build stage to the (sandboxed) final install location.
Yet the imathnumpy.so doesn't get installed at all by cmake's install call.

[meshula]

We set VERSION here, to PYIMATH_LIB_VERSION, but I haven't spotted where we set PYIMATH_LIB_VERSION. I wonder if that's the root cause of the problem.

Imath/src/python/config/ModuleDefine.cmake

Line 19 in 1fa1e00

VERSION ${PYIMATH_LIB_VERSION}

[waebbl]

That's the only appearance of PYIMATH_LIB_VERSION in the code. Shouldn't the soversion of Imath and PyImath match anyway? Maybe this PYIMATH_LIB_VERSION could be replaced by IMATH_LIB_VERSION from https://github.com/AcademySoftwareFoundation/Imath/blob/master/config/version.cmake#L13?

[meshula]

We set VERSION here, to PYIMATH_LIB_VERSION, where do we set PYIMATH_LIB_VERSION?

Imath/src/python/config/ModuleDefine.cmake

Line 19 in 1fa1e00

VERSION ${PYIMATH_LIB_VERSION}

We specify the following for pymathnumpy:

  set_target_properties(imathnumpy_python3 PROPERTIES
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/python${Python3_VERSION_MAJOR}_${Python3_VERSION_MINOR}/"
    LIBRARY_OUTPUT_NAME "imathnumpy"
    DEBUG_POSTFIX ""
  )

did the library appear in an unexpected place, or not at all?

[waebbl]

did the library appear in an unexpected place, or not at all?

Not at all

$ find /var/tmp/portage/dev-libs/imath-9999/image -name 'imath*so'
/var/tmp/portage/dev-libs/imath-9999/image/usr/lib/python3.8/site-packages/imath.so

Searching for *numpy* reveals no result.
Currently, I install it manually, using a helper function provided for packagers.

You set the target properties, but IIRC there's no install called for the PyIMathNumpy subdir.

[waebbl]

We set VERSION here, to PYIMATH_LIB_VERSION, where do we set PYIMATH_LIB_VERSION?

Grepping the code, I found no other appearance of the symbol. It's also not in the CMakeCache.txt file, so maybe it's not set at all?
I only found the https://github.com/AcademySoftwareFoundation/Imath/blob/master/config/version.cmake which defines IMATH_LIB_VERSION, but there's no equivalent file for the python bindings.

[cary-ilm]

Separate version variables for Imath and PyImath is a relic of the earlier organization where IlmBase and PyIlmBase were packaged independently, we should consolidate that into a single variable.

[meshula]

Cary, is this work: AcademySoftwareFoundation/openexr#929 the corresponding work on the OpenEXR side to consolidating the variables?

[waebbl]

There's also some redundancy in the PyImath related cmake files. Settings for c++ standard, installation of pkg-config files, LIB_SUFFIX, OUTPUT_SUBDIR, rpath settings, build and configuration types and OPENEXR_USE_CLANG_TIDY are repeated in Imath/src/python/config/PyImathSetup.cmake from Imath/config/ImathSetup.cmake

[meshula]

There's no reason they can't share a common setup.

[cary-ilm]

@meshula, yes, #929 also cleans up some remnants of the previous organization. I'll also look into cleaning up the *Setup.cmake files, these can be simplified.

[waebbl]

If you like, I can look into some of the cmake related topics.

[meshula]

@waebbl Is this issue, mentioned in PR #102, addressed in another commit, or is it still to be fixed? "Still todo is the -lPyImath${libsuffix} entry in PyImath.pc.in, which needs to be covered by the python variable changes as well."

[waebbl]

@meshula As far as I can see it still needs to be fixed. I just pulled the latest commits, since PR #102 has been merged. They don't contain any changes to the python selection. To solve this issue, the separation to build for only one python version has to be done, which seems not the case so far.

I was thinking about trying to do this, after the PR got approved, but was distracted by other projects since then.

[meshula]

Ok, thanks for the notes.

[meshula]

@waebbl The separation has now been committed ~ if you are still thinking of trying to do the libsuffix bit, I think there's nothing blocking it.

[waebbl]

@meshula It should have been solved in #109 with https://github.com/AcademySoftwareFoundation/Imath/pull/109/files#diff-6c153d467305d4bba8e3768a4d4a8b6960f5fc4a25a9f7538a403c9c5e01b842R15

[meshula]

Great, thanks for the confirmation.