Wheels: 0.16.0 #1677

ax3l · 2024-10-08T02:37:58Z

Update the cibuildwheel reference tracking to the 0.16.0 release.

patch in Fix 0.16.0 versions #1680 or wait for 0.16.1 [WIP] Release: 0.16.1 #1676 because of botched setup.py #Release notes: 0.16 #1648 (comment)
rebase once ADIOS2 v2.10.1 is building Wheels: ADIOS2 v2.10.1 #1554

Update the cibuildwheel reference tracking to the 0.16.0 release.

Include openPMD#1680

ax3l · 2024-10-11T00:57:16Z

@franzpoeschel somehow, these wheels did neither pick up ADIOS2 nor HDF5... have to yank them from PyPI and redo.

ax3l · 2024-10-11T01:01:08Z

Ahh... WarpX CI runs on Python 3.8 and thus downloads the source file... will push 3.8 back in here

WarpX CI still uses it. x-ref: #1677 #1554

ax3l · 2024-10-11T16:08:49Z

There is another error in the built wheels that shows up in WarpX CI now:

/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, imp, stat
Traceback (most recent call last):
  File "/home/vsts/work/1/s/Examples/Tests/laser_injection_from_file/inputs_test_1d_laser_injection_from_lasy_file_prepare.py", line 15, in <module>
    from lasy.laser import Laser
  File "/home/vsts/.local/lib/python3.8/site-packages/lasy/laser.py", line 11, in <module>
    from lasy.utils.openpmd_output import write_to_openpmd_file
  File "/home/vsts/.local/lib/python3.8/site-packages/lasy/utils/openpmd_output.py", line 4, in <module>
    import openpmd_api as io
  File "/home/vsts/.local/lib/python3.8/site-packages/openpmd_api/__init__.py", line 1, in <module>
    from . import openpmd_api_cxx as cxx
ImportError: cannot import name 'openpmd_api_cxx' from partially initialized module 'openpmd_api' (most likely due to a circular import) (/home/vsts/.local/lib/python3.8/site-packages/openpmd_api/__init__.py)

This usually means there is a shared library or so not shipped in our all-static (at least we aim for that) Pip build.

I am Yanked the 0.16.0 on PyPI now (latest again: 0.15.2 there) and we will fix this in 0.16.1.

ax3l · 2024-10-11T16:11:46Z

side note: Although no entirely trivial, we need to add a post-build selfcheck for wheels before we publish them in the scripts of the wheels branch that runs our cibuildwheel logic. This type of issue can happen from time to time and we need to catch it with simple tests.

Will now do the post-mortem analysis so we can fix it in 0.16.1.

ax3l · 2024-10-11T16:23:05Z

I took a look at the buildlogs and I found our Python 3.8 build actually did build against Python 3.7, which is not compatible:

Linux x86_64 build:

...
2024-10-11T01:05:21.6661032Z ^[[1m^[[34mBuilding cp38-manylinux_x86_64 wheel^[[0m
2024-10-11T01:05:21.6682310Z CPython 3.8 manylinux x86_64
2024-10-11T01:05:21.6682703Z
2024-10-11T01:05:21.6683271Z ##[group]Setting up build environment...
2024-10-11T01:05:21.6683716Z
2024-10-11T01:05:21.7167413Z     + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
2024-10-11T01:05:21.7965886Z     + which python
2024-10-11T01:05:21.7998455Z     + which pip
...
2024-10-11T01:05:46.5548600Z + PY_BIN=/opt/python/cp38-cp38/bin/python3
2024-10-11T01:05:46.5557244Z ++ /opt/python/cp38-cp38/bin/python3 -m pip show cmake
...
2024-10-11T01:11:10.0402208Z   -- Found Python: /opt/python/cp38-cp38/bin/python (found suitable version "3.8.20", minimum required is "3.7.0") found components: Interpreter Development.Module
2024-10-11T01:11:10.0409267Z   -- Downloading pybind11 ...
2024-10-11T01:11:10.0411957Z   -- pybind11 source: https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.tar.gz
2024-10-11T01:11:10.6220741Z   -- pybind11 v2.13.6
2024-10-11T01:11:10.6611156Z   -- Found PythonInterp: /usr/local/bin/python3.7 (found suitable version "3.7.17", minimum required is "3.7")
2024-10-11T01:11:11.0897247Z   -- Found PythonLibs: python3.7m
2024-10-11T01:11:11.0910646Z   -- Performing Test HAS_FLTO
2024-10-11T01:11:11.1865125Z   -- Performing Test HAS_FLTO - Success
2024-10-11T01:11:11.1882522Z   -- Python LTO/IPO: ON
2024-10-11T01:11:11.1930950Z
2024-10-11T01:11:11.1933595Z   openPMD build configuration:
2024-10-11T01:11:11.1936840Z     library Version: 0.16.0
2024-10-11T01:11:11.1938817Z     openPMD Standard: 1.1.0
2024-10-11T01:11:11.1941220Z     C++ Compiler: GNU 10.2.1
2024-10-11T01:11:11.1943955Z       /opt/rh/devtoolset-10/root/usr/bin/c++
2024-10-11T01:11:11.1946179Z
2024-10-11T01:11:11.1948808Z     Install with RPATHs: ON
2024-10-11T01:11:11.1951372Z     Installation prefix: /usr/local
2024-10-11T01:11:11.1953775Z           bin: bin
2024-10-11T01:11:11.1956123Z           lib: lib64
2024-10-11T01:11:11.1958340Z       include: include
2024-10-11T01:11:11.1960601Z         cmake: lib64/cmake/openPMD
2024-10-11T01:11:11.1962976Z        python: lib64/python3.8/site-packages
...

I am not sure how this happened yet, but we'll find out... 👀

ax3l · 2024-10-11T16:37:48Z

I think this is from either the FindPython in pybind11 or the new changes in CMake, where -DPython_EXECUTABLE=$(which python) is not a sufficient hint anymore these days...

How we hint in setup.py, which is what pip calls, which is what cibuildwheel calls:
https://github.com/openPMD/openPMD-api/blob/0.16.0/setup.py#L52

This was based on https://cmake.org/cmake/help/v3.30/module/FindPython.html#artifacts-specification This was never a proper HINT in CMake, just worked ok. We could either

hard-set more of these artifacts (Python_LIBRARY, Python_INCLUDE_DIR) or
limit the versions eligible for find via -DPYBIND11_PYTHON_VERSION=3.12 (doc) as we did here problems building on os X stddef.h and math.h not found ECP-WarpX/WarpX#5159
from the pybind11 docs & source, it also looks like it still uses the old ALL_CAPS hint PYTHON_EXECUTABLE internally, which might be the lib used here. So we might need to hint that again in setup.py!

franzpoeschel · 2024-10-11T16:59:05Z

2024-10-11T01:11:10.0402208Z   -- Found Python: /opt/python/cp38-cp38/bin/python (found suitable version "3.8.20", minimum required is "3.7.0") found components: Interpreter Development.Module
2024-10-11T01:11:10.0409267Z   -- Downloading pybind11 ...
2024-10-11T01:11:10.0411957Z   -- pybind11 source: https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.tar.gz
2024-10-11T01:11:10.6220741Z   -- pybind11 v2.13.6
2024-10-11T01:11:10.6611156Z   -- Found PythonInterp: /usr/local/bin/python3.7 (found suitable version "3.7.17", minimum required is "3.7")

I've seen this before when trying to make openPMD use a non-system Python. The best fix for that was always to use a virtualenv.
Can we be sure that Python_EXECUTABLE propagates correctly into the sub-builds when doing a superbuild?

ax3l · 2024-10-11T17:22:56Z

Trying to check if this is truly only for superbuilds or also in pre-installed pybind versions:
https://github.com/pybind/pybind11/blob/af67e87393b0f867ccffc2702885eea12de063fc/tools/pybind11Config.cmake.in#L4

I think only superbuilds, I cannot find a link (direct or transitive) to FindPythonLibsNew.cmake from pybind11Tools.cmake yet...

https://github.com/pybind/pybind11/blob/v2.13.6/CMakeLists.txt#L302

ax3l · 2024-10-11T17:27:31Z

Weird that it does not detect our use of find_package(Python ...) automatically:

ax3l · 2024-10-11T17:37:38Z

The while `find_package(PythonInterp ...) inside FindPythonLibsNew.cmake is deprecated.
We probably should try to use a build path that does not rely on it...

I cannot even find how FindPythonLibsNew.cmake is included at all with the latest release of pybind11...
Update: here it is called from:

Is pybind11 overwrititing the CMake stock FindPythonLibs.cmake...? And if so, why would it be used if we call the new find_package(Python ...) already...

ax3l · 2024-10-11T17:52:21Z

Ok, so if we set PYBIND11_FINDPYTHON=ON
https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake

Then pybind11 will call find_package(Python ...) again, which will see that we already called it (default find_package behavior), use our cached results and rely on the new logic
https://github.com/pybind/pybind11/blob/v2.13.6/tools/pybind11Common.cmake#L215-L223

Well, not obvious from the docs, but ok.
https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake

ax3l · 2024-10-11T18:59:49Z

Fix developed in #1684

Reuse our `find_package(Python ...)` call and use new CMake logic in pybind11. https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-version-selection X-ref: openPMD/openPMD-api#1677 (comment) Signed-off-by: Axel Huebl <[email protected]>

Reuse our `find_package(Python ...)` call and use new CMake logic in pybind11. https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-version-selection X-ref: openPMD/openPMD-api#1677 (comment) Fix #5159 Signed-off-by: Axel Huebl <[email protected]>

Reuse our `find_package(Python ...)` call and use new CMake logic in pybind11. https://pybind11.readthedocs.io/en/stable/compiling.html#modules-with-cmake https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-version-selection X-ref: openPMD/openPMD-api#1677 (comment) Fix ECP-WarpX#5159 Signed-off-by: Axel Huebl <[email protected]>

ax3l added the continuous deployment automated shipping label Oct 8, 2024

Wheels: 0.16.0

15692b8

Update the cibuildwheel reference tracking to the 0.16.0 release.

ax3l force-pushed the wheels-0.16.0 branch from e034746 to 15692b8 Compare October 8, 2024 17:05

ax3l changed the title ~~Wheels: 0.16.0~~ Wheels: 0.16.0.dev Oct 8, 2024

ax3l mentioned this pull request Oct 10, 2024

Fix 0.16.0 versions #1680

Merged

ax3l changed the title ~~Wheels: 0.16.0.dev~~ Wheels: 0.16.0 Oct 10, 2024

ax3l force-pushed the wheels-0.16.0 branch 9 times, most recently from fff6adb to 4013c07 Compare October 10, 2024 18:27

[Patch] Fix 0.16.0 versions

d5cffb4

Include openPMD#1680

ax3l force-pushed the wheels-0.16.0 branch from 4013c07 to d5cffb4 Compare October 10, 2024 18:29

ax3l requested a review from franzpoeschel October 10, 2024 18:30

ax3l assigned franzpoeschel Oct 10, 2024

ax3l merged commit 6d126ad into openPMD:wheels Oct 10, 2024
7 of 9 checks passed

ax3l deleted the wheels-0.16.0 branch October 10, 2024 19:56

ax3l added a commit that referenced this pull request Oct 11, 2024

Wheels: Keep Python 3.8 for now

9e1c4d8

WarpX CI still uses it. x-ref: #1677 #1554

This was referenced Oct 11, 2024

PYBIND11_FINDPYTHON=ON #1684

Merged

PYBIND11_FINDPYTHON=ON AMReX-Codes/pyamrex#381

Merged

This was referenced Oct 11, 2024

PYBIND11_FINDPYTHON=ON ECP-WarpX/WarpX#5390

Merged

PYBIND11_FINDPYTHON=ON ECP-WarpX/impactx#738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wheels: 0.16.0 #1677

Wheels: 0.16.0 #1677

ax3l commented Oct 8, 2024 •

edited

Loading

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

franzpoeschel commented Oct 11, 2024

ax3l commented Oct 11, 2024 •

edited

Loading

ax3l commented Oct 11, 2024 •

edited

Loading

ax3l commented Oct 11, 2024 •

edited

Loading

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

Wheels: 0.16.0 #1677

Wheels: 0.16.0 #1677

Conversation

ax3l commented Oct 8, 2024 • edited Loading

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

franzpoeschel commented Oct 11, 2024

ax3l commented Oct 11, 2024 • edited Loading

ax3l commented Oct 11, 2024 • edited Loading

ax3l commented Oct 11, 2024 • edited Loading

ax3l commented Oct 11, 2024

ax3l commented Oct 11, 2024

ax3l commented Oct 8, 2024 •

edited

Loading

ax3l commented Oct 11, 2024 •

edited

Loading

ax3l commented Oct 11, 2024 •

edited

Loading

ax3l commented Oct 11, 2024 •

edited

Loading