Skip to content

Commit

Permalink
apacheGH-39555: [Packaging][Python] Enable building pyarrow against n…
Browse files Browse the repository at this point in the history
…umpy 2.0 (apache#39557)

### Rationale for this change

Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly. This also ensures that other projects that use our nightly wheels to test together with numpy nightly can do that (numpy 2.0 changes the ABI, so to run with numpy 2.0, your package needs to be built with numpy 2.x; currently pyarrow installed with our nightly wheel will fail to import when also numpy nightly is installed).

See the parent issue apache#39532 for details, and https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for a direct link to the NumPy guidelines on updating build dependencies for NumPy 2.0.

* Closes: apache#39555

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
  • Loading branch information
2 people authored and thisisnic committed Mar 8, 2024
1 parent 6feecb5 commit 4ef2e5f
Show file tree
Hide file tree
Showing 7 changed files with 20 additions and 8 deletions.
5 changes: 3 additions & 2 deletions ci/docker/python-wheel-manylinux.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ ENV MANYLINUX_VERSION=${manylinux}
RUN yum install -y dnf

# Install basic dependencies
RUN dnf install -y git flex curl autoconf zip perl-IPC-Cmd wget kernel-headers
RUN dnf install -y git flex curl autoconf zip perl-IPC-Cmd wget

# A system Python is required for ninja and vcpkg in this Dockerfile.
# On manylinux2014 base images, system Python is 2.7.5, while
Expand Down Expand Up @@ -97,4 +97,5 @@ SHELL ["/bin/bash", "-i", "-c"]
ENTRYPOINT ["/bin/bash", "-i", "-c"]

COPY python/requirements-wheel-build.txt /arrow/python/
RUN pip install -r /arrow/python/requirements-wheel-build.txt
# TODO(GH-39848) Remove the `--pre --extra-index-url` for numpy nightly again before the 16.0 release
RUN pip install -r /arrow/python/requirements-wheel-build.txt --pre --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
3 changes: 2 additions & 1 deletion ci/docker/python-wheel-windows-vs2017.dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ RUN choco install -r -y --no-progress python --version=%PYTHON_VERSION%
RUN python -m pip install -U pip setuptools

COPY python/requirements-wheel-build.txt arrow/python/
RUN python -m pip install -r arrow/python/requirements-wheel-build.txt
# TODO(GH-39848) Remove the `--pre --extra-index-url` for numpy nightly again before the 16.0 release
RUN python -m pip install -r arrow/python/requirements-wheel-build.txt --pre --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"

# ENV CLCACHE_DIR="C:\clcache"
# ENV CLCACHE_COMPRESS=1
Expand Down
5 changes: 4 additions & 1 deletion ci/scripts/python_wheel_macos_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,15 @@ echo "=== (${PYTHON_VERSION}) Install Python build dependencies ==="
export PIP_SITE_PACKAGES=$(python -c 'import site; print(site.getsitepackages()[0])')
export PIP_TARGET_PLATFORM="macosx_${MACOSX_DEPLOYMENT_TARGET//./_}_${arch}"

# TODO(GH-39848) Remove the `--pre --extra-index-url` for numpy nightly again before the 16.0 release
pip install \
--upgrade \
--only-binary=:all: \
--target $PIP_SITE_PACKAGES \
--platform $PIP_TARGET_PLATFORM \
-r ${source_dir}/python/requirements-wheel-build.txt
-r ${source_dir}/python/requirements-wheel-build.txt \
--pre \
--extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
pip install "delocate>=0.10.3"

echo "=== (${PYTHON_VERSION}) Building Arrow C++ libraries ==="
Expand Down
7 changes: 6 additions & 1 deletion python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@
[build-system]
requires = [
"cython >= 0.29.31",
"oldest-supported-numpy>=0.14",
# Starting with NumPy 1.25, NumPy is (by default) as far back compatible
# as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION
# define). For older Python versions (where NumPy 1.25 is not yet avaiable)
# continue using oldest-support-numpy.
"oldest-supported-numpy>=0.14; python_version<'3.9'",
"numpy>=1.25; python_version>='3.9'",
"setuptools_scm < 8.0.0",
"setuptools >= 40.1.0",
"wheel"
Expand Down
3 changes: 2 additions & 1 deletion python/requirements-build.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
cython>=0.29.31
oldest-supported-numpy>=0.14
oldest-supported-numpy>=0.14; python_version<'3.9'
numpy>=1.25; python_version>='3.9'
setuptools_scm<8.0.0
setuptools>=38.6.0
3 changes: 2 additions & 1 deletion python/requirements-wheel-build.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
cython>=0.29.31
oldest-supported-numpy>=0.14
oldest-supported-numpy>=0.14; python_version<'3.9'
numpy>=1.25; python_version>='3.9'
setuptools_scm<8.0.0
setuptools>=58
wheel
2 changes: 1 addition & 1 deletion python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ def has_ext_modules(foo):


install_requires = (
'numpy >= 1.16.6, <2',
'numpy >= 1.16.6',
)


Expand Down

0 comments on commit 4ef2e5f

Please sign in to comment.