Skip to content

GH-39555: [Packaging][Python] Enable building pyarrow against numpy 2.0 #39557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jorisvandenbossche merged 5 commits into from
Feb 1, 2024
Merged

GH-39555: [Packaging][Python] Enable building pyarrow against numpy 2.0 #39557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d1a2df1
remove runtime upper pin, use plain numpy>=1.25 as build requirement
jorisvandenbossche Jan 11, 2024
185ffba
update requirements file, allow install numpy nightly in wheel builds
jorisvandenbossche Jan 11, 2024
fa80565
add comments
jorisvandenbossche Jan 30, 2024
88daada
Merge remote-tracking branch 'upstream/main' into gh-39555-build-numpy-2
jorisvandenbossche Jan 30, 2024
66748cd
Try to remove kernel-headers dependency
pitrou Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions ci/docker/python-wheel-manylinux.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ ENV MANYLINUX_VERSION=${manylinux}
RUN yum install -y dnf

# Install basic dependencies
RUN dnf install -y git flex curl autoconf zip perl-IPC-Cmd wget kernel-headers
RUN dnf install -y git flex curl autoconf zip perl-IPC-Cmd wget

# A system Python is required for ninja and vcpkg in this Dockerfile.
# On manylinux2014 base images, system Python is 2.7.5, while
Expand Down Expand Up @@ -97,4 +97,5 @@ SHELL ["/bin/bash", "-i", "-c"]
ENTRYPOINT ["/bin/bash", "-i", "-c"]

COPY python/requirements-wheel-build.txt /arrow/python/
RUN pip install -r /arrow/python/requirements-wheel-build.txt
# TODO(GH-39848) Remove the `--pre --extra-index-url` for numpy nightly again before the 16.0 release
RUN pip install -r /arrow/python/requirements-wheel-build.txt --pre --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added this for testing now, but so eventually this should only be done when this is being run as a nightly job (for creating the nightly wheels), and not on a release.

I am not sure how to detect that inside this dockerfile (passing some env variable?). Now, in practice that is maybe also not needed, because since we don't plan to backport this to the 15.x branch, this commit should only live on main and by the time of the next 16.x release cycle, we should be able to remove this again (by that time, numpy 2.0 should be released)

Copy link
Member

@pitrou pitrou Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just use the same logic as in ci/scripts/install_pandas.sh?

Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Jan 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we open an issue labeled as blocker for 16.0 to remove this again (to build with (by then) released numpy), we won't forget to update this, and for now I think it is fine to just always build with numpy nightly on the main branch

Copy link
Member

@pitrou pitrou Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building with Numpy nightly doesn't seem like a tremendous idea, is it? It is not required functionally and may introduce instability.

Copy link
Member

@pitrou pitrou Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least should add a comment, and open an issue to remove this.

Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Jan 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment and opened a blocker issue -> #39848

3 changes: 2 additions & 1 deletion ci/docker/python-wheel-windows-vs2017.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ RUN choco install -r -y --no-progress python --version=%PYTHON_VERSION%
RUN python -m pip install -U pip setuptools

COPY python/requirements-wheel-build.txt arrow/python/
RUN python -m pip install -r arrow/python/requirements-wheel-build.txt
# TODO(GH-39848) Remove the `--pre --extra-index-url` for numpy nightly again before the 16.0 release
RUN python -m pip install -r arrow/python/requirements-wheel-build.txt --pre --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"

# ENV CLCACHE_DIR="C:\clcache"
# ENV CLCACHE_COMPRESS=1
Expand Down
5 changes: 4 additions & 1 deletion ci/scripts/python_wheel_macos_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,15 @@ echo "=== (${PYTHON_VERSION}) Install Python build dependencies ==="
export PIP_SITE_PACKAGES=$(python -c 'import site; print(site.getsitepackages()[0])')
export PIP_TARGET_PLATFORM="macosx_${MACOSX_DEPLOYMENT_TARGET//./_}_${arch}"

# TODO(GH-39848) Remove the `--pre --extra-index-url` for numpy nightly again before the 16.0 release
pip install \
--upgrade \
--only-binary=:all: \
--target $PIP_SITE_PACKAGES \
--platform $PIP_TARGET_PLATFORM \
-r ${source_dir}/python/requirements-wheel-build.txt
-r ${source_dir}/python/requirements-wheel-build.txt \
--pre \
--extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
pip install "delocate>=0.10.3"

echo "=== (${PYTHON_VERSION}) Building Arrow C++ libraries ==="
Expand Down
7 changes: 6 additions & 1 deletion python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,12 @@
[build-system]
requires = [
"cython >= 0.29.31",
"oldest-supported-numpy>=0.14",
# Starting with NumPy 1.25, NumPy is (by default) as far back compatible
# as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION
Comment on lines +21 to +22
Copy link
Member

@pitrou pitrou Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this always be the case?

Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Jan 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean exactly with "always"? For all Python versions, or all numpy versions? Or the full API?

It's a guarantee that is given by NumPy starting with version 1.25: https://numpy.org/devdocs/release/1.25.0-notes.html#compiling-against-the-numpy-c-api-is-now-backwards-compatible-by-default

(see also some details I posted on the parent issue: #39532 (comment))

Copy link
Member

@pitrou pitrou Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that oldest-supported-numpy also accounts for bugs that cropped up in the past, e.g. broken ARM compatibility.
https://github.com/scipy/oldest-supported-numpy/blob/main/setup.cfg

Copy link
Member Author

@jorisvandenbossche jorisvandenbossche Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand, oldest-supported-numpy pins to a newer numpy version for some platforms / architectures (like arm) because the older version that is theoretically supported has bugs. But so that means that if we always use the latest available version for a platform, that should be fine?

Of course, there might be a new bug occurring in the latest version, and at that point, defaulting to "the latest available version" might temporarily give issues, compared to pinning to a slightly older but "known to be OK" version (for example, if we know 1.25 is fine, we could pin to that in case the newest 1.26.x would introduce a bug).
But that's a problem for all packages building against numpy, and so it would be good if numpy would have guidelines around this. And my understanding is that it is now recommended to built against the latest numpy version (although https://numpy.org/devdocs/dev/depending_on_numpy.html#build-time-dependency is not explicit about that, it just mentions that starting from 1.25, you can use that version to built packages that are also compatible with older numpy)

cc @seberg

Copy link
Contributor

@seberg seberg Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds all exactly right.

I suppose the docs linked could recommend to compile against the latest available version (unless there be bugs). This is required for major releases, i.e. NumPy 2 (you cannot build releases against it yet since there is no rc, though. I still need to opaquify the descriptor struct a bit, which you may notice).

To continue supporting building against pre 2.0 releases, you may have to vendor (numpy/_core/include/)numpy/npy_2_compat.h.

# define). For older Python versions (where NumPy 1.25 is not yet avaiable)
# continue using oldest-support-numpy.
"oldest-supported-numpy>=0.14; python_version<'3.9'",
"numpy>=1.25; python_version>='3.9'",
"setuptools_scm < 8.0.0",
"setuptools >= 40.1.0",
"wheel"
Expand Down
3 changes: 2 additions & 1 deletion python/requirements-build.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
cython>=0.29.31
oldest-supported-numpy>=0.14
oldest-supported-numpy>=0.14; python_version<'3.9'
numpy>=1.25; python_version>='3.9'
setuptools_scm<8.0.0
setuptools>=38.6.0
3 changes: 2 additions & 1 deletion python/requirements-wheel-build.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
cython>=0.29.31
oldest-supported-numpy>=0.14
oldest-supported-numpy>=0.14; python_version<'3.9'
numpy>=1.25; python_version>='3.9'
setuptools_scm<8.0.0
setuptools>=58
wheel
2 changes: 1 addition & 1 deletion python/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -449,7 +449,7 @@ def has_ext_modules(foo):


install_requires = (
'numpy >= 1.16.6, <2',
'numpy >= 1.16.6',
)


Expand Down