Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python][Packaging] Support Python 3.13 and upload wheels #43519

Closed
4 of 5 tasks
Tracked by #2518 ...
jorisvandenbossche opened this issue Aug 1, 2024 · 13 comments
Closed
4 of 5 tasks
Tracked by #2518 ...

[Python][Packaging] Support Python 3.13 and upload wheels #43519

jorisvandenbossche opened this issue Aug 1, 2024 · 13 comments
Assignees
Milestone

Comments

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Aug 1, 2024

Python 3.13 is currently in a prerelease status (a first release candidate was released today: https://discuss.python.org/t/python-3-13-0-release-candidate-1-released/59703) and is expected to have a final release in October this year (https://devguide.python.org/versions).

We should ensure we are fully ready to support Python 3.13 for the PyArrow 18 release that will happen around the same time (or potentially even sooner if we would have a 17.0.x bug fix release).

#42034 already fixed a build issue with Python 3.13, and on the short term I think we should try to have 1) a CI build using Python 3.13 to ensure this keeps working, and 2) set up a wheel build so we can upload nightly wheels for Python 3.13 for downstream testing.

(equivalent PR for Python 3.12: #37901)


@raulcd
Copy link
Member

raulcd commented Aug 1, 2024

I was planning, if necessary, to delay the feature freeze for 18.0.0 for a week to have Python 3.13 wheels.

@jorisvandenbossche
Copy link
Member Author

We can actually make wheels right now (and we should ideally produce nightly wheels well before the scheduled release of 18.0), because Python 3.13 is ABI stable now, and wheels built with the current Python 3.13 will work with the final 3.13 releases.

(so there is certainly no need to delay 18.0.0 AFAIK)

@jorisvandenbossche
Copy link
Member Author

jorisvandenbossche commented Aug 14, 2024

Follow-ups after #43539:

  • Also add py 3.13 conda build once that is supported in conda-forge (

    arrow/dev/tasks/tasks.yml

    Lines 1200 to 1201 in 712cfe6

    {% for python_version in ["3.8", "3.9", "3.10", "3.11", "3.12"] %}
    test-conda-python-{{ python_version }}:
    )
  • Update requirements-wheel-build.txt and requirements-wheel-test.txt to install released versions
  • Update the places where we currently use "3.13-rc" or "rc1" or "3.13.0-rc1" (dev/tasks/python-wheels/github.linux.yml, ci/scripts/install_python.sh, the windows docker files)
  • Enable GCS testing once google-testbench (grpcio) can be installed (update install_gcs_testbench.sh)

EDIT: added those to the top post

jorisvandenbossche added a commit that referenced this issue Aug 22, 2024
### Rationale for this change

Like #43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels).

### What changes are included in this PR?

I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as #37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well.
* GitHub Issue: #43519

Lead-authored-by: Lysandros Nikolaou <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
mapleFU pushed a commit to mapleFU/arrow that referenced this issue Sep 3, 2024
…e#43539)

### Rationale for this change

Like apache#43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels).

### What changes are included in this PR?

I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as apache#37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well.
* GitHub Issue: apache#43519

Lead-authored-by: Lysandros Nikolaou <[email protected]>
Co-authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Sep 23, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Sep 23, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Sep 24, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Oct 2, 2024
@jarrodmillman
Copy link

Python 3.13 is scheduled for Monday (10/7). Would it be possible to make a 18.0.0b0 or 18.0.0rc0 release on PyPI in the near future? I know you have wheels on https://anaconda.org/scientific-python-nightly-wheels/pyarrow/files, but having a pre-release on PyPI would make it easier for projects that aren't aware of the "nightly" wheels.

@wizard86pz
Copy link

I've installed today the v3.13 of Python, and it's affected by this pyarrow issue.
Should I wait for a new release of python 3.13 to have the fix?

@pitrou
Copy link
Member

pitrou commented Oct 9, 2024

According to https://discuss.python.org/t/launch-day-availability-of-python-3-13-in-conda-forge/67124/1, 3.13 is now available in conda-forge.

@pitrou
Copy link
Member

pitrou commented Oct 9, 2024

I've installed today the v3.13 of Python, and it's affected by this pyarrow issue.
Should I wait for a new release of python 3.13 to have the fix?

Yes, please wait for 18.0.0 which will provide binary wheels for 3.13.

jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Oct 10, 2024
raulcd added a commit to raulcd/arrow that referenced this issue Oct 10, 2024
raulcd pushed a commit that referenced this issue Oct 10, 2024
…test wheels on Python 3.13 (#44193)

### Rationale for this change

Now that numpy, cython and pandas all have releases that support Python 3.13, we can remove the extra index and ensure that we are building our wheels with the released version instead of nightly dev version.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
raulcd pushed a commit that referenced this issue Oct 10, 2024
…test wheels on Python 3.13 (#44193)

### Rationale for this change

Now that numpy, cython and pandas all have releases that support Python 3.13, we can remove the extra index and ensure that we are building our wheels with the released version instead of nightly dev version.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
raulcd added a commit to raulcd/arrow that referenced this issue Oct 10, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Oct 10, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Oct 10, 2024
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Oct 10, 2024
raulcd pushed a commit that referenced this issue Oct 11, 2024
### Rationale for this change

When adding Python 3.13 support (#43539), conda was not yet ready to support Python 3.13. But over the last weeks most conda packages having being updated, so trying again.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
raulcd pushed a commit that referenced this issue Oct 11, 2024
### Rationale for this change

When adding Python 3.13 support (#43539), conda was not yet ready to support Python 3.13. But over the last weeks most conda packages having being updated, so trying again.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
@raulcd
Copy link
Member

raulcd commented Oct 13, 2024

The only missing task if to update to the final released version of Python (we are just waiting for chocolatey or to migrate out of chocolatey).
I am closing this as done as I am close to create the first RC for 18.0.0.
Feel free to open a subtask for:

  • Update the places where we currently use "3.13-rc" or "rc1" or "3.13.0-rc1" (dev/tasks/python-wheels/github.linux.yml, ci/scripts/install_python.sh, the windows docker files)

@raulcd raulcd closed this as completed Oct 13, 2024
@bersbersbers
Copy link

we are just waiting for chocolatey or to migrate out of chocolatey

Wow - I was not aware it takes chocolatey more than a week to update the latest stable Python version. Considering that there are packages whose 3.13 support depends on arrow's, maybe you want to consider scoop in the future, which updated a week ago: ScoopInstaller/Main@ec70b27

@pllim
Copy link

pllim commented Oct 14, 2024

Given this issue is closed, where should I subscribe to make sure I am aware when pyarrow that is compatible is Python 3.13 is actually released? Thanks!

@edgarrmondragon
Copy link

Given this issue is closed, where should I subscribe to make sure I am aware when pyarrow that is compatible is Python 3.13 is actually released? Thanks!

@pllim you can subscribe to the PyPI feed: https://pypi.org/rss/project/pyarrow/releases.xml

@raulcd
Copy link
Member

raulcd commented Oct 28, 2024

Wheels for Python 3.13 have been uploaded to PyPI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants