[Python] Uploading nightly wheels to the scientific python nightly channel

[jarrodmillman]

Describe the enhancement requested

It looks like you already build nightly Python wheels:

• https://github.com/apache/arrow/blob/main/docs/source/developers/python.rst#installing-nightly-packages

Would you be interested/willing to upload nightly wheels to he scientific-python nightly channel instead:

• https://anaconda.org/scientific-python-nightly-wheels/

This would make the wheels easier to discover by hosting them in a "standard" place. It will also make it easier for downstream projects to test against multiple packages, since they won't need to specify multiple locations in their CI scripts.

You can read more about why and how we are building / testing against nightly wheels here:
https://scientific-python.org/specs/spec-0004/

We also provide a GitHub action to upload and manage nightly builds:
https://github.com/scientific-python/upload-nightly-action/

We @matthewfeickert, @larsoner, @tupui are happy to help.

Component(s)

Python

Linked PRs

• GH-40216: [CI][Packaging][Python] Upload pyarrow nightly wheels to scientific python channel on Anaconda #43862
• GH-40216: [Python][CI][Packaging] Upload nightly wheels to main label of scientific-python-nightly-wheels channel #43932
• GH-40216: [Python][CI][Packaging] Don't upload sdist to scientific-python nightly channel (only wheels) #43943

[jorisvandenbossche]

cc @raulcd

This sounds as an interesting idea to me. My understanding is that our nightly wheels are not "official" release artifacts anyway, so it shouldn't matter too much where it is hosted? (also not sure if the current hosting location is actually tied to the Apache foundation)

I suppose the main cost is the cost of switching for current users of this (and for them to first find out that the old link no longer gives them the latest packages, at the time we would decide to stop uploading there as well)

[matthewfeickert]

👋 @jorisvandenbossche Just wanted to see if the team had any further thoughts on this. Let us know if we can help answer any questions.

[raulcd]

We probably could upload wheels to both places. In terms of discoverability we have to advise to users that the wheels provided are not official releases and should be used at their own risk. Similar to what we have at the documentation:
These packages are not official releases. Use them at your own risk.
This is because the Apache Release Policy (https://www.apache.org/legal/release-policy.html#what) says:

During the process of developing software and preparing a release, various packages are made available to the developer community for testing purposes. Do not include any links on the project website that might encourage non-developers to download and use nightly builds, snapshots, release candidates, or any other similar package. The only people who are supposed to know about such packages are the people following the dev list (or searching its archives) and thus aware of the conditions placed on the package. If you find that the general public are downloading such test packages, then remove them.

I think we are ok publishing the nightly wheels under the scientific python nightly channel.

[matthewfeickert]

I think we are ok publishing the nightly wheels under the scientific python nightly channel.

@raulcd Great! To move forward we need someone in the Arrow organization to do the following:

• Create an account on Anaconda Cloud
• Reply here with the link to your account (https://anaconda.org/ktou/)
• Make a new Anaconda Cloud API token with permissions described in https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects

• Save the API token as a repo GitHub Actions secret named ANACONDA_ORG_UPLOAD_TOKEN
• Direct me (@matthewfeickert) to a pyarrow nightly wheel for me to upload to create the package in the registry
• Verify that I have added you to the pyarrow team "Group" on https://anaconda.org/scientific-python-nightly-wheels (I've already added @jorisvandenbossche)
• Verify that I have added the package scientific-python-nightly-wheels/pyarrow to the pyarrow team "Group"

If you'd also like to use the scientific/upload-nightly-action GitHub Action please check out the action repo's README. If you need to collect wheels from different CI jobs, consider creating a new GitHub Actions workflow with logic similar to matplotlib's or Awkward's.

[kou]

Could you add me https://anaconda.org/ktou/ to the pyarrow team "Group" too?

[matthewfeickert]

@kou done. As soon as you generate an API token you can start uploading to https://anaconda.org/scientific-python-nightly-wheels/pyarrow. 👍 Let's leave this Issue open though until the first automated CI/CD upload passes.

[kou]

Thanks! I confirmed that I'm a member of the group.

@assignUser We need to ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN secret to our GitHub Actions. Could you also join the group, generate your API token and ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN?

[matthewfeickert]

Could you also join the group, generate your API token and ask INFRA to set ANACONDA_ORG_UPLOAD_TOKEN?

@assignUser Can you also share your Anaconda Cloud account name so I can add you to the pyarrow group?

[assignUser]

@matthewfeickert Sorry I missed the notifications 🙈 https://anaconda.org/assignuser

[matthewfeickert]

@assignUser added. 👍

[assignUser]

I'll take care of adding the token with infra

[assignUser]

See https://issues.apache.org/jira/browse/INFRA-25697

[assignUser]

@kou infra just added the secret so we can create a workflow (will close the ticket once we can confirm it works)

[kou]

Thanks!

[assignUser]

INFRA will add the secret but with a ARROW_ prefix for organizational reasons, as we have to pull it into an envar anyway it shouldn't matter.

[matthewfeickert]

INFRA will add the secret but with a ARROW_ prefix for organizational reasons

That's totally fine. The secret name is arbitrary even if the GitHub Actions uploaded is used, but I just suggest a name to keep it simple for people. 👍

[kou]

OK. What is the next action then? We need to update our workflows?

[assignUser]

@kou As far as I understand we need to create a workflow similiar to r_nightly.yml that collects the pyarrow wheels from crossbow and then uploads them to the anaconda channel using the new secret.

[matthewfeickert]

we need to create a workflow similar to r_nightly.yml that collects the pyarrow wheels from crossbow and then uploads them to the anaconda channel using the new secret.

Yeah. #40216 (comment) has some linked examples of how other projects are doing this.

[kou]

I see! Thanks!

[matthewfeickert]

👋 @kou @assignUser I just wanted to follow up on this to see if there are any questions on your end RE: getting deployments up.

[kou]

@assignUser Can you work on this? Or should someone work on this?

[assignUser]

@kou I won't get to it quickly but I can take a look 'soon'(tm), maybe @matthewfeickert would be interested in working on it? The way we generate the wheels is a bit complex but it seems you have looked into that already?

[matthewfeickert]

@assignUser Realistically I don't have time at the moment, given that summer is peak everything time for academia.

[jorisvandenbossche]

As far as I understand we need to create a workflow similiar to r_nightly.yml that collects the pyarrow wheels from crossbow and then uploads them to the anaconda channel using the new secret.

For uploading the wheels, we currently have the github_upload_gemfury github actions macro, that is used to upload the built nightly wheels directly from the jobs building and testing them, for example at

arrow/dev/tasks/python-wheels/github.linux.yml

Line 108 in 712cfe6

{{ macros.github_upload_gemfury("arrow/python/repaired_wheels/*.whl")|indent }}

We could do this with a similar approach adding an extra macro to upload the wheels to anaconda.org?
Or @assignUser is there a reason for (or advantage to) using an approach like r_nightly.yml collecting all artifacts and uploading from a run on this repo? (the file mention "Due to authorization requirements, this upload can't be done from the crossbow repository.", but is that because this one is syncing with nightlies.apache.org? Uploading to gemfury doesn't seem to have that issue)

[assignUser]

Issue resolved by pull request 43862
#43862

[matthewfeickert]

I haven't looked at this closely and will be offline tomorrow for travel but while there are files (v18.0.0.dev236) on the index (https://anaconda.org/scientific-python-nightly-wheels/pyarrow/files), their metadata isn't getting properly picked up:

$ docker run --rm -ti python:3.12 /bin/bash
root@2131275711b3:/# python -m venv venv && . venv/bin/activate
(venv) root@2131275711b3:/# python -m pip index --pre --index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple versions pyarrow
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
ERROR: No matching distribution found for pyarrow
(venv) root@2131275711b3:/# python -m pip install --pre --upgrade --index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pyarrow
Looking in indexes: https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
ERROR: Could not find a version that satisfies the requirement pyarrow (from versions: none)
ERROR: No matching distribution found for pyarrow

[jorisvandenbossche]

I see the same issue. The wheels itself look fine though (also when downloaded and pointing pip directly to it, that installs fine). So wondering if this is something on the server side or related to how we upload them.

Now, the first nightly run of the packaging builds that will now automatically upload them still has to run (they should be scheduled shortly I think). So the current files were uploaded somewhere throughout the PR (although I wouldn't think this should make any difference).

[jorisvandenbossche]

Ah, this is because we are uploading the wheels to the dev label, instead of main.
Explicitly pointing to the equivalent url (--index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/label/dev/simple, as instructed by https://anaconda.org/scientific-python-nightly-wheels/pyarrow) does work.

All other packages are being uploaded to main (and the full channel is meant for nightly wheel, so there is no need to distinguish main vs dev, I suppose), so will correct that to also upload to main.

[jorisvandenbossche]

Something else: we might also want to consider to not upload the nightly sdist. While it can be useful to upload this somewhere, there are some drawbacks in uploading it the same index for the nightly wheels.
Especially when some of the nightly wheel builds fail for whatever reason, but the sdist is uploaded, and then a downstream project installing pyarrow from the nightly wheels might get the sdist (which will almost always fail) as latest version instead of the wheel of the day(s) before (I think downstream projects could solve that by using --prefer-binary flag to pip, but that is easy to forget)

[raulcd]

Some of the nightly wheels have also failed to upload similar to the failures that we found on the latest PR from @jorisvandenbossche :
wheel-macos-monterey-cp310-amd64
wheel-windows-cp310-amd64
wheel-windows-cp313-amd64
wheel-windows-cp39-amd64
Those errors might have been temporary network errors but we should monitor and maybe add some retry mechanism to the upload if this is common.

[matthewfeickert]

Quick comments as I'm in transit (sorry):

• yes, if you want the dev label please add that in addition to main.

c.f. https://github.com/scientific-python/upload-nightly-action/blob/b3d150d89b9500cd5623f211d6214f1d8b50fac5/action.yml#L21-L24 and https://github.com/scientific-python/upload-nightly-action/blob/b3d150d89b9500cd5623f211d6214f1d8b50fac5/cmd.sh#L44-L53

• In general we'd prefer to only upload wheels so that people using the index for testing don't accidentally run into situations in which they trigger a build from sdist.

[jorisvandenbossche]

• In general we'd prefer to only upload wheels so that people using the index for testing don't accidentally run into situations in which they trigger a build from sdist.

Opened a PR to remove the sdist uploading: #43943

[raulcd]

Issue resolved by pull request 43943
#43943

[raulcd]

With the latest changes I've closed the issue as done. Please re-open or open a new issue if any follow up is necessary.
Thanks @jorisvandenbossche for following this up!