-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add nightly build and upload for torchcodec #5814
Conversation
@ahmadsharif1 is attempting to deploy a commit to the Meta Open Source Team on Vercel. A member of the Team first needs to authorize it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Per our chat, landing this will start building torchcodec nightly and upload to S3. Once this lands, we can wait a day to manually verify if the file is there. Then we can update https://github.com/pytorch/test-infra/blob/main/s3_management/manage.py#L172 to update the indexing script to expose torchcode nightly binary.
Confirm that the S3 path is correct, i.e. the same path used by vision https://github.com/pytorch/vision/actions/runs/11498452468/job/32005357656#step:10:130 |
@NicolasHug was there a reason not to merge this? We eventually do want cuda wheels to be uploaded to nightly on a regular basis. |
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree we want nightly wheels!
We build wheels on every single PR, what I think you heard from me in the past is that we do not want those PR wheels to be uploaded as "nightly" (this is why we ruled out the "use the validation framework" suggestion during our discussions about the MacOS setup) :)
But uploading nightly wheels on a nightly basis - yes, we want that.
As far as I can tell from
test-infra/.github/workflows/_binary_upload.yml
Lines 110 to 113 in 61e6da7
AWS_CMD="aws s3 cp --dryrun" | |
if [[ "${NIGHTLY_OR_TEST:-0}" == "1" ]]; then | |
AWS_CMD="aws s3 cp" | |
fi |
we run no risk of uploading PR wheels, because the --dry-run
flag is only removed on a nightly tag. Can you confirm my understanding @huydhn ?
@huydhn the Vercel job needs to be re-run because I rebased. And since @NicolasHug was okay with it, once you approve it you can also perform the merge. |
Yup, @ahmadsharif1 and I confirmed that yesterday where the wheels from pytorch/torchcodec#272 weren't upload to S3. On the other hand, once this lands and the nightly trigger runs, those wheels will be uploaded. |
Ah, let me also create nightly branch on https://github.com/pytorch/torchcodec and add it into the list of protected branches |
|
My miss in reviewing #5814, having torchcodec in the list of options will allow folks to test the nightly build manually.
#5814 adds torchcodec nightly wheel upload to test-infra. This PR adds it to the index.
We want torchcodec's mac, cuda and cpu wheels to be automatically uploaded to download.pytorch.org.
Currently we are building the wheels on every PR but not actually uploading them as they are in dry-run mode. Example for cuda we do this:
https://github.com/pytorch/torchcodec/actions/runs/11485682426/job/31966672538?pr=272#step:10:97
@huydhn can you confirm that the dry-run is copying to the correct path for this cuda wheel?