Add nightly build and upload for torchcodec

[ahmadsharif1]

We want torchcodec's mac, cuda and cpu wheels to be automatically uploaded to download.pytorch.org.

Currently we are building the wheels on every PR but not actually uploading them as they are in dry-run mode. Example for cuda we do this:

https://github.com/pytorch/torchcodec/actions/runs/11485682426/job/31966672538?pr=272#step:10:97

@huydhn can you confirm that the dry-run is copying to the correct path for this cuda wheel?

[vercel]

@ahmadsharif1 is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

[huydhn]

LGTM! Per our chat, landing this will start building torchcodec nightly and upload to S3. Once this lands, we can wait a day to manually verify if the file is there. Then we can update https://github.com/pytorch/test-infra/blob/main/s3_management/manage.py#L172 to update the indexing script to expose torchcode nightly binary.

[huydhn]

Confirm that the S3 path is correct, i.e. the same path used by vision https://github.com/pytorch/vision/actions/runs/11498452468/job/32005357656#step:10:130

[ahmadsharif1]

@NicolasHug was there a reason not to merge this? We eventually do want cuda wheels to be uploaded to nightly on a regular basis.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
torchci	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 25, 2024 7:20pm

[NicolasHug]

I agree we want nightly wheels!

We build wheels on every single PR, what I think you heard from me in the past is that we do not want those PR wheels to be uploaded as "nightly" (this is why we ruled out the "use the validation framework" suggestion during our discussions about the MacOS setup) :)

But uploading nightly wheels on a nightly basis - yes, we want that.

As far as I can tell from

test-infra/.github/workflows/_binary_upload.yml

Lines 110 to 113 in 61e6da7

	AWS_CMD="aws s3 cp --dryrun"
	if [[ "${NIGHTLY_OR_TEST:-0}" == "1" ]]; then
	AWS_CMD="aws s3 cp"
	fi

we run no risk of uploading PR wheels, because the --dry-run flag is only removed on a nightly tag. Can you confirm my understanding @huydhn ?

[ahmadsharif1]

@huydhn the Vercel job needs to be re-run because I rebased. And since @NicolasHug was okay with it, once you approve it you can also perform the merge.

[huydhn]

we run no risk of uploading PR wheels, because the --dry-run flag is only removed on a nightly tag. Can you confirm my understanding @huydhn ?

Yup, @ahmadsharif1 and I confirmed that yesterday where the wheels from pytorch/torchcodec#272 weren't upload to S3.

On the other hand, once this lands and the nightly trigger runs, those wheels will be uploaded.

[huydhn]

Ah, let me also create nightly branch on https://github.com/pytorch/torchcodec and add it into the list of protected branches

[huydhn]

• Nightly branch created
• Add nightly branch protection to only allow repo owners and pytorchbot to push to it
• Add pytorchbot to pytorch/torchcodec repo as a contributor