adding aistore datapipe

[gaikwadabhishek]

Fixes #{517}

Changes

• Added aisio.py (Iterable Datapipe for AIStore backends)
• Added unit tests in test/test_local_io.py test/test_aistore.py
• Added GitHub action for running AIStore in CI.yml .github/aistore_ci.yml workflow

Questions to maintainers

• We are unsure about the documentation generated on PyTorch to refer it in README.md, so I have tentatively added a URL similar to s3io functions. (see torchdata/datapipes/iter/load/README.md)

Signed-off-by: Abhishek Gaikwad [email protected]

[facebook-github-bot]

Hi @gaikwadabhishek!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[NivekT]

This PR mostly looks good to me. I will let @ejguan comment on the proper CI setup.

Question: Are there AWS/GCP/Azure remote setups that your library currently test against for every build? We might want to something similar here for different DataPipes (not just aistore).

[gaikwadabhishek]

Question: Are there AWS/GCP/Azure remote setups that your library currently test against for every build? We might want to something similar here for different DataPipes (not just aistore).

Hi @NivekT
Yes, we are testing on AWS, GCP, HDFS, and K8s on GitLab pipeline. It's pretty straightforward for us as AIStore uses existing backend connectors, we just have to provide the credentials for the cloud providers for the test.
Reference: https://github.com/NVIDIA/aistore/blob/master/.gitlab-ci.yml#L200

[msaroufim]

This PR mostly looks good to me. I will let @ejguan comment on the proper CI setup.

Question: Are there AWS/GCP/Azure remote setups that your library currently test against for every build? We might want to something similar here for different DataPipes (not just aistore).

Let's talk about this offline more @NivekT I was thinking of something similar for benchmarks where I can setup some machine configuration an instance type with some specific configuration for example HDD vs SSD and then run a benchmark script directly on it. There's a few options like ray up, cloudformation, ec2 launch, eks, teraform and I'm in the process of figuring out what's most convenient. In your case is the multi cloud requirement a must? I can send you a POC offline.

[ejguan]

Thank you for your contribution. Generally looks good to me. And, could you please sign the CLA before we can merge the PR?

[gaikwadabhishek]

Thank you for your contribution. Generally looks good to me. And, could you please sign the CLA before we can merge the PR?

Hi @ejguan @NivekT
I am still waiting for my FB CLA to be approved. I work for NVIDIA and I've been added to the corporate CLA by my employer. We are waiting for the changes to reflect here. We have also mailed to [email protected] regarding this and are waiting for their reply.

[gaikwadabhishek]

@ejguan fixed the mypi.ini configuration to ignore the missing import of aistore.. Can you please retrigger the workflow

[ejguan]

FYI: The failures from TorchVision CI are unrelated to your PR.

[msaroufim]

@gaikwadabhishek your CLA application should have been approved just retrigger CI with an empty commit and the Facebook CLA check should pass

[msaroufim]

@zpao as FYI looks like CLA Check is still red

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[gaikwadabhishek]

Hi @ejguan @NivekT @msaroufim,
Thanks for your support.
CLA is approved now. Can we merge this?

[facebook-github-bot]

@msaroufim has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.