[DataPipe] Adding kwargs for fs.open() in fsspec DataPipes

[NivekT]

Stack from ghstack:

• -> [DataPipe] Adding kwargs for fs.open() in fsspec DataPipes #804

Fixes #803

I left FSSpecFileLister untouched since I don't think it will be useful for fs.ls() to accept kwargs.

Differential Revision: D40038331

[NivekT]

@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[ejguan]

LGTM, thank you. And, could you please add corresponding tests, if possible?

[NivekT]

@NivekT has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[NivekT]

@dorbittonn This PR should allow you to pass version_id as part of kwargs_for_open to FSSpecFileOpener. You will need to install from source or wait for the PR to land into the nightly release.

Note that the usage of these fsspec DataPipes require the installation of fsspec and s3fs. This documentation provides more details on bucket version awareness.

We are looking into the extending the same options for S3 DataPipes but it will be low priority for us if the fsspec solution works.

[dorbittonn]

@dorbittonn This PR should allow you to pass version_id as part of kwargs_for_open to FSSpecFileOpener. You will need to install from source or wait for the PR to land into the nightly release.

Note that the usage of these fsspec DataPipes require the installation of fsspec and s3fs. This documentation provides more details on bucket version awareness.

We are looking into the extending the same options for S3 DataPipes but it will be low priority for us if the fsspec solution works.

Thanks for the tag @NivekT , but using s3fs library is not a good alternative for me, since this library is too slow.
You can observe this post on towardsdatascience examining different ways for loading from s3 using pytorch: Training in pytorch from s3 bucket

[NivekT]

@dorbittonn We have recently ran our own benchmark on AWS EC2 to load data from S3, we actually found that fsspec is faster than the S3 DataPipe implementation, especially if you use open tar archives via streaming .load_from_tar(mode="r|"). Can you please try it and let us know if you got something different?

[dorbittonn]

@dorbittonn We have recently ran our own benchmark on AWS EC2 to load data from S3, we actually found that fsspec is faster than the S3 DataPipe implementation, especially if you use open tar archives via streaming .load_from_tar(mode="r|"). Can you please try it and let us know if you got something different?

Do you have any example for streaming data from s3 using fsspec using load from tar?

[NivekT]

This is what I have used, let me know if you need more info.

dp = IterableWrapper(s3_paths)  # Replace this with a DataPipe that gives up paths to S3 archives
dp = dp.open_files_by_fsspec(mode="rb", anon=True).load_from_tar(mode="r|")  # Note that it is r| instead of r:, | enables streaming of uncompressed archives

[dorbittonn]

This is what I have used, let me know if you need more info.

dp = IterableWrapper(s3_paths)  # Replace this with a DataPipe that gives up paths to S3 archives
dp = dp.open_files_by_fsspec(mode="rb", anon=True).load_from_tar(mode="r|")  # Note that it is r| instead of r:, | enables streaming of uncompressed archives

I get
from torch.utils.data.graph import traverse_dps ImportError: cannot import name 'traverse_dps' from 'torch.utils.data.graph'

when trying to import torchdata after installing this PR

[NivekT]

In order to install TorchData from source, you will need to install PyTorch from source as well.

[ejguan]

Maybe try nightly release of torchdata? It should include nightly pytorch as well.

[ejguan]

BTW, I will cherry-pick this PR to the release branch by the end of today.

[dorbittonn]

In order to install TorchData from source, you will need to install PyTorch from source as well.

Which version of pytorch should I install? the last commit? last release of 1.12.1 is enough?
I tried to use torch 1.12.1 and installed the last commit of torchdata from source, and the same error I've mentioned appears

[ejguan]

@dorbittonn Please check this link. There is a Preview (nightly) version of PyTorch. You can install torch and torchdata preview releases.
For linux cpu: pip3 install --pre torch torhcdata --extra-index-url https://download.pytorch.org/whl/nightly/cpu

[dorbittonn]

@dorbittonn Please check this link. There is a Preview (nightly) version of PyTorch. You can install torch and torchdata preview releases. For linux cpu: pip3 install --pre torch torhcdata --extra-index-url https://download.pytorch.org/whl/nightly/cpu

I tried it (with cpu and with nightly gpu by replacing the url you gave with https://download.pytorch.org/whl/nightly/torchrec_nightly_3.8_cu11.whl/ )
And it seems that the preview(nightly) version of pytorch doesnt include this PR

thanks for your help!

[ejguan]

I tried it (with cpu and with nightly gpu by replacing the url you gave with https://download.pytorch.org/whl/nightly/torchrec_nightly_3.8_cu11.whl/ )
And it seems that the preview(nightly) version of pytorch doesnt include this PR

Emmm. If you need to install torch and torchdata nightly with cuda11. You can install via
pip install --pre torch torhcdata --extra-index-url https://download.pytorch.org/whl/nightly/cu117

Edit: And, this commit should be present in torchdata nightly. Could you provide the log when you execute pip install?

[dorbittonn]

Here's the log :
TDLOG.txt
And I'm still getting
AttributeError: 'IterableWrapperIterDataPipe' object has no attribute 'openfiles_by_fsspec

Thank you again

edit: Sorry, my typo @NivekT

[NivekT]

Here's the log : TDLOG.txt And I'm still getting AttributeError: 'IterableWrapperIterDataPipe' object has no attribute 'openfiles_by_fsspec

Thank you again

Are you missing an underscore? It should be open_files_by_fsspec