Increase DALI adoption by providing a PyTorch dataset wrapper

[FuriouslyCurious]

Hi DALI team,

Thank you for a wonderfully performant library! I think there is a way to dramatically boost DALI adoption with a couple of pull requests.

PyTorch TorchVision provides many dataset loaders here:
https://pytorch.org/docs/stable/torchvision/datasets.html

TorchVision ImageFolder dataset is the closest to DALI interface. Creating "DALI" loader through a pull request will make this library much easier to use for the PyTorch ecosystem.
ImageFolder code begins here:
https://github.com/pytorch/vision/blob/2f1399e86309fa3c286771176c017fdfaf3ce00e/torchvision/datasets/__init__.py

How do you feel about this idea? Thanks for the feedback!

[jantonguirao]

Hi @FuriouslyCurious and thanks for your suggestion. I am not sure I understood it fully so please correct me if I got it wrong.

Do you mean, create a DALI wrapper in the PyTorch project?
That is basically the idea of nvidia.dali.plugin.pytorch.DALIGenericIterator which just replaces your PyTorch data loader completely. The specifics of your data loading and processing go inside the DALI pipeline definition.
https://docs.nvidia.com/deeplearning/dali/user-guide/docs/examples/frameworks/pytorch/pytorch-basic_example.html

Please provide more information in case you meant something different.

[JanuszL]

If your ask is about making a dedicated set of predefined DALI pipelines inside the PyTorch repository then it sounds like a good idea. However we don't have the resources now to do so, we might do this in the future. However, we would be more than happy to see and review a community originated PR integrating DALI into PyTorch more tightly.