AttributeError: 'SequenceWrapperMapDataPipe' object has no attribute 'prefetch . when using MultiProcessingReadingService

[One-sixth]

🐛 Describe the bug

from torchdata.datapipes.map import SequenceWrapper
from torchdata.dataloader2 import DataLoader2
from torchdata.dataloader2 import MultiProcessingReadingService


class SimpleDataset:
    def __len__(self):
        return 1000

    def __getitem__(self, i):
        if i >= len(self): raise StopIteration()
        return i


if __name__ == '__main__':
    ds = SimpleDataset()
    warp_ds = SequenceWrapper(ds)

    rs = MultiProcessingReadingService(num_workers=2)
    # no error
    # dl = DataLoader2(warp_ds)
    # raise error
    dl = DataLoader2(warp_ds, reading_service=rs)

    for x in dl:
        print(x)

Versions

torchdata nightly

[NivekT]

The issue is that DataLoader2 attempts to apply Prefetch, even though Prefetcher only works with IterDataPipe.

A temporary work around is to use .shuffle() or .to_iterdatapipe() prior to passing it into DataLoader2.

I see a few potential solution to this:

1. Always convert MapDataPipe to IterDataPipe within DataLoader2
   • I think this will not cause any loss in functionality because the end goal of DataLoader2 is to return the dataset in some order (e.g. sequential or shuffled). If anyone has any concern with this, feel free to comment.
2. Add prefetch support for MapDataPipe

[One-sixth]

@NivekT Thank you reply. I have successfully used it.
If the error message is more easier to understand, or if it is automatically converted to IterDataPipe, it would be even better.