You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The issue is that DataLoader2 attempts to apply Prefetch, even though Prefetcher only works with IterDataPipe.
A temporary work around is to use .shuffle() or .to_iterdatapipe() prior to passing it into DataLoader2.
I see a few potential solution to this:
Always convert MapDataPipe to IterDataPipe within DataLoader2
I think this will not cause any loss in functionality because the end goal of DataLoader2 is to return the dataset in some order (e.g. sequential or shuffled). If anyone has any concern with this, feel free to comment.
@NivekT Thank you reply. I have successfully used it.
If the error message is more easier to understand, or if it is automatically converted to IterDataPipe, it would be even better.
…fault (#1146)
Summary:
Pull Request resolved: #1146Fixes#1143
I believe since `DataLoader2` would always call `iter(map_data_pipe)` at some point anyway, it should make no different to perform the conversion with `to_iter_datapipe()` within `__init__`.
This will also allow prefetchers to be attached to the DataPipe.
Please let me know if this may be BC-breaking.
Test Plan: Imported from OSS
Reviewed By: ejguan
Differential Revision: D45452068
Pulled By: NivekT
fbshipit-source-id: 608dc000735b16641e41e442553fe9283a1b8829
🐛 Describe the bug
Versions
torchdata nightly
The text was updated successfully, but these errors were encountered: