You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The tutorial needs to update the actual reason about shuffling before sharding. It's not accurate.
Shuffling before sharding is required to achieve global shuffling rather than only shuffling inside each shard.
Suggest a potential alternative/fix
No response
The text was updated successfully, but these errors were encountered:
In order for DataPipe sharding to work with DataLoader, we need to add the following. It is crucial to add ShardingFilter after Shuffler to ensure that all worker processes have the same order of data for sharding.
📚 The doc issue
The tutorial needs to update the actual reason about shuffling before sharding. It's not accurate.
Shuffling before sharding is required to achieve global shuffling rather than only shuffling inside each shard.
Suggest a potential alternative/fix
No response
The text was updated successfully, but these errors were encountered: