[QUESTION] Look into loading multiple datasets

[Quentin-Anthony]

We'd like to be able to load in, dedup, and shuffle multiple datasets for a training run. I believe this is as simple as passing multiple paths in the config, but I want to make this more concrete.

If it helps, I also saw: https://github.com/Zyphra/Megatron-LM/tree/main/tools/openwebtext for cleaning and dedup.

• Explore options for loading multiple datasets. Understand what exists and if we need to write anything ourselves
• Test loading/cleaning/deduping multiple datasets using our MLM fork
• Write up a small README.md on this and push to main