Moved data shuffling to just before tokenization step

[levmckinney]

This is particularly useful for datasets like togethercomputer/RedPajama-Data-1T-Sample that do not come preshuffled. Often the local shuffling done by the dataloader is insufficient for datasets this large.

[codecov]

Codecov Report

Merging #110 (7b3172c) into main (51e988b) will decrease coverage by 0.05%.
Report is 6 commits behind head on main.
The diff coverage is 71.42%.

❗ Current head 7b3172c differs from pull request most recent head 21db0bf. Consider uploading reports for the commit 21db0bf to get more accurate results

@@            Coverage Diff             @@
##             main     #110      +/-   ##
==========================================
- Coverage   81.08%   81.04%   -0.05%     
==========================================
  Files          32       32              
  Lines        2141     2147       +6     
==========================================
+ Hits         1736     1740       +4     
- Misses        405      407       +2     

Files Changed	Coverage Δ
tuned_lens/scripts/ingredients.py	85.56% <71.42%> (-0.63%)	⬇️

[AdamGleave]

LGTM.

I do want to flag one possible change in default behaviour. IIUC previously you always shuffled after tokenization, and never before tokenizaiton. Now you never shuffle after tokenization, and by default do not shuffle before tokenization. So we have gone from default to some kind of shuffling to default to no kind of shuffling.

I don't have enough context on the codebase to know if this is desirable or not, but do want to make sure it's an intentional change, and you may want to consider documenting it.