Activity
Update indexed_binary_mmap_reader.py
Update indexed_binary_mmap_reader.py
Merge branch 'main' into bugfix/checkpointing
Merge branch 'main' into bugfix/checkpointing
revert changes
revert changes
added preprocessing step
added preprocessing step
fixed type for overflow warning
fixed type for overflow warning
fxied deepspeed
fxied deepspeed
use mpiio to generate offsets
use mpiio to generate offsets
more accurate representation of deepspeed.
more accurate representation of deepspeed.
fix epochs
fix epochs
switch order of faster tests
switch order of faster tests
fixed calculation
fixed calculation
fixed formula
fixed formula
make size of tensor even integer.
make size of tensor even integer.
New improved modelling for LLM Deepspeed.
New improved modelling for LLM Deepspeed.
increase buffer to 16 MB
increase buffer to 16 MB
optimization for reading in deep speed framework.
optimization for reading in deep speed framework.
fix type of number for offset and size
fix type of number for offset and size
For sample indexing we fix the uneven sampling (#226)
For sample indexing we fix the uneven sampling (#226)
changed log level field to output section
changed log level field to output section
Update utility.py
Update utility.py