Skip to content

Activity

Update indexed_binary_mmap_reader.py

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • 2ffa7cb…ecc384a • 
on Oct 30, 2024

Update overview.rst (#228)

Pull request merge
zhenghh04pushed 1 commit to master • 866828c…64d8f39 • 
on Oct 30, 2024

Change max to abs for preprocess time (#240)

Pull request merge
zhenghh04pushed 1 commit to main • c9225fb…8614311 • 
on Oct 30, 2024

enable option to disable pin_memory in pytorch (#239)

Pull request merge
zhenghh04pushed 1 commit to main • cc5abbc…c9225fb • 
on Oct 30, 2024

fix wrong tracing location of fetch data (#238)

Pull request merge
zhenghh04pushed 1 commit to main • cda4df3…cc5abbc • 
on Oct 30, 2024

Merge branch 'main' into bugfix/checkpointing

hariharan-devarajanpushed 6 commits to bugfix/checkpointing • 85f8490…2ffa7cb • 
on Oct 30, 2024

fix last step is not executed (#236)

Pull request merge
zhenghh04pushed 1 commit to main • aed80d4…cda4df3 • 
on Oct 17, 2024

Fix wrong configuration for hdf5 chunking (#237)

Pull request merge
zhenghh04pushed 1 commit to main • edb4b27…aed80d4 • 
on Oct 17, 2024

Bugfix: fix type of number for offset and size (#229)

Pull request merge
zhenghh04pushed 1 commit to main • b95fb26…edb4b27 • 
on Oct 17, 2024

fix negative value of computation time when stdev exists (#233)

Pull request merge
zhenghh04pushed 1 commit to main • 05d67d1…b95fb26 • 
on Oct 16, 2024

fix misleading generator message (#231)

Pull request merge
zhenghh04pushed 1 commit to main • 3732663…05d67d1 • 
on Oct 16, 2024

revert changes

hariharan-devarajanpushed 1 commit to bugfix/index_binary_type • 90d812c…35b0554 • 
on Oct 16, 2024

added preprocessing step

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • b05eb73…85f8490 • 
on Oct 11, 2024

fixed type for overflow warning

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • 6b50605…b05eb73 • 
on Oct 10, 2024

fxied deepspeed

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • 7104043…6b50605 • 
on Oct 10, 2024

use mpiio to generate offsets

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • aabe149…7104043 • 
on Oct 10, 2024

more accurate representation of deepspeed.

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • b4000d3…aabe149 • 
on Oct 10, 2024

fix epochs

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • acc2b4f…b4000d3 • 
on Oct 10, 2024

switch order of faster tests

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • b557f08…acc2b4f • 
on Oct 5, 2024

fixed calculation

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • c412725…b557f08 • 
on Oct 5, 2024

fixed formula

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • 9f12a50…c412725 • 
on Oct 5, 2024

make size of tensor even integer.

hariharan-devarajanpushed 1 commit to bugfix/checkpointing • b2bf926…9f12a50 • 
on Oct 5, 2024

New improved modelling for LLM Deepspeed.

hariharan-devarajancreated bugfix/checkpointing • b2bf926 • 
on Oct 5, 2024

increase buffer to 16 MB

hariharan-devarajanpushed 1 commit to bugfix/index_binary_type • 4e63674…90d812c • 
on Sep 30, 2024

optimization for reading in deep speed framework.

hariharan-devarajanpushed 1 commit to bugfix/index_binary_type • 6061115…4e63674 • 
on Sep 30, 2024

fix type of number for offset and size

hariharan-devarajanpushed 1 commit to bugfix/index_binary_type • 3732663…6061115 • 
on Sep 30, 2024

For sample indexing we fix the uneven sampling (#226)

hariharan-devarajancreated bugfix/index_binary_type • 3732663 • 
on Sep 30, 2024

For sample indexing we fix the uneven sampling (#226)

Pull request merge
zhenghh04pushed 1 commit to main • bbe9bae…3732663 • 
on Sep 24, 2024

changed log level field to output section

zhenghh04pushed 1 commit to feature/log • b9bc737…ad0ea30 • 
on Sep 4, 2024

Update utility.py

zhenghh04pushed 1 commit to feature/log • 140d86c…b9bc737 • 
on Sep 3, 2024