1-bit adam #353
Merged
1-bit adam #353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* testing_onebit * test_passed * test_passed * updated compressed_allreduce * 123 * cpu2gpu test added * Add non-cuda-aware code path. Segfaults for > 2 procs. * Works for 4 procs with numpy buffers now. TODO: cleanup, evalute perf. * Fix gather. Cleanup. * Add new tests. * Reduce memory footprint. BERT large with BS=16 works. * Revert "Reduce memory footprint. BERT large with BS=16 works." This reverts commit e7f38fc. * Update optim to support bert-large. * with initialization added on bert_onebit_adam * This works!! * Force igather for cupy. Better performance. Need cleanup and reorg to support TCP now. * X * testing the fintune task for FP32 training * Added the fintune taks for FP32 training * With the control flag of freeze_step added * added the freeze_step inside fp32_onebitadam * Seperate freeze_kernnel added * Added the sanity test for the Compressed_Allreduce * Test for Compressed_Allreduce passed, but AllGather need sync. * add checks for finetuning. * Running passed for finetune on EastUs * Add one bit adam clean file. * Refactor comms. code and move it to a new file. * fix compile/run errors. * Adding changes for onebit_adam from Hank * Save memory by modifying in-place. Co-authored-by: Your Name <[email protected]> Co-authored-by: Ammar Ahmad Awan <[email protected]> Co-authored-by: tanghl1994 <[email protected]> Co-authored-by: Hank <[email protected]> Co-authored-by: root <[email protected]>
* Refactor to correct locations. * Deleted unused files. * Fix imports for refactored codebase. * update the com reduce test. * Fix some errors. * Fix optimizer name * Delete unused tests. * Fix formatting for pre-commit. * Add cupy dependencies. * add cupy for cuda 10add cupy for cuda 10 * Add mpi4py requirement. Co-authored-by: Ammar Ahmad Awan <[email protected]>
awan-10
requested review from
arashashari,
cli99,
conglongli,
eltonzheng,
jeffra,
minjiaz,
niumanar,
RezaYazdaniAminabadi,
samyam,
ShadenSmith and
tjruwase
as code owners
jeffra
reviewed
Sep 3, 2020
jeffra
reviewed
Sep 3, 2020
jeffra
reviewed
Sep 3, 2020
… importing mpi4py
niumanar
reviewed
Sep 4, 2020
|
Marked PR as a draft until we are ready to merge. |
jeffra
reviewed
Sep 8, 2020
jeffra
reviewed
Sep 8, 2020
jeffra
reviewed
Sep 8, 2020
jeffra
approved these changes
Sep 8, 2020
jeffra
added a commit
that referenced
this pull request
Sep 9, 2020
* 1-bit adam (#353) Co-authored-by: Jeff Rasley <[email protected]> Co-authored-by: Your Name <[email protected]> Co-authored-by: tanghl1994 <[email protected]> Co-authored-by: Hank <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Ammar Ahmad Awan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.