Add 1-bit Adam support to DeepSpeed #380
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* 1-bit Adam v1 (squash) (#346) * testing_onebit * test_passed * test_passed * updated compressed_allreduce * 123 * cpu2gpu test added * Add non-cuda-aware code path. Segfaults for > 2 procs. * Works for 4 procs with numpy buffers now. TODO: cleanup, evalute perf. * Fix gather. Cleanup. * Add new tests. * Reduce memory footprint. BERT large with BS=16 works. * Revert "Reduce memory footprint. BERT large with BS=16 works." This reverts commit e7f38fc. * Update optim to support bert-large. * with initialization added on bert_onebit_adam * This works!! * Force igather for cupy. Better performance. Need cleanup and reorg to support TCP now. * X * testing the fintune task for FP32 training * Added the fintune taks for FP32 training * With the control flag of freeze_step added * added the freeze_step inside fp32_onebitadam * Seperate freeze_kernnel added * Added the sanity test for the Compressed_Allreduce * Test for Compressed_Allreduce passed, but AllGather need sync. * add checks for finetuning. * Running passed for finetune on EastUs * Add one bit adam clean file. * Refactor comms. code and move it to a new file. * fix compile/run errors. * Adding changes for onebit_adam from Hank * Save memory by modifying in-place. Co-authored-by: Your Name <[email protected]> Co-authored-by: Ammar Ahmad Awan <[email protected]> Co-authored-by: tanghl1994 <[email protected]> Co-authored-by: Hank <[email protected]> Co-authored-by: root <[email protected]> * Staging 1bit adam v1 (#348) * Refactor to correct locations. * Deleted unused files. * Fix imports for refactored codebase. * update the com reduce test. * Fix some errors. * Fix optimizer name * Delete unused tests. * Fix formatting for pre-commit. * Add cupy dependencies. * add cupy for cuda 10add cupy for cuda 10 * Add mpi4py requirement. Co-authored-by: Ammar Ahmad Awan <[email protected]> * Use correct initialization for exp_avg. * Cleanup onebit adam. * minor wording fix. * Cleanup custom collectives. * Fixes for TCP support. * fix formatting. fix formatting. * move 1bit adam reqs * delay importing 1bit adam unless it's used, this will ensure we delay importing mpi4py * Fix cuda version parsing. * Temporary tcp fix. * Update install.sh * Refactor code to properly support cuda-aware comm. * Fix imports. * add cuda_aware flag to tests. * Cleanup. Add copyrights. * Add 1-bit Adam tutorial v1. * Minor fixes to copyright and print statements. * Update utils.py * Update utils.py Co-authored-by: Jeff Rasley <[email protected]> Co-authored-by: Your Name <[email protected]> Co-authored-by: tanghl1994 <[email protected]> Co-authored-by: Hank <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Ammar Ahmad Awan <[email protected]>
* refactor pdsh and add openmpi launcher * fix bugs/cleanup
awan-10
requested review from
arashashari,
cli99,
conglongli,
eltonzheng,
jeffra,
minjiaz,
niumanar,
RezaYazdaniAminabadi,
samyam,
ShadenSmith and
tjruwase
as code owners
… into staging-1bit-adam-v2
… into staging-1bit-adam-v2
… into staging-1bit-adam-v2
… into staging-1bit-adam-v2
… into staging-1bit-adam-v2
… into staging-1bit-adam-v2
… into staging-1bit-adam-v2
jeffra
approved these changes
Sep 9, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.