Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeepSpeed 0.2 support #21

Merged
merged 65 commits into from
May 14, 2020
Merged

DeepSpeed 0.2 support #21

merged 65 commits into from
May 14, 2020

Conversation

jeffra
Copy link
Contributor

@jeffra jeffra commented May 14, 2020

  1. ZeRO-2 support
  2. DeepSpeed transformer kernel support

eltonzheng and others added 30 commits April 24, 2020 21:48
@eltonzheng
apply cuda-ext to bing_bert model
c0b56e3
@eltonzheng
apply cuda-ext to BingBertSquad
ab17a14
@eltonzheng
update batch size for fine-tuning and validation
2b5bbd4
@eltonzheng
apply cuda-ext to bing_bert model
e1788f3
@eltonzheng
apply cuda-ext to BingBertSquad
e4fea05
@eltonzheng
update batch size for fine-tuning and validation
15d5c5e
@eltonzheng
Merge branch 'eltonz/apply-cuda-ext' of github.com:microsoft/DeepSpee…
e6edbb5 
…dExamples into eltonz/apply-cuda-ext
@eltonzheng
merge cuda_ext config to deepspeed config
95704fb
@eltonzheng
fix one config issue of 512-seq testing
37b92a0
@eltonzheng
support new interface of transformer kernel config
38abab4
@eltonzheng
replace postLN with preLN
695907c
@eltonzheng
add preLN modeling.py
7591d3f
@eltonzheng
apply the new transformer kernel interface to fine-tuning test
ff9bbd5
@eltonzheng
remove postLN modeling.py
f91db11
@eltonzheng
update on new interface and change settings to golden result
56004ea
@eltonzheng
change batch size back to 6
69148d7
@eltonzheng
fix code review comments
58d873b
@samyam
Support to use deepspeed.checkpointing for activation checkpointing.
c4319a2
@samyam
Removing ZeRO-1 config and script files
4e13b22
@eltonzheng
change back to using gradient_clippling
6949a22
@samyam
Removing DLWS references from scripts
fe82703
@samyam
Removing DLWS references from scripts and unused variables
3ce3215
@samyam
Merge pull request #3 from microsoft/samyamr/staging
fdec44b 
Samyamr/staging
@jeffra
clean-up
3138e82
@jeffra
remove nvidia/optimizers.py
ead4292
@eltonzheng
fix code review comments
7cad78d
@eltonzheng
Merge branch 'eltonz/apply-cuda-ext' of github.com:microsoft/DeepSpee…
5a40e78 
…dExamples-internal into eltonz/apply-cuda-ext
@eltonzheng
update epoch number to 150 for 128-seq training, and 160 for 512-seq …
8955cc3 
…training
@RezaYazdaniAminabadi
Merge pull request #2 from microsoft/eltonz/apply-cuda-ext
7acc3ce 
apply transformer kernel
@samyam
Adding a user flag to enable deepspeed activation checkpointing
05f65f2
jeffra and others added 26 commits May 12, 2020 00:19
@jeffra
add lr schedules to hf code
466c5ea
@jeffra
update config and run scripts
953981c
@jeffra
use 64k batch size
9b7e2bd
@jeffra
add kernel flag, set checkpointing to every epoch for now, added todos
a58dedc
@jeffra
fix squad run script (still needs a lot of refactoring)
76cc5de
@jeffra
Merge pull request #6 from microsoft/jeffra/bert_bug_fixes
5879d65 
bert bug fixes
@tjruwase
Fix API bug
3049d5b
@tjruwase
Merge pull request #7 from microsoft/olruwase/staging_squad_baseline_bug
0172a17 
Fix BingBertSquad API bug
fix 512 run
600eb8a
@RezaYazdaniAminabadi
Merge pull request #8 from microsoft/reyazda/fix_512_run
d805c86 
fix 512 run
@arashashari
Making BingSquad evaluation a module; mainly for use in testing. (#9)
8567e49 
* simplifying bert config files

* removed test argument

* removed test argument flag

* applying comment
@tjruwase
Add --attention_dropout_checkpoint
7236bfc
fixing the preln run for the BingBertSquad
971a1d4
@tjruwase
Change default value
8c295aa
@tjruwase
Merge pull request #10 from microsoft/olruwase/attn_dropout_checkpoint
632b3e3 
attention_dropout re-materialization optimization option
add deepspeed launcher
2da9731
add deepspeed launcher at run_squad_deepspeed.sh
e0cc190
@eltonzheng
add --ckpt_to_save argument to bing-bert
2439b6e
change deepscale to deepspeed
deb1df2
@RezaYazdaniAminabadi
Merge pull request #11 from microsoft/reyazda/fix_preln_for_BingBertS…
8c759d9 
…quad

Reyazda/fix preln for bing bert squad
fix
c49c742
change the transformer instantiation
a17ab38
@RezaYazdaniAminabadi
Merge pull request #13 from microsoft/reyazda/fix_shell_scripts_add_m…
66dd3cb 
…emOpt_flags

fix indentation error and add optimization flags
@eltonzheng
Merge branch 'staging' into eltonz/ckpt_to_save
68c060e
@eltonzheng
Merge pull request #12 from microsoft/eltonz/ckpt_to_save
a452839 
add --ckpt_to_save argument to bing-bert
@jeffra
formatting of all non-megatron examples (#14)
097f75c
@jeffra jeffra requested a review from ShadenSmith May 14, 2020 20:52
@jeffra jeffra merged commit 5426d18 into master May 14, 2020
@Modas-Li Modas-Li mentioned this pull request Apr 21, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ShadenSmith ShadenSmith ShadenSmith approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@jeffra @ShadenSmith @eltonzheng @samyam @RezaYazdaniAminabadi @arashashari @tjruwase