Code

Another thing to merge. (MY EYES HURT)#1

Code

Merged

bobisapotato merged 199 commits intobobisai:masterbobisai/DeepSpeed:masterfrom microsoft:mastermicrosoft/DeepSpeed:masterCopy head branch name to clipboardJan 24, 2021

+24,112-5,218

Commits

Commits on Jul 21, 2020

• only global rank 0 can log tensorboard data; avoid multi gpu/node race for the log directory (#296)

  arashashari

  authored
  1f97242

  Copy full SHA for 1f97242

Commits on Jul 22, 2020

• Update setup.py (#298)

  jeffra

  authored
  871f7e6

  Copy full SHA for 871f7e6

• Avoid deadlock for unsynchronized non-zero checkpointing (#297)

  Show description for 3cc96e1
  

  tjruwase

  and

  jeffra

  authored
  3cc96e1

  Copy full SHA for 3cc96e1

Commits on Jul 23, 2020

• updates to amp to support grad clip and grad accumulation (#290)

  Show description for eb74c3f
  jeffra

  authored
  eb74c3f

  Copy full SHA for eb74c3f

Commits on Jul 24, 2020

• pass steps_per_print to tput timer (#299)

  jeffra

  authored
  ec94341

  Copy full SHA for ec94341

• bump DSExamples (#300)

  jeffra

  authored
  0f94f7e

  Copy full SHA for 0f94f7e

Commits on Jul 25, 2020

• DeepSpeed webinar announcement (#301)

  

  Shaden Smith

  authored
  7ae8f8b

  Copy full SHA for 7ae8f8b

Commits on Jul 27, 2020

• Update README.md (#302)

  jeffra

  authored
  67821f9

  Copy full SHA for 67821f9

Commits on Jul 28, 2020

• Fixing a typo (#303)

  MannyKayy

  authored
  97c5427

  Copy full SHA for 97c5427

• Fix nv_peer_mem version (#304)

  Show description for e50b883
  jeffra

  authored
  e50b883

  Copy full SHA for e50b883

Commits on Aug 1, 2020

• NameError: name 'mpu' is not defined (#305)

  Show description for 9d07d75
  MannyKayy

  authored
  9d07d75

  Copy full SHA for 9d07d75

Commits on Aug 7, 2020

• Removing () from assertion. (#307)

  Show description for c35e944
  

  Shaden Smith

  authored
  c35e944

  Copy full SHA for c35e944

• Add webinar link (#309)

  Show description for 29c5fe2
  jeffra

  authored
  29c5fe2

  Copy full SHA for 29c5fe2

Commits on Aug 8, 2020

• updates website gems after kramdown alert (#311)

  

  Shaden Smith

  authored
  903a41a

  Copy full SHA for 903a41a

Commits on Aug 10, 2020

• Fix+tests for get_lr from lr_scheduler before training starts (#310)

  Show description for cd68e6e
  jeffra

  authored
  cd68e6e

  Copy full SHA for cd68e6e

Commits on Aug 12, 2020

• bumping DSE commit for pillow security fix (#312)

  

  Shaden Smith

  authored
  892ece6

  Copy full SHA for 892ece6

• Update deepspeed_lr_schedules.py (#314)

  jeffra

  authored
  3437342

  Copy full SHA for 3437342

Commits on Aug 13, 2020

• Update fan out flag for pdsh (#315)

  Show description for 6855ba1
  jeffra

  authored
  6855ba1

  Copy full SHA for 6855ba1

• attach empty grad to its param to ensure it's copied after reduction (#316)

  jeffra

  authored
  e1bea67

  Copy full SHA for e1bea67

Commits on Aug 14, 2020

• bump DSE (#317)

  jeffra

  authored
  de0523d

  Copy full SHA for de0523d

Commits on Aug 18, 2020

• Turn off multi-node launch if only 1 node (#322)

  Show description for e69b1ee
  jeffra

  authored
  e69b1ee

  Copy full SHA for e69b1ee

Commits on Aug 27, 2020

• Add code owners for DeepSpeed team (#335)

  Show description for 21d5f63
  jeffra

  authored
  21d5f63

  Copy full SHA for 21d5f63

Commits on Aug 28, 2020

• bump DSE

  jeffra

  committed
  6823db3

  Copy full SHA for 6823db3

Commits on Aug 31, 2020

• Update deepspeed_checkpointing.py (#336)

  Show description for 458c0d9
  

  samyam

  and

  jeffra

  authored
  458c0d9

  Copy full SHA for 458c0d9

Commits on Sep 1, 2020

• Samyamr/grad acc stage2 (#338)

  Show description for 7240abf
  

  samyam

  and

  jeffra

  authored
  7240abf

  Copy full SHA for 7240abf

• Rename ds_config_func_bs8_zero2_gas10.json to ds_config_func_bs8_zero2_gas3.json

  samyam

  authored
  7a356b2

  Copy full SHA for 7a356b2

• Rename ds_config_func_bs8_zero0_gas10.json to ds_config_func_bs8_zero0_gas3.json

  samyam

  authored
  6122a74

  Copy full SHA for 6122a74

• Update run_func_test.py

  Show description for f4726b7
  samyam

  authored
  f4726b7

  Copy full SHA for f4726b7

• Update .gitignore

  jeffra

  authored
  e8dd47d

  Copy full SHA for e8dd47d

• Switches BBS example to use mbsize=3 and gas=2 to fit in 16GB of memory. (#341)

  

  Shaden Smith

  authored
  838f53b

  Copy full SHA for 838f53b

Commits on Sep 2, 2020

• Sparse attn + ops/runtime refactor + v0.3.0 (#343)

  Show description for e5bbc2e
  

  jeffra

  and

  arashashari

  authored
  e5bbc2e

  Copy full SHA for e5bbc2e

• Update Dockerfile

  jeffra

  authored
  8716540

  Copy full SHA for 8716540

• Update Dockerfile

  Show description for 5518aae
  jeffra

  authored
  5518aae

  Copy full SHA for 5518aae

• update DSE and rename SA tests

  jeffra

  committed
  1661e83

  Copy full SHA for 1661e83

Commits on Sep 3, 2020

• Update test_sparse_attention.py

  jeffra

  authored
  1ebcd6c

  Copy full SHA for 1ebcd6c

• Adding link to Sparse Attention in Navigation page (#355)

  Show description for 6deac82
  arashashari

  authored
  6deac82

  Copy full SHA for 6deac82

Commits on Sep 4, 2020

• Jekyll installation instructions (#351)

  

  Shaden Smith

  authored
  ac12833

  Copy full SHA for ac12833

Commits on Sep 5, 2020

• fixed a typo; this was fixed before but seems like it has been lost in the refactor (#364)

  arashashari

  authored
  a64b0ab

  Copy full SHA for a64b0ab

• Move code quality tests to Azure-hosted agents. (#368)

  

  Shaden Smith

  authored
  4d4eafb

  Copy full SHA for 4d4eafb

Commits on Sep 6, 2020

• Update installation instructions (#362)

  tjruwase

  authored
  9e83ef2

  Copy full SHA for 9e83ef2

• Update Sparse Attention Tutorial (#357)

  Show description for 9dadf38
  

  arashashari

  and

  arashashari

  authored
  9dadf38

  Copy full SHA for 9dadf38

Commits on Sep 9, 2020

• adding sparse attention to feature index page (#377)

  arashashari

  authored
  b73894d

  Copy full SHA for b73894d

• temp disable model tests

  jeffra

  authored
  234bba0

  Copy full SHA for 234bba0

• Add 1-bit Adam support to DeepSpeed (#380)

  Show description for 01726ce
  

  6 peopleauthored
  01726ce

  Copy full SHA for 01726ce

• fixing a link issue with SA tutorial (#387)

  Show description for 161e8e6
  

  arashashari

  and

  jeffra

  authored
  161e8e6

  Copy full SHA for 161e8e6

• Update test triggers to exclude docs

  jeffra

  authored
  79093d7

  Copy full SHA for 79093d7

Commits on Sep 10, 2020

• ZeRO-Offload release (#391)

  Show description for 41db1c2
  

  9 peopleauthored
  41db1c2

  Copy full SHA for 41db1c2

• Pipeline parallel training engine. (#392)

  Show description for 65c2f97
  

  Shaden Smith

  and

  jeffra

  authored
  65c2f97

  Copy full SHA for 65c2f97

• Update documentation for 1-bit Adam (#388)

  Show description for 093f09f
  

  awan-10

  and

  jeffra

  authored
  093f09f

  Copy full SHA for 093f09f

• Fix datatype issue with sparse attention softmax (#363)

  Show description for dca0b78
  

  jeffra

  and

  Shaden Smith

  authored
  dca0b78

  Copy full SHA for dca0b78

• Add openmpi to dockerfile

  jeffra

  authored
  c0d5424

  Copy full SHA for c0d5424

• ZeRO tutorials (#384)

  Show description for 2dea61f
  

  3 peopleauthored
  2dea61f

  Copy full SHA for 2dea61f

• fix for 16GB v100 nodes (#393)

  jeffra

  authored
  b1d4bd7

  Copy full SHA for b1d4bd7

• Sparse attention: updating code tag in documentation (#394)

  Show description for be4b94b
  

  arashashari

  and

  jeffra

  authored
  be4b94b

  Copy full SHA for be4b94b

• Minjiaz/zero offload (#382)

  Show description for 59ce90d
  

  minjiaz

  and

  jeffra

  authored
  59ce90d

  Copy full SHA for 59ce90d

• Adding sparse attention news index item (#376)

  Show description for c76769c
  

  arashashari

  and

  jeffra

  authored
  c76769c

  Copy full SHA for c76769c

• Landing page updates (#395)

  Show description for a8a8b3d
  

  3 peopleauthored
  a8a8b3d

  Copy full SHA for a8a8b3d

• Update README.md

  jeffra

  authored
  7baf3c3

  Copy full SHA for 7baf3c3

• Website edits (#398)

  Show description for 6bb5c69
  

  Shaden Smith

  and

  jeffra

  authored
  6bb5c69

  Copy full SHA for 6bb5c69

• update docker image and bump DSE

  

  jeffra

  authored and

  Ubuntu

  committed
  b29229b

  Copy full SHA for b29229b

• only add 1bit adam reqs if mpi is installed, update cond build for cpu-adam (#400)

  jeffra

  authored
  240ea97

  Copy full SHA for 240ea97

• bump DSE and doc tweak

  

  jeffra

  authored and

  Ubuntu

  committed
  4b1df25

  Copy full SHA for 4b1df25

• Update README.md

  jeffra

  authored
  9693595

  Copy full SHA for 9693595

• Update _config.yml

  jeffra

  authored
  ea92ed2

  Copy full SHA for ea92ed2

• Update news site with press release link

  jeffra

  authored
  5dc4d6c

  Copy full SHA for 5dc4d6c

• Update ZeRO-Offload blog post link (#401)

  Show description for d15015e
  

  3 peopleauthored
  d15015e

  Copy full SHA for d15015e

• remove old pt file

  jeffra

  committed
  15ca99c

  Copy full SHA for 15ca99c

• readthedocs upgrade (#402)

  

  Shaden Smith

  authored
  c82756c

  Copy full SHA for c82756c

Commits on Sep 11, 2020

• supporting different intermediate sizes other than 4 * hidden_dim (#389)

  Show description for e549be6
  

  RezaYazdaniAminabadi

  and

  jeffra

  authored
  e549be6

  Copy full SHA for e549be6

• Revert "supporting different intermediate sizes other than 4 * hidden_dim (#389)" (#404)

  Show description for 4ac9bf6
  jeffra

  authored
  4ac9bf6

  Copy full SHA for 4ac9bf6

Commits on Sep 13, 2020

• scales throughput by logging freq (#408)

  

  Shaden Smith

  authored
  473ff98

  Copy full SHA for 473ff98

Commits on Sep 15, 2020

• pytest skips for tests requiring certain ops (#411)

  Show description for 91b4a93
  jeffra

  authored
  91b4a93

  Copy full SHA for 91b4a93

• fix bug related to stitching reduced grads across communication partitions (#318)

  jeffra

  authored
  55ed105

  Copy full SHA for 55ed105

• add cpu-adam, reformat, add colors (#413)

  jeffra

  authored
  a9e8325

  Copy full SHA for a9e8325

Commits on Sep 16, 2020

• Add Linear warmup+decay lr schedule (#414)

  Show description for 0e942df
  tjruwase

  authored
  0e942df

  Copy full SHA for 0e942df

• Minor doc fixes (#417)

  Show description for 7d91be9
  

  3 peopleauthored
  7d91be9

  Copy full SHA for 7d91be9

• Overflow fix (#416)

  Show description for f5cce75
  

  Shaden Smith

  authored
  f5cce75

  Copy full SHA for f5cce75

• Fix a typo in comments (#415)

  Show description for 4fef478
  

  eric-haibin-lin

  and

  jeffra

  authored
  4fef478

  Copy full SHA for 4fef478

Commits on Sep 17, 2020

• readthedocs yaml configuration (#410)

  Show description for 5812e84
  

  Shaden Smith

  and

  jeffra

  authored
  5812e84

  Copy full SHA for 5812e84

• Fix few typos in the docs (#418)

  gowthamvadisetti

  authored
  c66f388

  Copy full SHA for c66f388

• Remove pip --use-feature (#419)

  jeffra

  authored
  5bc7d4e

  Copy full SHA for 5bc7d4e

Commits on Sep 18, 2020

• Activation checkpointing bugfix and unit tests (#420)

  Show description for 01b6e27
  

  Shaden Smith

  and

  jeffra

  authored
  01b6e27

  Copy full SHA for 01b6e27

• Revert "Activation checkpointing bugfix and unit tests (#420)" (#422)

  Show description for a74a604
  

  jeffra

  and

  ShadenSmith

  authored
  a74a604

  Copy full SHA for a74a604

• Fix activation checkpoint unit tests for GPU systems (#421)

  

  Shaden Smith

  authored
  a825f99

  Copy full SHA for a825f99

Commits on Sep 21, 2020

• Add configurable intermediate size to transformer kernels (#423)

  RezaYazdaniAminabadi

  authored
  a148bd3

  Copy full SHA for a148bd3

• DSE bump (#427)

  

  Shaden Smith

  authored
  71f7df3

  Copy full SHA for 71f7df3

Commits on Sep 22, 2020

• support dynamic sequence length in transformer kernels (#424)

  Show description for f0f2a70
  

  RezaYazdaniAminabadi

  and

  conglongli

  authored
  f0f2a70

  Copy full SHA for f0f2a70

Commits on Sep 24, 2020

• Fix urls in tutorial (#436)

  Show description for 5d40f00
  conglongli

  authored
  5d40f00

  Copy full SHA for 5d40f00

• Update azure.md (#437)

  conglongli

  authored
  192cf7c

  Copy full SHA for 192cf7c

Commits on Sep 25, 2020

• Update pipeline.md (#439)

  eric-haibin-lin

  authored
  0ca8215

  Copy full SHA for 0ca8215

• link fix part two :-) (#441)

  

  Shaden Smith

  authored
  6d176c4

  Copy full SHA for 6d176c4

• unit test rename (#442)

  

  Shaden Smith

  authored
  5412a33

  Copy full SHA for 5412a33

Commits on Sep 28, 2020

• fix typos (#446)

  eric-haibin-lin

  authored
  6f28ea3

  Copy full SHA for 6f28ea3

Commits on Sep 29, 2020

• Disable default installation of CPU Adam (#450)

  Show description for 7b8be2a
  tjruwase

  authored
  7b8be2a

  Copy full SHA for 7b8be2a

Commits on Oct 1, 2020

• Use parentesis around min and max to enable Windows build (#449)

  Show description for 9557557
  

  4 peopleauthored
  9557557

  Copy full SHA for 9557557

Commits on Oct 5, 2020

• Update engine.py (#458)

  Show description for 6717638
  Zixxy

  authored
  6717638

  Copy full SHA for 6717638

Commits on Oct 7, 2020

• temporarily disable lr unit tests

  jeffra

  committed
  11cf47e

  Copy full SHA for 11cf47e

• turning off different tests (temp)

  jeffra

  committed
  679fc13

  Copy full SHA for 679fc13

• gan tutorial (#462)

  Show description for 2efea69
  

  niumanar

  and

  jeffra

  authored
  2efea69

  Copy full SHA for 2efea69

• Fix printing momentum for non-deepspeed optimizer (#464)

  Show description for c39a76f
  Zixxy

  authored
  c39a76f

  Copy full SHA for c39a76f

Commits on Oct 10, 2020

• Add DeepSpeed_Adam optimizer (#468)

  Show description for 23fc48f
  

  3 peopleauthored
  23fc48f

  Copy full SHA for 23fc48f

Commits on Oct 12, 2020

• fixing typo (#460)

  RezaYazdaniAminabadi

  authored
  e25f2a2

  Copy full SHA for e25f2a2

• add compute cap of 6.0 to transformer kernels

  Show description for b8eb40e
  jeffra

  authored
  b8eb40e

  Copy full SHA for b8eb40e

• revert previous (accidental) change

  jeffra

  authored
  1afca8f

  Copy full SHA for 1afca8f

Commits on Oct 14, 2020

• Add support for p100 in transformer kernels (#470)

  Show description for 7ddfda8
  jeffra

  authored
  7ddfda8

  Copy full SHA for 7ddfda8

Commits on Oct 19, 2020

• updating website dependencies (#475)

  

  Shaden Smith

  authored
  d720fdb

  Copy full SHA for d720fdb

Commits on Oct 30, 2020

• Add CPUAdam optimizer for zero-offload in deepspeed engine (#484)

  Show description for f5aa254
  

  RezaYazdaniAminabadi

  and

  jeffra

  authored
  f5aa254

  Copy full SHA for f5aa254

• fixing the AVX_256 compatibility (#497)

  RezaYazdaniAminabadi

  authored
  4c37d70

  Copy full SHA for 4c37d70

Commits on Nov 5, 2020

• Fixing CPU-Adam convergence issue (#503)

  Show description for 7d4d742
  RezaYazdaniAminabadi

  authored
  7d4d742

  Copy full SHA for 7d4d742

Commits on Nov 9, 2020

• PLD documentation (#514)

  Show description for e351090
  tjruwase

  authored
  e351090

  Copy full SHA for e351090

• Fix PLD news url (#515)

  Show description for 41fb24b
  tjruwase

  authored
  41fb24b

  Copy full SHA for 41fb24b

Commits on Nov 10, 2020

• updating pld docs (#517)

  minjiaz

  authored
  e082d47

  Copy full SHA for e082d47

• PLD release (#513)

  Show description for be1147c
  

  4 peopleauthored
  be1147c

  Copy full SHA for be1147c

Commits on Nov 11, 2020

• fix bug on non-DLTS infra when no output path set (#523)

  jeffra

  authored
  eea1c28

  Copy full SHA for eea1c28

• Update zero.md tutorial (#495)

  Show description for 0ad4fd8
  

  samyam

  and

  jeffra

  authored
  0ad4fd8

  Copy full SHA for 0ad4fd8

Commits on Nov 12, 2020

• DeepSpeed JIT op + PyPI support (#496)

  Show description for 31f46fe
  

  3 peopleauthored
  31f46fe

  Copy full SHA for 31f46fe

• ds_report bug fix on cpu and guard torch import in setup.py (#524)

  Show description for ca9ab12
  jeffra

  authored
  ca9ab12

  Copy full SHA for ca9ab12

• Installation documentation updates. (#525)

  Show description for d779bd5
  

  Shaden Smith

  authored
  d779bd5

  Copy full SHA for d779bd5

Commits on Nov 14, 2020

• Dependency pruning (#528)

  Show description for 0dc8420
  jeffra

  authored
  0dc8420

  Copy full SHA for 0dc8420

• bump version

  jeffra

  committed
  9941ce7

  Copy full SHA for 9941ce7

Commits on Nov 18, 2020

• Fix layout bug in ZeRO Stage 1 checkpoint logic (#531)

  Show description for 7752dc5
  tjruwase

  authored
  7752dc5

  Copy full SHA for 7752dc5

• append job-name if explicit output dir is given (#539)

  jeffra

  authored
  5b09be6

  Copy full SHA for 5b09be6

Commits on Nov 19, 2020

• more fine-grained manifest file for includes/excludes (#540)

  jeffra

  authored
  fdd81c3

  Copy full SHA for fdd81c3

• ZeRO-1 tune max-elems + bug fix (#532)

  Show description for 08c96a1
  jeffra

  authored
  08c96a1

  Copy full SHA for 08c96a1

• bump to v0.3.3

  jeffra

  authored
  9de21b7

  Copy full SHA for 9de21b7

• backwards compatability w. v020 ckpts, fix issue with zero-1 ckpts (#543)

  jeffra

  authored
  dce054d

  Copy full SHA for dce054d

• Fix setup.py for cpu-only environment installation (#538)

  Show description for d81cb26
  

  harrydrippin

  and

  jeffra

  authored
  d81cb26

  Copy full SHA for d81cb26

• Discover variables for NCCL backend on AML without mpi4py (#542)

  Show description for 1b45917
  

  awan-10

  and

  jeffra

  authored
  1b45917

  Copy full SHA for 1b45917

• bump version 0.3.4

  jeffra

  committed
  6b28bc5

  Copy full SHA for 6b28bc5

Commits on Nov 20, 2020

• Fix unbalanced gradients bug in ZeRO-2 gradient accumulation (#545)

  Show description for 0178e6c
  tjruwase

  authored
  0178e6c

  Copy full SHA for 0178e6c

Commits on Nov 21, 2020

• Support non-tensor state in checkpoint (#548)

  tjruwase

  authored
  6021b70

  Copy full SHA for 6021b70

Commits on Nov 23, 2020

• Adding static_loss_scale to unfused optimizer (#546)

  samyam

  authored
  bcd56f9

  Copy full SHA for bcd56f9

• Bug fix for norm calculation in absence of model parallel group (#551)

  Show description for 00c3a25
  samyam

  authored
  00c3a25

  Copy full SHA for 00c3a25

• bump to 0.3.5

  jeffra

  committed
  16313a9

  Copy full SHA for 16313a9

Commits on Nov 24, 2020

• Create main.yml

  jeffra

  authored
  c18fb0d

  Copy full SHA for c18fb0d

• Switch to CI to GitHub Actions (#556)

  jeffra

  authored
  3347460

  Copy full SHA for 3347460

• Update badges and CI name (#557)

  jeffra

  authored
  1ef5cd2

  Copy full SHA for 1ef5cd2

• Deprecate client ability to disable gradient reduction (#552)

  Show description for 6e65c2c
  

  tjruwase

  and

  jeffra

  authored
  6e65c2c

  Copy full SHA for 6e65c2c

Commits on Nov 25, 2020

• Simplify dist init and only init if needed. (#553)

  Show description for 0e831e2
  

  awan-10

  and

  jeffra

  authored
  0e831e2

  Copy full SHA for 0e831e2

• Turn back on PP tests (#558)

  jeffra

  authored
  eec44af

  Copy full SHA for eec44af

• Adds long_description to setup.py (#560)

  

  Shaden Smith

  authored
  6009713

  Copy full SHA for 6009713

• bump to 0.3.6 and fix manifest to include reqs (#561)

  jeffra

  authored
  73c3262

  Copy full SHA for 73c3262

• update manifest

  jeffra

  committed
  e4e2066

  Copy full SHA for e4e2066

• bump to 0.3.7

  jeffra

  committed
  c51fa65

  Copy full SHA for c51fa65

Commits on Nov 28, 2020

• [doc] typo fix and clarification (#563)

  Show description for 17f36f1
  stas00

  authored
  17f36f1

  Copy full SHA for 17f36f1

Commits on Dec 1, 2020

• supporting different hidden dimensions (#559)

  Show description for c78c29f
  

  RezaYazdaniAminabadi

  and

  jeffra

  authored
  c78c29f

  Copy full SHA for c78c29f

• tracking optimizer step in cpu-adam when loading checkpoint (#564)

  Show description for 9f52a36
  RezaYazdaniAminabadi

  authored
  9f52a36

  Copy full SHA for 9f52a36

Commits on Dec 2, 2020

• [cifar tutorial] improve readability (#567)

  Show description for 7a75f8b
  stas00

  authored
  7a75f8b

  Copy full SHA for 7a75f8b

• Add 'latest' checkpoint save/load support (#569)

  jeffra

  authored
  845921b

  Copy full SHA for 845921b

Commits on Dec 3, 2020

• [engine] train should be able to get mode arg (#571)

  stas00

  authored
  2d1f7c0

  Copy full SHA for 2d1f7c0

• Add compute capability 8.0 if on cuda 11+ (#572)

  jeffra

  authored
  be33bea

  Copy full SHA for be33bea

• [build] build against installed cuda-11.1 while torch built w/ cuda-11.0 (#570)

  stas00

  authored
  ff58fa7

  Copy full SHA for ff58fa7

Commits on Dec 4, 2020

• Fix potential random layout inconsistency issues in sparse attention modules (#534)

  Show description for 1e44d48
  

  3 peopleauthored
  1e44d48

  Copy full SHA for 1e44d48

Commits on Dec 7, 2020

• [build] make builder smarter and configurable wrt compute capabilities + docs (#578)

  stas00

  authored
  ce363d0

  Copy full SHA for ce363d0

• [build] add compute_86 (#577)

  Show description for e8b126d
  

  stas00

  and

  jeffra

  authored
  e8b126d

  Copy full SHA for e8b126d

Commits on Dec 8, 2020

• Pipeline warnings and checkpoint portability (#588)

  Show description for 2f62697
  

  Shaden Smith

  authored
  2f62697

  Copy full SHA for 2f62697

Commits on Dec 9, 2020

• Pin triton to 0.2.3 for now, 0.3.0 is broken

  jeffra

  authored
  d901a6d

  Copy full SHA for d901a6d

• bump to 0.3.8

  jeffra

  authored
  cb7c7da

  Copy full SHA for cb7c7da

• Add papers/videos to readme/website (#592)

  jeffra

  authored
  19acd6c

  Copy full SHA for 19acd6c

• Add AML video link

  jeffra

  authored
  7300f3e

  Copy full SHA for 7300f3e

Commits on Dec 11, 2020

• add manual workflow to run tests with precompiled ops

  jeffra

  committed
  0518252

  Copy full SHA for 0518252

• [build] fix computer capability arch flags, add PTX, handle PTX (#591)

  Show description for 8a184b6
  

  stas00

  and

  jeffra

  authored
  8a184b6

  Copy full SHA for 8a184b6

• add DeepSpeedZeroConfig repr method (#596)

  Show description for 66268bd
  

  stas00

  and

  jeffra

  authored
  66268bd

  Copy full SHA for 66268bd

• Supported customizing kwargs for lr_scheduler (#584)

  Show description for a4763f5
  

  carefree0910

  and

  jeffra

  authored
  a4763f5

  Copy full SHA for a4763f5

• Update launcher to set local rank environ variable (#597)

  Show description for c5a449f
  jeffra

  authored
  c5a449f

  Copy full SHA for c5a449f

Commits on Dec 14, 2020

• implement missing get_last_lr (#595)

  Show description for 9f8e8f3
  

  stas00

  and

  jeffra

  authored
  9f8e8f3

  Copy full SHA for 9f8e8f3

Commits on Dec 15, 2020

• [doc] xref to hostfile discussion (#604)

  Show description for 007466e
  stas00

  authored
  007466e

  Copy full SHA for 007466e

• Fixes for RTD build errors (#606)

  Show description for 6380ee3
  

  jeffra

  and

  Shaden Smith

  authored
  6380ee3

  Copy full SHA for 6380ee3

Commits on Dec 17, 2020

• Transformer-kernel - supporting any arbitrary sequence-length (#587)

  Show description for fd2f970
  

  RezaYazdaniAminabadi

  and

  jeffra

  authored
  fd2f970

  Copy full SHA for fd2f970

Commits on Dec 18, 2020

• Ability to initialize distributed backend outside deepspeed runtime (#608)

  jeffra

  authored
  7435b2f

  Copy full SHA for 7435b2f

Commits on Dec 23, 2020

• Elastic training support (#602)

  Show description for 81aeea3
  

  jeffra

  and

  samyam

  authored
  81aeea3

  Copy full SHA for 81aeea3

Commits on Jan 4, 2021

• update SA comp check to fix torch-cpu issue (#631)

  jeffra

  authored
  24e0739

  Copy full SHA for 24e0739

• Support initialization with dict configuration (#632)

  tjruwase

  authored
  e6ac731

  Copy full SHA for e6ac731

Commits on Jan 5, 2021

• Allow DeepSpeed models to be initialized with optimizer=None (#469)

  Show description for a9a83a6
  

  gcooper-isi

  and

  Shaden Smith

  authored
  a9a83a6

  Copy full SHA for a9a83a6

• change dist to torch.distributed to fix bug in assert. (#638)

  awan-10

  authored
  d38ad6a

  Copy full SHA for d38ad6a

• docs: minor spelling tweaks (#623)

  Show description for 46d2e28
  

  brettkoonce

  and

  jeffra

  authored
  46d2e28

  Copy full SHA for 46d2e28

• Fix docstring format (#640)

  tjruwase

  authored
  5ab1279

  Copy full SHA for 5ab1279

Commits on Jan 6, 2021

• Module replacement support (#586)

  Show description for 44bd538
  

  3 peopleauthored
  44bd538

  Copy full SHA for 44bd538

Commits on Jan 7, 2021

• Update builder.py (#642)

  sxjscience

  authored
  64461da

  Copy full SHA for 64461da

• Bump nokogiri from 1.10.10 to 1.11.0 in /docs (#630)

  Show description for 8cea96d
  

  dependabot[bot]

  and

  jeffra

  authored
  8cea96d

  Copy full SHA for 8cea96d

• Add deepspeed.init_distributed to RTD page (#645)

  Show description for 4e2dc4e
  

  jeffra

  and

  tjruwase

  authored
  4e2dc4e

  Copy full SHA for 4e2dc4e

Commits on Jan 8, 2021

• document deepspeed.initialize() (#644)

  Show description for 828d75b
  

  stas00

  and

  jeffra

  authored
  828d75b

  Copy full SHA for 828d75b

• add additional validation checks in elastic config (#646)

  jeffra

  authored
  bc046dc

  Copy full SHA for bc046dc

• Remove a very verbose print statement. (#649)

  Show description for af212f6
  awan-10

  authored
  af212f6

  Copy full SHA for af212f6

• version bump to 0.3.10

  jeffra

  committed
  c14b839

  Copy full SHA for c14b839

• LR scheduler unit tests (#429)

  Show description for da5563a
  

  tjruwase

  and

  jeffra

  authored
  da5563a

  Copy full SHA for da5563a

Commits on Jan 12, 2021

• Handle actvitation checkpointing args that are None or non-tensors (#660)

  Show description for adcfd26
  

  Shaden Smith

  authored
  adcfd26

  Copy full SHA for adcfd26

Commits on Jan 13, 2021

• squash latest flops profiling changes (#1) (#664)

  Show description for e2fbe4d
  

  cli99

  and

  jeffra

  authored
  e2fbe4d

  Copy full SHA for e2fbe4d

• Move workspace memory-allocation to PyTorch (#661)

  Show description for 981bc7d
  RezaYazdaniAminabadi

  authored
  981bc7d

  Copy full SHA for 981bc7d

Commits on Jan 14, 2021

• Validate consistent ckpt tags across ranks (#667)

  jeffra

  authored
  f032e56

  Copy full SHA for f032e56

Commits on Jan 15, 2021

• Support optimizer AdamW type (#670)

  tjruwase

  authored
  865104b

  Copy full SHA for 865104b

• skip empty lines in hostfile (#669)

  jeffra

  authored
  6217a6c

  Copy full SHA for 6217a6c

• Add AdamW to the supported optimizers (#672)

  Show description for c5e4264
  

  stas00

  and

  jeffra

  authored
  c5e4264

  Copy full SHA for c5e4264

• add missing config menu entries (#652)

  Show description for e729a3f
  

  stas00

  and

  jeffra

  authored
  e729a3f

  Copy full SHA for e729a3f

• doc fix (#651)

  Show description for 7b07e12
  

  stas00

  and

  jeffra

  authored
  7b07e12

  Copy full SHA for 7b07e12

Commits on Jan 19, 2021

• add zero-offload paper (#680)

  Show description for 82cecf6
  jeffra

  authored
  82cecf6

  Copy full SHA for 82cecf6

Commits on Jan 20, 2021

• [tutorials] typos (#676)

  Show description for 7b0bee0
  

  stas00

  and

  jeffra

  authored
  7b0bee0

  Copy full SHA for 7b0bee0

• make test_pipe more stable (#683)

  

  Shaden Smith

  authored
  e59ba12

  Copy full SHA for e59ba12

• Fix ZeRO 2 + Pipelining (#677)

  Show description for 34c83a5
  leogao2

  authored
  34c83a5

  Copy full SHA for 34c83a5