Code

bf16+pipeline parallelism#1801

Code

Merged

tjruwase merged 52 commits intomastermicrosoft/DeepSpeed:masterfrom olruwase/bf16-updatesmicrosoft/DeepSpeed:olruwase/bf16-updatesCopy head branch name to clipboardApr 19, 2022

+964-146

Commits

Commits on Feb 18, 2022

• bf16 updates

  jeffra

  committed
  fb0dc00

Commits on Feb 21, 2022

• Got bf16 working

  tjruwase

  committed
  6eb4f1f

Commits on Feb 23, 2022

• fp32 reduction; flattened tensors

  tjruwase

  committed
  a3d3576

Commits on Feb 24, 2022

• bf16+zero_stage_1 first cut

  tjruwase

  committed
  6f5ebc3

• finish zero_stage 1 sharding

  tjruwase

  committed
  819abe2

Commits on Feb 27, 2022

• Matching fp16 with debugging codes

  tjruwase

  committed
  e48035b

• Matching loss with fp16

  tjruwase

  committed
  8245053

Commits on Feb 28, 2022

• Fix gradient clipping

  tjruwase

  committed
  1529313

Commits on Mar 1, 2022

• bf16 gradient clipping fix

  Show description for 27e5b95
  tjruwase

  committed
  27e5b95

• Unscale grad norm

  tjruwase

  committed
  f497702

Commits on Mar 2, 2022

• Fix grad norm scaling

  tjruwase

  committed
  0ad7c7d

Commits on Mar 4, 2022

• Enable loading fp16_zero_1 into bf16_zero_1 engine and vice versa

  tjruwase

  committed
  b81d862

• Fix clip_grad key error

  tjruwase

  committed
  35ea380

Commits on Mar 5, 2022

• Reduce tied weight gradients

  tjruwase

  committed
  37011a9

• Rebase with master

  tjruwase

  committed
  8fbd4bf

• Fix grad norm for moe

  tjruwase

  committed
  61d51fd

Commits on Mar 7, 2022

• Merge branch 'master' into olruwase/bf16-updates

  jeffra

  authored
  3ee61cd

Commits on Mar 8, 2022

• Merge branch 'master' into olruwase/bf16-updates

  jeffra

  authored
  46cc2ce

• Reduce specified gradients

  tjruwase

  committed
  de3616c

• Merge branch 'olruwase/reduce_specified_gradients' of github.com:microsoft/DeepSpeed into olruwase/bf16-updates

  tjruwase

  committed
  89e054d

Commits on Mar 9, 2022

• Use O(n) instead of O(n^2)

  tjruwase

  committed
  ab61edb

• Remove optimizer restriction for bf16

  tjruwase

  committed
  b7d64fd

Commits on Mar 10, 2022

• Link bf16 & fp32 params

  tjruwase

  committed
  1919868

Commits on Mar 11, 2022

• Clip gradients of last stage tied weights

  tjruwase

  committed
  77b649d

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  4a505ec

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  ff99cb2

Commits on Mar 15, 2022

• Merge branch 'master' into olruwase/bf16-updates

  jeffra

  authored
  20fdba3

Commits on Mar 16, 2022

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  86fa437

• Merge branch 'master' into olruwase/bf16-updates

  jeffra

  authored
  71499a8

Commits on Mar 17, 2022

• Merge branch 'master' into olruwase/bf16-updates

  jeffra

  authored
  7e7fa60

• Simplify tied weights reduction logic

  tjruwase

  committed
  2aa612a

Commits on Mar 18, 2022

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  2cd21f1

• Merge branch 'olruwase/bf16-updates' of github.com:microsoft/DeepSpeed into olruwase/bf16-updates

  tjruwase

  committed
  a4cbf0c

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  67ea260

Commits on Mar 19, 2022

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  6a4d6e6

Commits on Mar 22, 2022

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  4e1dcfd

Commits on Mar 24, 2022

• Also clip all tp rank parameters

  tjruwase

  committed
  e24814a

• Merge branch 'olruwase/bf16-updates' of github.com:microsoft/DeepSpeed into olruwase/bf16-updates

  tjruwase

  committed
  88cdf61

Commits on Mar 25, 2022

• lp to hp mapping

  tjruwase

  committed
  20697bc

Commits on Mar 28, 2022

• Link lp/hp/optim state; Refresh links after checkpoint load

  tjruwase

  committed
  4e8f7ff

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  52a2f10

• Merge branch 'olruwase/bf16-updates' of github.com:microsoft/DeepSpeed into olruwase/bf16-updates

  tjruwase

  committed
  3ed5703

• Remove debug print

  tjruwase

  committed
  5481b86

• Remove debug print

  tjruwase

  committed
  d911e67

Commits on Mar 30, 2022

• Simplify zero_grad logic

  tjruwase

  committed
  144f652

• fp32 accessors

  tjruwase

  committed
  bb70816

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  89b4b3f

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  a9bfaee

Commits on Mar 31, 2022

• Fix update bug

  tjruwase

  committed
  fa4ff11

• Merge branch 'olruwase/bf16-updates' of github.com:microsoft/DeepSpeed into olruwase/bf16-updates

  tjruwase

  committed
  cfd5638

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  5ea1c60

Commits on Apr 18, 2022

• Merge branch 'master' into olruwase/bf16-updates

  tjruwase

  authored
  0e2a1c5