Code

No-ZeRO reshaping#289

Code

Open

Muennighoff wants to merge 37 commits intomainbigscience-workshop/Megatron-DeepSpeed:mainfrom nozero_reshapebigscience-workshop/Megatron-DeepSpeed:nozero_reshapeCopy head branch name to clipboard

+1,625-97

Commits

Commits on Jan 24, 2022

• Reshape deepspeed checkpoint

  tjruwase

  committed
  67c08f0

• Merge remote-tracking branch 'origin/main' into ds_ckpt_reshape

  stas00

  committed
  fec1ec5

Commits on Jan 25, 2022

• add checkpoint tests

  stas00

  committed
  675f12c

• Validate input folder

  tjruwase

  committed
  e379065

• Tests for tp/pp reshape

  tjruwase

  committed
  a1068e4

• remove debug folders

  stas00

  committed
  115bd31

• fix test_checkpoint_reshaping_empty_dir

  stas00

  committed
  cc2fad1

Commits on Jan 26, 2022

• Fix unit tests

  tjruwase

  committed
  b6733d5

Commits on Feb 7, 2022

• Remove deepspeed checkpoint utils

  tjruwase

  committed
  9bf7ac5

Commits on Feb 23, 2022

• Use DS 3D reshaping utils

  tjruwase

  committed
  29ca2bc

Commits on May 18, 2022

• Merge remote-tracking branch 'origin/main' into ds_ckpt_reshape

  stas00

  committed
  a3ef778

• convert to bf16

  stas00

  committed
  6d86358

Commits on May 20, 2022

• wip universal chkpt

  stas00

  committed
  804b497

• rename

  stas00

  committed
  c29d336

• rename

  stas00

  committed
  9c44793

Commits on May 24, 2022

• wip on fragments dealing

  stas00

  committed
  7e0a81b

• cleanup

  stas00

  committed
  d300512

Commits on May 26, 2022

• Loading universal checkpoint with reshaping

  tjruwase

  committed
  ab0a7f8

Commits on May 28, 2022

• all gpu1<->2 reshapes work

  stas00

  committed
  d5e33de

Commits on May 31, 2022

• param attrs

  tjruwase

  committed
  85ff56c

Commits on Jun 1, 2022

• make the tests adaptable to the number of available gpus

  stas00

  committed
  f01fa4a

• WIP

  tjruwase

  committed
  f29bacc

• WIP

  tjruwase

  committed
  dd0aeb6

Commits on Jun 2, 2022

• WIP

  tjruwase

  committed
  3bf14fd

• WIP

  tjruwase

  committed
  7ae002d

Commits on Jun 4, 2022

• Debug functions

  tjruwase

  committed
  55bb514

• args should be required, don't create another latest file

  stas00

  committed
  795fedb

Commits on Jun 6, 2022

• Parallelize shard extraction

  tjruwase

  committed
  cc8810b

Commits on Jun 7, 2022

• close+join pool; add tqdm; comment out noise

  stas00

  committed
  04d9ad0

• rename

  stas00

  committed
  bca5af4

• parameterize

  stas00

  committed
  721380b

• Parallel slice merging

  tjruwase

  committed
  e8a1ccf

• Cleanup

  tjruwase

  committed
  a247614

Commits on Jun 22, 2022

• Merge branch 'main' into nozero_reshape

  Muennighoff

  committed
  d94e1e2

Commits on Jun 23, 2022

• Add many-to-many non-ZeRO reshaping

  Muennighoff

  committed
  c14df23

• Explain many-to-many status

  Muennighoff

  committed
  e8ba66a

Commits on Jul 4, 2022

• Merge main

  Muennighoff

  committed
  86c74e6