Code

Merge with latest upstream#5

Code

Open

Quentin-Anthony wants to merge 78 commits intoZyphra:mainZyphra/Megatron-DeepSpeed:mainfrom microsoft:mainmicrosoft/Megatron-DeepSpeed:mainCopy head branch name to clipboard

+55,340-597

Commits

Commits on Nov 9, 2023

• fix circular dependencies when using Lora (#284)

  zhangsheng377

  authored
  27d7865

• Universal ckp fixes (#276)

  Show description for 6824e31
  mosheisland

  authored
  6824e31

• Enable the combination of sequence length warmup and RoPE (#285)

  conglongli

  authored
  4c0bc7f

Commits on Nov 14, 2023

• Add the __init__.py for including the vision folder in installation package (#288)

  Show description for e7f0201
  yuanwu2017

  authored
  e7f0201

• Change the text name of sample for compatible with Huggingface trainer (#289)

  Show description for 8760390
  yuanwu2017

  authored
  8760390

• Enable the args.deepspeed_config to use dict type (#290)

  Show description for 15355af
  yuanwu2017

  authored
  15355af

• Add tgs metrics (#286)

  CokeDong

  authored
  155ce98

Commits on Nov 15, 2023

• Fix lm_eval_harness for GPT models (#292)

  conglongli

  authored
  37050b8

Commits on Nov 16, 2023

• universal-ckp: support llama model (#287)

  Show description for 2348eed
  mosheisland

  authored
  2348eed

Commits on Nov 21, 2023

• fuse q and kve parameters for qga case (#291)

  RezaYazdaniAminabadi

  authored
  8415d03

• fix dropout of flash attention (#295)

  tohtana

  authored
  b93495a

Commits on Nov 27, 2023

• fix typo error (#299)

  inkcherry

  authored
  7ca477d

Commits on Nov 30, 2023

• fix reshape for split qga (#307)

  inkcherry

  authored
  61d5d61

Commits on Dec 5, 2023

• support huggingface tokenizer (#306)

  zhangjian94cn

  authored
  bdef2b0

Commits on Dec 6, 2023

• add RMSnorm torch fallback path (#312)

  inkcherry

  authored
  aad7ad9

Commits on Dec 12, 2023

• Update Universal Checkpointing README visualization PR (#314)

  lekurile

  authored
  b2b8b01

Commits on Dec 14, 2023

• Universal Checkpoint for Sequence Parallelism (#305)

  Show description for 71e8407
  samadejacobs

  authored
  71e8407

Commits on Jan 5, 2024

• Revert "Modify the args_default usage" (#325)

  Show description for d65921c
  yuanwu2017

  authored
  d65921c

Commits on Jan 10, 2024

• bug fix on args.deepspeed_config_dict (#328)

  ys950902

  authored
  a4f8079

Commits on Jan 23, 2024

• Add worker option for preprocess_data.py (#339)

  xu-song

  authored
  a3635ad

• Align variable before all_reduce with other files (#340)

  kamil-kaczor

  authored
  141bfbe

Commits on Jan 24, 2024

• use fused_adam in deepspeed (#335)

  ys950902

  authored
  b09c6a6

Commits on Jan 25, 2024

• fix rms norm import on non cuda device (#341)

  nrailg

  authored
  f9323e3

Commits on Jan 26, 2024

• Supervised Fine-tuning for HugginFace pretrained weight. (#318)

  Show description for 11f2d93
  

  inkcherry

  and

  conglongli

  authored
  11f2d93

Commits on Jan 29, 2024

• Fixed incorrect argument supplied to deepspeed init (#323)

  Show description for 89e03fd
  SulRash

  authored
  89e03fd

Commits on Feb 1, 2024

• Track additional metrics with W&B in megatron/training.py (#343)

  saforem2

  authored
  6d4c535

Commits on Feb 16, 2024

• Clean up UC scripts and update UC README (#344)

  Show description for d47f3cd
  lekurile

  authored
  d47f3cd

Commits on Feb 21, 2024

• Add TensorBoard analysis script to Universal Checkpointing Example (#345)

  Show description for 3557944
  lekurile

  authored
  3557944

• Remove duplicate ctx save backward in cross_entropy.py (#347)

  

  Puneesh Khanna

  authored
  3a30913

• Update Megatron type check (#346)

  Show description for ea82c14
  lekurile

  authored
  ea82c14

• Track additional metrics with W&B in megatron/training.py (#348)

  Show description for c934137
  saforem2

  authored
  c934137

Commits on Feb 22, 2024

• Add steps for running TensorBoard analysis in Universal Checkpointing README (#349)

  Show description for 9ba0dcb
  lekurile

  authored
  9ba0dcb

Commits on Feb 25, 2024

• Support configuration of RoPE theta (#351)

  Show description for 81d68a3
  mosheisland

  authored
  81d68a3

Commits on Feb 26, 2024

• Support loading checkpoint specific tag (#352)

  Show description for 31e2584
  mosheisland

  authored
  31e2584

Commits on Feb 27, 2024

• Update pretrain_bert.py (#355)

  Show description for a9856ce
  lzzmm

  authored
  a9856ce

Commits on Mar 10, 2024

• Support universal checkpoint for GPTModel (#361)

  Show description for df0e2e4
  mosheisland

  authored
  df0e2e4

• MOE: Support disable top2 2nd expert sampling (#362)

  Show description for ebe8025
  mosheisland

  authored
  ebe8025

Commits on Apr 2, 2024

• fix an issue for DP on Megatron-DeepSpeed (#368)

  ys950902

  authored
  888a63a

• remove contiguous copy for flash-attn opbuilder (#372)

  Show description for 3c5f475
  YizhouZ

  authored
  3c5f475

Commits on Apr 9, 2024

• Support MoE for GPTModelPipe (#373)

  Show description for bcedecd
  mosheisland

  authored
  bcedecd

Commits on May 13, 2024

• get distributed backend name via accelerator and check loss_scale before writing to tb (#374)

  Show description for 7eb36a1
  polisettyvarma

  authored
  7eb36a1

Commits on Jun 26, 2024

• Add steps and results for running ZeRO stage 3 withUniversal Checkpointing (#383)

  xylian86

  authored
  634e37b

Commits on Jun 27, 2024

• Add Zero Bubble Pipeline Parallelism H1 Schedule (#396)

  Show description for 527957e
  

  3 peopleauthored
  527957e

• Fix ParallelMLP and enable accelerator test (#403)

  xinyu-intel

  authored
  f2d7589

• Fix test_deallocate_output_tensor (#404)

  xinyu-intel

  authored
  ea4b67a

Commits on Jul 1, 2024

• Fixed missing BookCorpus dataset. (#407)

  Show description for 08f5a99
  

  costin-eseanu

  and

  Costin Eseanu

  authored
  08f5a99

• Set proper arguments when constructing models in unit tests (#408)

  xinyu-intel

  authored
  c3a13be

Commits on Jul 8, 2024

• use split/squeeze instead of slice for performance (#409)

  Show description for 330f9f2
  polisettyvarma

  authored
  330f9f2

• improve performance by keeping attention_mask on device and run ops further on device (#411)

  Show description for af06d14
  polisettyvarma

  authored
  af06d14

Commits on Jul 11, 2024

• Improve RoPE perf by using cached sin/cos tensors (#410)

  Show description for ec3f1f4
  polisettyvarma

  authored
  ec3f1f4

Commits on Jul 12, 2024

• Extend test utilities to support more accelerators (#418)

  Show description for 354e420
  xinyu-intel

  authored
  354e420

• clear document (#395)

  inkcherry

  authored
  73252c0

Commits on Jul 15, 2024

• add PyTorch profiler support (#414)

  Show description for 0971e68
  polisettyvarma

  authored
  0971e68

Commits on Jul 16, 2024

• [Wandb] Refine wandb logging function (#416)

  Show description for 73029ed
  billishyahao

  authored
  73029ed

Commits on Jul 17, 2024

• add kill switch file support to gracefully exit training at runtime (#412)

  Show description for fc989b8
  polisettyvarma

  authored
  fc989b8

Commits on Jul 18, 2024

• add support to run custom Hf tokenizer for training and dataset pre-processing (#421)

  Show description for 7d23e33
  polisettyvarma

  authored
  7d23e33

Commits on Jul 19, 2024

• improve repeat_kv GQA perf (#419)

  polisettyvarma

  authored
  13f2673

• acquire device when required (#420)

  Show description for 3af2e25
  polisettyvarma

  authored
  3af2e25

• Add basic compilation test (#426)

  loadams

  authored
  08b9376

• Update yml to be valid (#427)

  loadams

  authored
  3afd267

Commits on Jul 29, 2024

• Update/add GPT/Llama universal checkpointing scripts (#391)

  Show description for 8822a5c
  lekurile

  authored
  8822a5c

Commits on Aug 1, 2024

• fixing the bug of flash_attn import and the wrong gather index when using flash_attn_cuda in sequence parallel (#406)

  Show description for 1bfc35c
  

  YJHMITWEB

  and

  Jinghan Yao

  authored
  1bfc35c

Commits on Aug 4, 2024

• add fused_rms_norm support on XPU device (#431)

  ys950902

  authored
  53b241f

Commits on Aug 7, 2024

• pass batch_dim_idx to deepspeed sequence parallel distributed attention for supporting batch size larger than 1 (#433)

  Show description for 61350c5
  

  YJHMITWEB

  and

  Jinghan Yao

  authored
  61350c5

Commits on Aug 10, 2024

• [LLaMa] Adding support converting checkpoint from mds to hf (#432)

  Show description for f132876
  billishyahao

  authored
  f132876

Commits on Aug 14, 2024

• add device check when import ipex (#436)

  ys950902

  authored
  cdf5194

Commits on Aug 19, 2024

• fix TFLOPs calculation (#371)

  Show description for b7b2d5e
  polisettyvarma

  authored
  b7b2d5e

Commits on Aug 24, 2024

• fix nan issue when running megatron-deepspeed (#434)

  ys950902

  authored
  4f9f1f6

Commits on Aug 26, 2024

• enable empty cache on XPU device (#438)

  ys950902

  authored
  8e9d973

Commits on Aug 27, 2024

• [wandb] disable wandb more gracefully (#422)

  Show description for 543543a
  

  billishyahao

  and

  loadams

  authored
  543543a

• [Bug] Fix crash when logging optimizer state to tb (#417)

  billishyahao

  authored
  1280f59

Commits on Sep 4, 2024

• Enable Sequence Parallelism (#429)

  polisettyvarma

  authored
  0d6e379

Commits on Sep 20, 2024

• grad_wei can't be NoneType when running with DeepSpeed, for zero3 will divided the gradient (#428)

  ys950902

  authored
  598c092

Commits on Oct 4, 2024

• fix init issue for rms_norm in squence_parallel (#448)

  ys950902

  authored
  8be7f48

Commits on Oct 8, 2024

• enable profiler for specific ranks (#451)

  ranzhejiang

  authored
  4448492

Commits on Oct 17, 2024

• fix init issue for silently ignoring the deepspeed config (#452)

  xylian86

  authored
  deb95cd

Commits on Oct 18, 2024

• fix moe tflops (#445)

  ranzhejiang

  authored
  6acc370

Commits on Dec 5, 2024

• Adding the new feature of FPDT (#441)

  Show description for 676a482
  

  3 peopleauthored
  676a482