[Example] Refactor and Polish Cifar10-DeepSpeed Code Example. (#843)

* Polish and Refactor Cifar10 Code Example * fix typos --------- Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Conglong Li <[email protected]>
microsoft · Jan 26, 2024 · 107681e · 107681e
1 parent 8216f5f
commit 107681e
Show file tree

Hide file tree

Showing 4 changed files with 331 additions and 384 deletions.
diff --git a/training/cifar/README.md b/training/cifar/README.md
@@ -1,21 +1,22 @@
 Thanks Gopi Kumar for contributing this example, demonstrating how to apply DeepSpeed to CIFAR-10 model.
 
-cifar10_tutorial.py
+`cifar10_tutorial.py`
     Baseline CIFAR-10 model.
 
-cifar10_deepspeed.py
+`cifar10_deepspeed.py`
     DeepSpeed applied CIFAR-10 model.
 
-ds_config.json
-    DeepSpeed configuration file.
-
-run_ds.sh
+`run_ds.sh`
     Script for running DeepSpeed applied model.
 
-run_ds_moe.sh
+`run_ds_moe.sh`
     Script for running DeepSpeed model with Mixture of Experts (MoE) integration.
 
-* To run baseline CIFAR-10 model - "python cifar10_tutorial.py"
-* To run DeepSpeed CIFAR-10 model - "bash run_ds.sh"
-* To run DeepSpeed CIFAR-10 model with Mixture of Experts (MoE) - "bash run_ds_moe.sh"
-* To run with different data type (default='fp16') and zero stages (default=0) - "bash run_ds.sh --dtype={fp16|bf16} --stage={0|1|2|3}"
+`run_ds_prmoe.sh`
+    Script for running DeepSpeed model with Pyramid Residual MoE (PR-MoE) integration.
+
+* To run baseline CIFAR-10 model - `python cifar10_tutorial.py`
+* To run DeepSpeed CIFAR-10 model - `bash run_ds.sh`
+* To run DeepSpeed CIFAR-10 model with Mixture of Experts (MoE) - `bash run_ds_moe.sh`
+* To run DeepSpeed CIFAR-10 model with Pyramid Residual MoE (PR-MoE) - `bash run_ds_prmoe.sh`
+* To run with different data type (default=`fp16`) and zero stages (default=`0`) - `bash run_ds.sh --dtype={fp16|bf16} --stage={0|1|2|3}`