ZeRO++

[HeyangQin]

We are releasing ZeRO++, a system of communication optimization strategies built on top of ZeRO to offer unmatched efficiency for large model training, regardless of batch size limitations or cross-device bandwidth constraints. ZeRO++ leverages quantization, in combination with data, and communication remapping, to reduce total communication volume by 4x compared with ZeRO, without impacting model quality.

ZeRO++ has three sets of communication optimizations:

• Quantized weight communication for ZeRO (qwZ)
• Hierarchical weight partition for ZeRO (hpZ)
• Quantized gradient communication for ZeRO (qgZ)

Please refer to our ZeRO++ paper and blog for details.

ZeRO++ is particularly useful for:

a) Small batch-size per GPU: Whether pre-training large models on thousands of GPUs or fine-tuning them on hundreds or even dozens of GPUs, when batch-size per GPU is small, ZeRO++ offers up to 2.2x higher throughput compared to ZeRO, directly reducing training time and cost.

b) Low-bandwidth clusters: ZeRO++ enables low-bandwidth clusters to achieve similar throughput as those with 4x higher bandwidth. Therefore, ZeRO++ makes efficient large model training accessible across a wider variety of clusters.

Contributors:
This code release was made possible by the contributions of the following people from the DeepSpeed Team:
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Connor Holmes, Samyam Rajbhandari, Olatunji Ruwase, Ammar Ahmad Awan, Jeff Rasley, Michael Wyatt, Yuxiong He (team lead)

[jeffra]

🎉🎉🎉