[FEATURE] Speculative Decoding

[casper-hansen]

Hi @mrwyattii @jeffra, I believe you could significantly speed up DeepSpeed-MII further by implementing Speculative Decoding in combination with the great framework presented here. Speculative Decoding is widely known by now for giving a ~2x speed up of models, based on the research from Google's DeepMind.

Paper: https://arxiv.org/abs/2211.17192
Code (not original): https://github.com/jaymody/speculative-sampling