Add DPO support for DeepSpeed-Chat

[stceum]

Considering the advantages of DPO(Direct Preference Optimization) as being "stable, performant, and computationally lightweight, eliminating the need for fitting a reward model, sampling from the LM during fine-tuning, or performing significant hyperparameter tuning", we add DPO support for DeepSpeed-Chat.

[stceum]

Accidentally closed the PR.. Sorry :(

[boqiny]

Hi, Thanks for sharing this. Any updates about this PR? Are we planning to merge this feature in?

[stceum]

Hi, Thanks for sharing this. Any updates about this PR? Are we planning to merge this feature in?

Yes, of course! The code for step2 DPO seems to be running as expected. I have done my best to make the code easy to understand and maintain the same style as the other steps. After reviewing the PR again just now, it appears to be missing a README. I will complete it ASAP.

I really appreciate your reminder :)

[tjruwase]

@stceum, thanks for this great contribution. Apologies for the delayed review. I have approved. Please let me know when it is good to merge.

[stceum]

@stceum, thanks for this great contribution. Apologies for the delayed review. I have approved. Please let me know when it is good to merge.

Hi, I've committed the README.md and the example training log. I believe it's ready on my end. Please let me know if you have any questions or find any issues with the PR. I apologize for any inconvenience during the holiday season. Happy New Year! 🎉🎉🎉