Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add internlm-xcomposer2, forward and sft supported #511

Merged
merged 52 commits into from
Apr 28, 2024
Merged

Add internlm-xcomposer2, forward and sft supported #511

merged 52 commits into from
Apr 28, 2024

Conversation

cocoshe
Copy link
Contributor

@cocoshe cocoshe commented Apr 20, 2024

对齐 internlm-xcomposer2-7b 的前向,以及sft的step loss和step lr

Copy link

paddle-bot bot commented Apr 20, 2024

Thanks for your contribution!

@LokeZhou
Copy link
Collaborator

曲线图是否是减了block或layer数的?如果是,标注一下;另外这个曲线不太平滑,是否torch的也这样?

@cocoshe
Copy link
Contributor Author

cocoshe commented Apr 23, 2024

主要参数配置:
训练参数 gradient_accumulation_steps=1
组网配置 num_hidden_layers=1, dtype="float32"

  • pytorch 与 paddle 的 step loss

step_loss_accu_1

  • pytorch 与 paddle 的 step lr

step_lr_accu_1

  • pytorch 与 paddle 的 diff loss(差值)

diff_loss_accu_1

  • pytorch 与 paddle 的 diff lr(差值) (全为0)

diff_lr_accu_1

@LokeZhou LokeZhou merged commit 3d5ec4d into PaddlePaddle:develop Apr 28, 2024
3 checks passed
westfish pushed a commit to westfish/PaddleMIX that referenced this pull request Sep 25, 2024
对齐 internlm-xcomposer2-7b 的前向,以及sft的step loss和step lr

---------

Co-authored-by: LokeZhou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants