Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ZeRO2 with Pipeline parallelism #411

Closed
hyunwoongko opened this issue Sep 25, 2021 · 4 comments
Closed

ZeRO2 with Pipeline parallelism #411

hyunwoongko opened this issue Sep 25, 2021 · 4 comments
Labels
feature request New feature or request

Comments

@hyunwoongko
Copy link
Member

hyunwoongko commented Sep 25, 2021 •
edited
Loading

I found out that the GPT-NeoX team has implemented pipeline parallelism with ZeRO2 (#62). Does this currently run efficiently? I'd love to hear about the current status of this work.
@hyunwoongko hyunwoongko added the feature request New feature or request label Sep 25, 2021
@hyunwoongko
Copy link
Member Author

hyunwoongko commented Sep 25, 2021 •
edited
Loading

https://github.com/EleutherAI/DeeperSpeed/blob/main/deepspeed/runtime/pipe/engine.py#L63
it seems to be preventing the use of ZeRO2 and the pipeline engine together. (The original deep speed also has this line.) Have you concluded that ZeRO2 is incompatible with pipeline parallelism?

@hyunwoongko
Copy link
Member Author

hyunwoongko commented Sep 25, 2021 •
edited
Loading

@stas00 @siddk @deepak94
I ping you guys because it's a topic that's fairly close to our work.

@hyunwoongko hyunwoongko mentioned this issue Sep 25, 2021
Closed
@hyunwoongko
Copy link
Member Author

hyunwoongko commented Sep 25, 2021 •
edited
Loading

I found this microsoft/DeepSpeed#823 and microsoft/DeepSpeed#980 ZeRO2 is not compatible with pipeline parallelism. I close the issue.

@StellaAthena
Copy link
Member

Hahaha yup. We basically went through that same process of thought four months ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@StellaAthena @hyunwoongko