Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement tensor parallelism: nccl support #72

Closed
wants to merge 4 commits into from
Closed

Implement tensor parallelism: nccl support #72

wants to merge 4 commits into from

Conversation

EricLBuehler
Copy link
Owner

@EricLBuehler EricLBuehler commented Apr 4, 2024 •
edited
Loading

Refs #50.

Changes:

  • New feature flags:
    • nccl: cuda + flash-attn

@EricLBuehler EricLBuehler added new feature New feature or request optimization backend Backend work labels Apr 4, 2024
@EricLBuehler EricLBuehler self-assigned this Apr 4, 2024
@EricLBuehler EricLBuehler changed the title Implement tensor parallelism Implement tensor parallelism: nccl support Apr 4, 2024
@EricLBuehler
Copy link
Owner Author

Will reopen in a new PR to restart this.

@EricLBuehler EricLBuehler deleted the parallel_layers branch April 20, 2024 11:25
@hugoabonizio
Copy link
Contributor

hugoabonizio commented Apr 29, 2024 •
edited
Loading

Are there any plans to support this for quantized models in the near future? 👀

@b0xtch
Copy link

b0xtch commented Jun 21, 2024

Any update on the new branch?

@b0xtch b0xtch mentioned this pull request Jun 21, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@EricLBuehler EricLBuehler

Labels
backend Backend work new feature New feature or request optimization
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@EricLBuehler @hugoabonizio @b0xtch