Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Hardware][CPU] Add embedding models support for CPU backend #10193

Merged
merged 7 commits into from
Nov 11, 2024
Merged

[Hardware][CPU] Add embedding models support for CPU backend #10193

merged 7 commits into from
Nov 11, 2024

Conversation

Isotr0py
Copy link
Collaborator

@Isotr0py Isotr0py commented Nov 10, 2024 •
edited by github-actions bot
Loading

FIX #9379

  • Implement CPU embedding model runner
  • Add encoder-only attention support to SDPA backend

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mergify mergify bot added the ci/build label Nov 10, 2024
@DarkLight1337 DarkLight1337 self-assigned this Nov 10, 2024
@Isotr0py
fix cpu tests
184055d 
Signed-off-by: Isotr0py <[email protected]>
@Isotr0py Isotr0py marked this pull request as ready for review November 11, 2024 06:09
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 11, 2024
DarkLight1337
DarkLight1337 approved these changes Nov 11, 2024
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, see if the tests pass.

In the future we should also add core_model and cpu_model tags to embedding model tests.

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) November 11, 2024 07:24
@DarkLight1337 DarkLight1337 merged commit 58170d6 into vllm-project:main Nov 11, 2024
66 of 67 checks passed
@Isotr0py Isotr0py deleted the cpu-embed branch November 11, 2024 09:13
@DarkLight1337
Copy link
Member

Oh, the test failed but the CI still passed 😅

@Isotr0py
Copy link
Collaborator Author

Ah, the encoder-decoder tests are broken due to runner refactor. Let me make a quick PR to fix it. 😅

@Isotr0py Isotr0py mentioned this pull request Nov 11, 2024
Merged
@Isotr0py
Copy link
Collaborator Author

I have opened #10218 to fix the broken encoder-decoder CPU runner.

jeejeelee pushed a commit to jeejeelee/vllm that referenced this pull request Nov 11, 2024
@Isotr0py @jeejeelee
[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…
b65a229 
…oject#10193)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 13, 2024
@Isotr0py @rickyyx
[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…
6bcf00b 
…oject#10193)

Signed-off-by: Isotr0py <[email protected]>
omer-dayan pushed a commit to omer-dayan/vllm that referenced this pull request Nov 14, 2024
@Isotr0py @omer-dayan
[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…
84a77ac 
…oject#10193)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: OmerD <[email protected]>
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
@Isotr0py @sumitd2
[Hardware][CPU] Add embedding models support for CPU backend (vllm-pr…
2441c74 
…oject#10193)

Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Sumit Dubey <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

Assignees

@DarkLight1337 DarkLight1337

Labels
ci/build ready ONLY add when PR is ready to merge/full CI is needed x86 CPU
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Unable to embed any text using the vLLM CPU server
2 participants
@Isotr0py @DarkLight1337