[Frontend] Chat-based Embeddings API #9759
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
DarkLight1337
changed the title
mergify
bot
added
documentation
|
This pull request has merge conflicts that must be resolved before it can be |
DarkLight1337
force-pushed
the
chat-embeddings-api
branch
from
3cc07d5
to
9cd1ac3
Compare
| Since VLM2Vec has the same model architecture as Phi-3.5-Vision, we have to explicitly pass ``--task embedding`` | ||
| to run this model in embedding mode instead of text generation mode. | ||
|
|
||
| Since this schema is not defined by OpenAI client, we post a request to the server using the lower-level ``requests`` library: |
There was a problem hiding this comment.
Just leaving this as a thought here: should we perhaps have a fork of the openai client that support our extensions explicitly?
There was a problem hiding this comment.
This sounds good, but not sure whether we have bandwidth to maintain it 😅
There was a problem hiding this comment.
I suggest opening an issue for this.
| @@ -7,7 +7,7 @@ class PoolingParams( | |||
| msgspec.Struct, | |||
| omit_defaults=True, # type: ignore[call-arg] | |||
| array_like=True): # type: ignore[call-arg] | |||
| """Pooling parameters for pooling. | |||
| """Pooling parameters for embeddings API. | |||
There was a problem hiding this comment.
I might be missing something, but the additional_data attribute doesn't seem to be used anywhere. Which is good, because it can by anything and is passed without validation from the request to the Pooler.forward() method as part of the PoolingMetadata object. If there is no use case for this, can we remove it in this PR?
There was a problem hiding this comment.
@robertgshaw2-neuralmagic originally added this (#4800 (comment)). I am not sure whether this is still relevant since we can now set the pooling strategy via CLI (#9697).
There was a problem hiding this comment.
@robertgshaw2-neuralmagic can you comment on this?
There was a problem hiding this comment.
Meanwhile let's merge this PR first.
github-actions
bot
added
the
ready
| except asyncio.CancelledError: | ||
| return self.create_error_response("Client disconnected") | ||
| except ValueError as e: | ||
| # TODO: Use a vllm-specific Validation Error | ||
| return self.create_error_response(str(e)) | ||
|
|
||
| try: |
There was a problem hiding this comment.
@DarkLight1337 apologies I didn't get a chance to review this last week .. ran into some things while trying to resolve conflicts with another pending PR :)
It looks like there are a few changes not directly related to chat embeddings.
Wondering the reason for splitting into two try/except blocks here in particular (I guess similar in serving_embedding.py)
There was a problem hiding this comment.
It is just to make clear that asyncio.CancelledError can only happen while iterating through the result generator.
There was a problem hiding this comment.
This isn't necessary and makes the code more convoluted imo... I may open a PR to change and we can discuss there :)
Signed-off-by: Linkun Chen <[email protected]>
Signed-off-by: Richard Liu <[email protected]>
These were changed to separate blocks in vllm-project#9759 but I feel it's cleaner/clearer as a single block. It actually doesn't matter which parts of the block raise the specific exceptions in the except clauses, we still want to handle them in the same way.
These were changed to separate blocks in vllm-project#9759 but I feel it's cleaner/clearer as a single block. It actually doesn't matter which parts of the block raise the specific exceptions in the except clauses, we still want to handle them in the same way.
These were changed to separate blocks in vllm-project#9759 but I feel it's cleaner/clearer as a single block. It actually doesn't matter which parts of the block raise the specific exceptions in the except clauses, we still want to handle them in the same way. Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Loc Huynh <[email protected]>
Signed-off-by: Sumit Dubey <[email protected]>
This PR extends the existing Embeddings API to accept chat conversations similar to Chat Completions API. This enables multi-modal conversations to be passed to the embedding model.
To reduce code duplication, I've also factored out the common code for handling completion and chat-based inputs into the base
OpenAIServingclass.FIX #8967
FIX #9303 (comment)