[Model] Adding Support for Qwen2VL as an Embedding Model. Using MrLight/dse-qwen2-2b-mrl-v1

[FurtherAI]

[Model] Adding Support for Qwen2VL as an Embedding Model. Using MrLight/dse-qwen2-2b-mrl-v1

---

This is related to #9303 and #9759 and adds support for Qwen2VL as an embedding model. This is only a couple of lines of changes and a jinja template for the API.

I have tested that this is working correctly with MrLight/dse-qwen2-2b-mrl-v1. The test file is a bit ugly though. I don't think the hf_runner supports the input processing that needs to be done for this model because it uses process_vision_info and applies a chat template manually. It also takes a minimum size image placeholder for text embedding.

Looking for a little help with how the test should be written to cleanly handle these differences.

• Update documentation to include this.
• Add a test for the API maybe.

I have tested the API, but didn't look into how to add it to test_vision_embedding.py yet.

cc @DarkLight1337

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[FurtherAI]

@DarkLight1337 Do you want any changes to the vision embedding API test? It looks like generally those tests are more about the endpoint than if the model they're calling is returning the correct output.

[DarkLight1337]

Do you want any changes to the vision embedding API test? It looks like generally those tests are more about the endpoint than if the model they're calling is returning the correct output.

It looks quite complicated. Can you factor out the common code so it is more like the other embedding model tests?

[FurtherAI]

Do you want any changes to the vision embedding API test? It looks like generally those tests are more about the endpoint than if the model they're calling is returning the correct output.

It looks quite complicated. Can you factor out the common code so it is more like the other embedding model tests?

Do you mean the offline test looks complicated? I think the question still stands about if you want anything added to the API test.

[DarkLight1337]

Do you want any changes to the vision embedding API test? It looks like generally those tests are more about the endpoint than if the model they're calling is returning the correct output.

It looks quite complicated. Can you factor out the common code so it is more like the other embedding model tests?

Do you mean the offline test looks complicated? I think the question still stands about if you want anything added to the API test.

I was referring to the offline API tests. I don't see any changes to the online API tests though (which are under entrypoints/openai) - what do you mean by API tests?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @FurtherAI.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[FurtherAI]

Do you want any changes to the vision embedding API test? It looks like generally those tests are more about the endpoint than if the model they're calling is returning the correct output.

It looks quite complicated. Can you factor out the common code so it is more like the other embedding model tests?

Do you mean the offline test looks complicated? I think the question still stands about if you want anything added to the API test.

I was referring to the offline API tests. I don't see any changes to the online API tests though (which are under entrypoints/openai) - what do you mean by API tests?

Those are the API tests I was referring to.

I can improve the code for the test I think. I'll test it next week.

[DarkLight1337]

There is no need to update online API tests for new models.

[DarkLight1337]

PTAL at the failing model tests. Can we omit using qwen2_vl_utils?

[FurtherAI]

PTAL at the failing model tests. Can we omit using qwen2_vl_utils?

Yeah I can probably drop that. I think it is only resizing the image

[DarkLight1337]

Please also merge from main to fix the broken CI.

[DarkLight1337]

The tests have passed, thanks for your patience!