[Model] Initialize deepseek-vl support

[liuyancong-enflame-tech]

Test On NVIDIA L40S

• Test Example

from vllm import LLM
from PIL import Image


from vllm import SamplingParams

sample_params = SamplingParams(temperature=0, max_tokens=1024)

model_path = "/pretrained_models/deepseek-vl-7b-chat"


llm = LLM(
    model=model_path,
    max_model_len=3072,
)
print("model load finish")
prompt = f"You are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.\n User: <image_placeholder> Describe each stage of this image detail.\nAssistant:"

image = Image.open("/opt/TV/VLM01/tests/images/cherry_blossom.jpg")
image = image.convert("RGB")
outputs = llm.generate(
    {
        "prompt": prompt,
        "multi_modal_data": {"image": image},
    },
    sample_params,
)
for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

• Output

The image captures a scene where a tall tower, which appears to be a communication or observation tower, is partially obscured by cherry blossom trees in full bloom. The tower is situated in the background, and the sky behind it is a clear blue.

The cherry blossom trees, which are in the foreground, are in various stages of bloom. Some branches are densely packed with pink blossoms, while others have fewer or no flowers at all. The blossoms are in various shades of pink, ranging from light to deep, indicating the progression of the bloom cycle.

The perspective of the image is from below, looking upwards towards the tower, which gives a sense of scale and grandeur to the tower. The branches of the cherry blossom trees frame the tower, creating a natural border that draws the viewer's eye towards the tower.

There are no discernible texts or other objects in the image. The focus is solely on the tower and the cherry blossom trees, with the blue sky providing a contrasting backdrop. The image does not contain any people or moving elements, suggesting a still, serene moment captured in time.

FIX #3356
FIX #4982

[liuyancong-enflame-tech]

Contributed by enflame-tech

[DarkLight1337]

Thanks for the contribution! I have a few initial comments.

Apart from that, can you add a test case (similar to test_llava.py) to test the correctness of the model in CI?

[liuyancong-enflame-tech]

now support deepseek-ai/deepseek-vl-7b-chat deepseek-ai/deepseek-vl-1.3b-chat

[liuyancong-enflame-tech]

This model depends on timm>=0.9.16, which depends on torch, but it will conflict with the dependencies of other third-party components and cause the pipeline to fail. Therefore, running this model requires additional installation. I don’t know if this is appropriate. In addition, it depends on many modules of timm, which is difficult to remove.

[DarkLight1337]

This model depends on timm>=0.9.16, which depends on torch, but it will conflict with the dependencies of other third-party components and cause the pipeline to fail. Therefore, running this model requires additional installation. I don’t know if this is appropriate. In addition, it depends on many modules of timm, which is difficult to remove.

Can you implement the individual timm modules inside vLLM? (where possible, you should use vLLM-specific layers to improve the performance anyway)

[liuyancong-enflame-tech]

This model depends on timm>=0.9.16, which depends on torch, but it will conflict with the dependencies of other third-party components and cause the pipeline to fail. Therefore, running this model requires additional installation. I don’t know if this is appropriate. In addition, it depends on many modules of timm, which is difficult to remove.

Can you implement the individual timm modules inside vLLM? (where possible, you should use vLLM-specific layers to improve the performance anyway)

OK，I will try to do this and I think it will take some time

[DarkLight1337]

This model depends on timm>=0.9.16, which depends on torch, but it will conflict with the dependencies of other third-party components and cause the pipeline to fail. Therefore, running this model requires additional installation. I don’t know if this is appropriate. In addition, it depends on many modules of timm, which is difficult to remove.

Can you implement the individual timm modules inside vLLM? (where possible, you should use vLLM-specific layers to improve the performance anyway)

OK，I will try to do this and I think it will take some time

You can make use of our implementation of CLIPVisionModel to save some effort.

[DarkLight1337]

To speed up the CI queue for #5905, I've cancelled the distributed tests for the latest CI run in this PR since they won't pass anyway until #5905 has been merged. Please merge main into your branch after that happens so that the CI can pass once again.

[liuyancong-enflame-tech]

This test [tests/models/test_deepseek_vl.py] case depends on the project https://github.com/deepseek-ai/DeepSeek-VL, and it seems that pip installation will fail when build the docker .I think it is possible not to add this test case

And The [examples/deepseek_vl_example.py] can run successfully.

[DarkLight1337]

This test [tests/models/test_deepseek_vl.py] case depends on the project https://github.com/deepseek-ai/DeepSeek-VL, and it seems that pip installation will fail when build the docker .I think it is possible not to add this test case

In this case it won't function for users of vLLM either since they can't install it (so you should still keep the tests). Can you figure out which dependency is causing the issue?

[liuyancong-enflame-tech]

This test [tests/models/test_deepseek_vl.py] case depends on the project https://github.com/deepseek-ai/DeepSeek-VL, and it seems that pip installation will fail when build the docker .I think it is possible not to add this test case

In this case it won't function for users of vLLM either since they can't install it (so you should still keep the tests). Can you figure out which dependency is causing the issue?
In the tests/models/test_deepseek_vl.py test, when hf runner loads the model, if we do not import deepseek_vl, an error will occur
(ValueError: The checkpoint you are trying to load has model type multi_modality but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.)
This is because the model part of the code is not in the hf repository, but in the github repository. In the code: https://github.com/deepseek-ai/DeepSeek-VL/blob/main/deepseek_vl/models/modeling_vlm.py, we can see some registration codes:
AutoConfig.register("multi_modality", MultiModalityConfig)
AutoModelForCausalLM.register(MultiModalityConfig, MultiModalityCausalLM)
If I add the dependency in vllm\requirements-test.txt:
deepseek_vl@git+https://github.com/deepseek-ai/DeepSeek-VL.git@main
Then, docker will report an error when building the image. The error is that metadata is None and cannot be recognized. The package can be packaged into whl, but it will not be found when it is installed.

[DarkLight1337]

You can manually register the model to HuggingFace inside the test case.

[liuyancong-enflame-tech]

You can manually register the model to HuggingFace inside the test case.

Ok, I'll try it

[liuyancong-enflame-tech]

buildkite/fastcheck/pr/tensorizer-metrics-tracing-test — Failed (exit status 1)
What functionality does this test case test?

[DarkLight1337]

It's unrelated to this PR.

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

[SinanAkkoyun]

Will this PR get merged?

[SinanAkkoyun]

Or rather where could I help, what is missing or needs work, only the models/test_deepseek_vl.py?

[DarkLight1337]

Or rather where could I help, what is missing or needs work, only the models/test_deepseek_vl.py?

Sorry for forgetting about this! I think we need to fix the merge conflicts and update the tests (you might need to fork this branch and create a new PR). Afterwards we can work on removing redundant code in the model file.

[SinanAkkoyun]

Sorry for forgetting about this!

No problem at all! I am sorry to have prematurely commented here, as Qwen2 VL architecturally fits my usecase way better.
Although I am having problems with the OpenAI endpoint, as Qwen2 VL is the only model that generates junk, I opened an issue for that #9732