[Feature]: Inquiry about Multi-modal Support in VLLM for MiniCPM-V2.6

[Dong148]

🚀 The feature, motivation and pitch

I am currently exploring the capabilities of the VLLM library and am interested in understanding its support for multi-modal inputs, particularly for models like MiniCPM-V2.6. I would like to know if VLLM is designed to handle multi-image and video inputs for such models.

Alternatives

1. Model of Interest: MiniCPM-V2.6
2. Types of Input: Multi-image and video
3. Current Understanding:
   • I have reviewed the documentation and initial examples provided with VLLM.

• It seems that both multiple 'image_url' input and list value in image_url is currently not supported.
• However, I am not sure if it supports the processing of multiple images or videos as input to a model like MiniCPM-V2.6.

Questions

1. Does VLLM support the integration of MiniCPM-V2.6 for processing multi-image and video inputs?
2. If yes, could you provide an example or a guide on how to set up and use this feature?
3. If not, are there any plans to extend VLLM's capabilities to support such inputs in the future?

Additional context

[DarkLight1337]

Multi-image input is currently supported for MiniCPU-V specifically (#7122), with some caveats:

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

We are actively working on extending the support for multi-image input - please refer to #4194 for details.

[Dong148]

Multi-image input is currently supported for MiniCPU-V specifically (#7122), with some caveats:

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

We are actively working on extending the support for multi-image input - please refer to #4194 for details.

Thank you for your assistance and for taking the time to help me out. I look forward to exploring more features of VLLM and potentially contributing to its development in the future.

[Patrick10203]

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

Are you sure that building the main branch supports multi image over the open ai api? Because the line https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/chat_utils.py#L179 is still in the main branch

[DarkLight1337]

• It only works in offline inference, not the OpenAI API-compatible server.
• Until the next release, you have to build from source (main branch) to use it.

Are you sure that building the main branch supports multi image over the open ai api? Because the line https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/chat_utils.py#L179 is still in the main branch

I was referring to multi-modal support for MiniCPM-V specifically, not for multi-modal models (+OpenAI server) in general.