[Bug]: Model architectures ['LlavaForCausalLM'] are not supported for now in vllm 0.4.0.post1

[stikkireddy]

Your current environment

The output of `python collect_env.py`

🐛 Describe the bug

I am using:

from vllm import LLM
model = LLM(
            model="llava-hf/llava-1.5-13b-hf",
            image_input_type="pixel_values",
            download_dir="/tmp/models",
            image_token_id=32000,
            image_input_shape="1,3,336,336",
            image_feature_size=576,
        )

which throws error: ValueError: Model architectures ['LlavaForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'LlamaForCausalLM', 'LlavaForConditionalGeneration', 'LLaMAForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'OLMoForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PhiForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'XverseForCausalLM']

After checking huggingface llava-1.5-7b-hf uses LlavaForConditionalGeneration and llava-1.5-13b-hf uses LlavaForCausalLM?

Any easy workaround / fix for this?

[stikkireddy]

maybe this is a feature request idk enough about the intent of supporting llava.

[DarkLight1337]

Don't know how I missed that when writing PR #3978. Interesting... was the 7b model modified specifically to facilitate the proof-of-concept for vLLM?

[stikkireddy]

Unsure I went and reviewed this #3042 and it seems llava 7b and 13b have different classes. We only supported LlavaForConditionalGeneration but not LlavaForCausalLM.

[stikkireddy]

@xwjiang2010 do you happen to know the effort to support LLaVa 13b. Pinging you since you worked on the initial PR for vision models.

[DarkLight1337]

I think it's just a typo in their HuggingFace config.json. I can successfully load the 13b model in vLLM by pretending that it has LlavaForConditionalGeneration architecture. I applied the following patch on top of DarkLight1337:openai-vision-api:

diff --git a/tests/conftest.py b/tests/conftest.py
index a7e8963..61ca4fe 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -131,6 +131,7 @@ _STR_DTYPE_TO_TORCH_DTYPE = {
 
 _VISION_LANGUAGE_MODELS = {
     "llava-hf/llava-1.5-7b-hf": LlavaForConditionalGeneration,
+    "llava-hf/llava-1.5-13b-hf": LlavaForConditionalGeneration,
 }
 
 
diff --git a/vllm/model_executor/models/__init__.py b/vllm/model_executor/models/__init__.py
index 17fc970..54a077d 100755
--- a/vllm/model_executor/models/__init__.py
+++ b/vllm/model_executor/models/__init__.py
@@ -33,6 +33,8 @@ _MODELS = {
     "LlamaForCausalLM": ("llama", "LlamaForCausalLM"),
     "LlavaForConditionalGeneration":
     ("llava", "LlavaForConditionalGeneration"),
+    "LlavaForCausalLM":
+    ("llava", "LlavaForConditionalGeneration"),
     # For decapoda-research/llama-*
     "LLaMAForCausalLM": ("llama", "LlamaForCausalLM"),
     "MistralForCausalLM": ("llama", "LlamaForCausalLM"),

Update: I have tested with some other models:

• There is no need to apply this patch to llava-hf/bakLlava-v1-hf since it has LlavaForConditionalGeneration architecture as advertised.
• Even with this patch, I cannot load models with LlavaLlamaForCausalLM architecture, notably liuhaotian/llava-v1.5-7b.

[stikkireddy]

hmm okay then i can probably clone their model and just modify the config.json to use the LlavaForConditionalGeneration architecture.

[DarkLight1337]

I have updated my PR accordingly.

[Jianzhao-Huang]

I think it's just a typo in their HuggingFace config.json. I can successfully load the 13b model in vLLM by pretending that it has LlavaForConditionalGeneration architecture. I applied the following patch on top of DarkLight1337:openai-vision-api:

diff --git a/tests/conftest.py b/tests/conftest.py
index a7e8963..61ca4fe 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -131,6 +131,7 @@ _STR_DTYPE_TO_TORCH_DTYPE = {
 
 _VISION_LANGUAGE_MODELS = {
     "llava-hf/llava-1.5-7b-hf": LlavaForConditionalGeneration,
+    "llava-hf/llava-1.5-13b-hf": LlavaForConditionalGeneration,
 }
 
 
diff --git a/vllm/model_executor/models/__init__.py b/vllm/model_executor/models/__init__.py
index 17fc970..54a077d 100755
--- a/vllm/model_executor/models/__init__.py
+++ b/vllm/model_executor/models/__init__.py
@@ -33,6 +33,8 @@ _MODELS = {
     "LlamaForCausalLM": ("llama", "LlamaForCausalLM"),
     "LlavaForConditionalGeneration":
     ("llava", "LlavaForConditionalGeneration"),
+    "LlavaForCausalLM":
+    ("llava", "LlavaForConditionalGeneration"),
     # For decapoda-research/llama-*
     "LLaMAForCausalLM": ("llama", "LlamaForCausalLM"),
     "MistralForCausalLM": ("llama", "LlamaForCausalLM"),

Update: I have tested with some other models:

• There is no need to apply this patch to llava-hf/bakLlava-v1-hf since it has LlavaForConditionalGeneration architecture as advertised.
• Even with this patch, I cannot load models with LlavaLlamaForCausalLM architecture, notably liuhaotian/llava-v1.5-7b.

Thank you for your effort in supporting llava in vllm! Indeed this is a great job.

After checking the code in original llava code repo(https://github.com/haotian-liu/LLaVA), I found exactly llava-1.5-13b use the LlavaForCausalLM class, so I think it's not a typo in the HuggingFace's config.json.

I don't know whether there will be a runtime bug or some performance losses after directly modifying the config.json to use the LlavaForConditionalGeneration architecture in llava-1.5-13b. Could you please check this issue?

[DarkLight1337]

Thank you for your effort in supporting llava in vllm! Indeed this is a great job.

After checking the code in original llava code repo(https://github.com/haotian-liu/LLaVA), I found exactly llava-1.5-13b use the LlavaForCausalLM class, so I think it's not a typo in the HuggingFace's config.json.

I don't know whether there will be a runtime bug or some performance losses after directly modifying the config.json to use the LlavaForConditionalGeneration architecture in llava-1.5-13b. Could you please check this issue?

@Jianzhao-Huang It doesn't look like HuggingFace has LlavaForCausalLM, so I think both models should use LlavaForConditionalGeneration as the name.

In any case, I have just pushed a commit to #3978 which adds the 13b model to the LLaVA test case that checks its consistency against the native HuggingFace model.

[DarkLight1337]

Hmm, seems that the GPU memory fails to be freed between testing each model. Does anyone know how to fix this issue?

[DarkLight1337]

I have investigated further and it seems that the CI/CD infrastructure cannot even load the 13B model into memory (I removed all other LLaVA models from the test and it still OOMed). Not sure what I can do about that...

[Iven2132]

Any fix?

[DarkLight1337]

The incorrect architecture in config.json has been fixed on HuggingFace, as per this discussion.