[LLAVA-NEXT] Warning: The model weights are not tied. Please use the tie_weightsmethod before using the infer_auto_device function.

[aliencaocao]

System Info

• transformers version: 4.40.0.dev0
• Platform: Linux-5.19.0-051900rc6-generic-x86_64-with-glibc2.35
• Python version: 3.9.18
• Huggingface_hub version: 0.21.1
• Safetensors version: 0.4.2
• Accelerate version: 0.28.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.4.0.dev20240326+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@amyeroberts

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import LlavaNextForConditionalGeneration
model = LlavaNextForConditionalGeneration.from_pretrained('panoyo9829/llava-v1.6-mistral-7b-bnb-4bit', low_cpu_mem_usage=True, device_map='auto')

Running this will yield a The model weights are not tied. Please use the tie_weightsmethod before using theinfer_auto_device function. warning. Inference result seems correct though, but I'm not sure if the warning should still be paid attention to.

Expected behavior

No warning

[amyeroberts]

Hi @aliencaocao, thanks for opening this issue!

I'm unable to replicate this error when running the snippet on main.

The checkpoint in the example panoyo9829/llava-v1.6-mistral-7b-bnb-4bit doesn't appear to be one compatible with a model in the transformers library.

It maps to LlavaMistralForCausalLM which isn't defined in transformers.

When running the snippet, the warning I get is about weights in the model being randomly initialized.

[aliencaocao]

I can reproduce this with the original untouched fp16 model as long as I use device_map:

from transformers import LlavaNextForConditionalGeneration
model = LlavaNextForConditionalGeneration.from_pretrained('llava-hf/llava-v1.6-mistral-7b-hf', low_cpu_mem_usage=True, device_map='auto')

I can repro in fresh colab using latest main branch too: https://colab.research.google.com/drive/1pKm8yQ2JBfeQmFQbXgFNoozfSAckEgJu?usp=sharing

Note the notebook cannot be executed as it OOM RAM when loading fp16 weights. Notheless, the warning still came out:

It might be better for accelerate to resolve this issue though, since warning come from it. Feel free to close this if you think it is not a transformers issue.

[amyeroberts]

Thanks for sharing the details.

The error indicates that the check_tied_parameters_in_config(model) for the model is evaluating as True.

@muellerzr Any suggestion on how to handle this? The warning indicates calling model.tie_weights(), but it's not obvious how this is supposed to be done if we're loading with from_pretrained and device_map="auto"

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[aliencaocao]

issue still there

[LysandreJik]

@SunMarc if you have a second to check this :)

[SunMarc]

Hi @aliencaocao, the above PR should fix this. This happens because a config attribute was not set correctly but there was no impact apart from triggering a warning message.
@amyeroberts We shouldn't get this warning at all since in from_pretained(), we make sure to call tie_weights before infer_auto_device_map. In this case, the issue was that the tie_word_embedding was incorrectly set in the config of the model.

[aliencaocao]

there was no impact apart from triggering a warning message.

Glad to have that confirmation.

Thanks for the fix! Would not have noticed this as I also checked the source code and saw the tie_weights call too.