Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration #10022

Merged
merged 8 commits into from
Nov 6, 2024
Merged

[Model][LoRA]LoRA support added for Qwen2VLForConditionalGeneration #10022

Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
534910b
Support lora of Qwen2VLForConditionalGeneration
ericperfect Nov 5, 2024
ff74379
format code
ericperfect Nov 5, 2024
c1eeb5d
format code
ericperfect Nov 5, 2024
394fa07
format code
ericperfect Nov 6, 2024
3e15502
format code
ericperfect Nov 6, 2024
2c6f7fc
format code
ericperfect Nov 6, 2024
21c5aa4
format code
ericperfect Nov 6, 2024
bf0dc7a
add TODO(TODO Support LoRA for the visual encoder in the future.) mes…
ericperfect Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 27 additions & 4 deletions vllm/model_executor/models/qwen2_vl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@

from vllm.attention import AttentionMetadata
from vllm.attention.selector import _Backend
from vllm.config import CacheConfig, MultiModalConfig
from vllm.config import CacheConfig, LoRAConfig, MultiModalConfig
from vllm.distributed import get_pp_group, parallel_state
from vllm.distributed import utils as dist_utils
from vllm.inputs import (INPUT_REGISTRY, DecoderOnlyInputs, DummyData,
Expand All @@ -66,7 +66,7 @@
from vllm.transformers_utils.config import uses_mrope
from vllm.transformers_utils.processor import cached_get_processor

from .interfaces import SupportsMultiModal, SupportsPP
from .interfaces import SupportsLoRA, SupportsMultiModal, SupportsPP
from .utils import (PPMissingLayer, get_vit_attn_backend,
is_pp_missing_parameter,
make_empty_intermediate_tensors_factory)
Expand Down Expand Up @@ -929,13 +929,36 @@ def input_processor_for_qwen2_vl(
@INPUT_REGISTRY.register_dummy_data(dummy_data_for_qwen2_vl)
@INPUT_REGISTRY.register_input_processor(input_processor_for_qwen2_vl)
class Qwen2VLForConditionalGeneration(nn.Module, SupportsMultiModal,
SupportsPP):
SupportsLoRA, SupportsPP):
packed_modules_mapping = {
"qkv_proj": [
"q_proj",
"k_proj",
"v_proj",
],
"gate_up_proj": [
"gate_proj",
"up_proj",
],
}

# LoRA specific attributes
supported_lora_modules = [
"qkv_proj",
"o_proj",
"gate_up_proj",
"down_proj",
]
embedding_modules = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although vllm currently only supports lora inference for language model, considering that inference for vision modules will be supported in the future, we should also add vision encoder related modules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, currently fine-tuning only the language model with LoRA achieves good results and meets the needs of most users. Further optimizations can be made later if necessary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree with your view. I just think it might be better to complete the additions first. If you're interested, you can refer to Qwen-VL's implementation, see: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen.py#L1015-L1046.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll take a look.

embedding_padding_modules = []

def __init__(self,
config: Qwen2VLConfig,
multimodal_config: MultiModalConfig,
cache_config: Optional[CacheConfig] = None,
quant_config: Optional[QuantizationConfig] = None) -> None:
quant_config: Optional[QuantizationConfig] = None,
lora_config: Optional[LoRAConfig] = None) -> None:

super().__init__()

assert not cache_config.enable_prefix_caching, \
Expand Down