Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix runtime errors reported when using long input sequence lengths with LoRA #339

Merged
merged 1 commit into from
Sep 27, 2024
Merged

Fix runtime errors reported when using long input sequence lengths with LoRA #339

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions vllm/lora/layers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1203,6 +1203,8 @@ def _get_logits(
).index_select(0, indices_padded).nan_to_num_(nan=float("-inf"),
posinf=float("inf"),
neginf=float("-inf")))
if current_platform.is_hpu():
lora_logits = lora_logits[:logits.shape[0], :]
logits[:,
self.base_layer.org_vocab_size:self.base_layer.org_vocab_size +
lora_logits.shape[1], ] = lora_logits
Expand Down
2 changes: 1 addition & 1 deletion vllm/lora/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,7 +432,7 @@ def __init__(
self.long_lora_context: Optional[LongContextLoRAContext] = None
if current_platform.is_hpu():
self.punica_wrapper = GaudiPunicaWrapper(
max_num_batched_tokens,
3 * max_num_batched_tokens,
max_batches=self.max_num_seqs,
device="hpu")
else:
Expand Down
20 changes: 6 additions & 14 deletions vllm/worker/habana_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1184,9 +1184,9 @@ def prepare_input_tensors(

if self.lora_config:
lora_mapping = LoRAMapping(
lora_index_mapping,
lora_prompt_mapping,
)
**dict(index_mapping=lora_index_mapping,
prompt_mapping=lora_prompt_mapping,
is_prefill=(num_prefills > 0)))
else:
lora_mapping = None

Expand Down Expand Up @@ -1350,9 +1350,9 @@ def warmup_scenario(self,
times = 3 if use_graphs or is_pt_profiler_run else 1
if self.lora_config and not is_lora_profile_run:
lora_mapping = LoRAMapping(
[0] * batch_size * seq_len,
[0] * batch_size * seq_len,
)
**dict(index_mapping=[0] * batch_size * seq_len,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does introducing dict here affect memory footprint? since we dont have control on when its garbage collected and size grows with batch_size*seq_len

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michalkuligowski Nothing has changed from existing code, we are still populating the same dataclass (LoRAMapping) as before, **dict simply is to use key, value notation initialization of class members. Regarding memory footprint, this dataclass needs to be available all the time if LoRA is enabled, it is a must have for LoRA operation. Again nothing specific to Gaudi/HPU, this is the way LoRA feature is implemented in vLLM.

prompt_mapping=[0] * batch_size * seq_len,
is_prefill=is_prompt))
self.set_active_loras(set(), lora_mapping)
if is_prompt:
seqs = [
Expand Down Expand Up @@ -1894,14 +1894,6 @@ def execute_model(
)

if self.lora_config:
from vllm.lora.layers import VocabParallelEmbeddingWithLoRA
modules = unwrap_model(self.model.model)
for module in modules:
if isinstance(module, VocabParallelEmbeddingWithLoRA):
for i in range(0, len(module.punica_wrapper.indices_len)):
module.punica_wrapper.indices_len[
i] = sampling_metadata.selected_token_indices.numel(
)
lora_logits_mask: torch.Tensor = model_input.lora_logits_mask
LoraMask.setLoraMask(
lora_logits_mask.index_select(
Expand Down
Loading