Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[habana_main bugfix] Fix min bucket boundary calculation #239

Merged
merged 7 commits into from
Sep 5, 2024
Merged

[habana_main bugfix] Fix min bucket boundary calculation #239

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a40794b
Do not pass warmup_mode to execute_model_kwargs
kzawora-intel Sep 2, 2024
637d2df
bugfix: correctly handle large bucket minimums
kzawora-intel Sep 3, 2024
9341617
Merge remote-tracking branch 'origin/habana_main' into private/kzawor…
kzawora-intel Sep 3, 2024
9933ec5
Revert "Do not pass warmup_mode to execute_model_kwargs"
kzawora-intel Sep 3, 2024
e32d203
formatting
kzawora-intel Sep 3, 2024
1475f2c
Fix min bucket boundary calculation
kzawora-intel Sep 4, 2024
3a0bbcf
Merge remote-tracking branch 'origin/habana_main' into private/kzawor…
kzawora-intel Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions vllm/worker/habana_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ def warmup_buckets(bs_bucket_config, seq_bucket_config,
return captured_buckets, omitted_buckets


def next_pow2(value: int):
res = 1
def next_pow2(value: int, base: int):
res = base
while value > 1:
value = (value + 1) // 2
res *= 2
Expand All @@ -150,12 +150,10 @@ def round_up(value: int, k: int):


def find_bucket(value: int, config: Tuple[int, int, int]):
bmin, bstep, bmax = config
if value < bstep:
result = min(next_pow2(value), bstep)
else:
result = round_up(value, bstep)
return result
bmin, bstep, _ = config
next_step = round_up(value, bstep)
next_pow = next_pow2(value, bmin)
return max(bmin, min(next_step, next_pow))


def subtuple(obj: object,
Expand Down
Loading