Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bugfix] Fix divide by zero when serving Mamba models #9617

Merged
merged 1 commit into from
Oct 23, 2024
Merged

[Bugfix] Fix divide by zero when serving Mamba models #9617

merged 1 commit into from
Oct 23, 2024

Commits on Oct 23, 2024

  1. [Bugfix] Fix divide by zero when serving Mamba models 

    `num_total_gpu` ends up being 0 for attention-free models, which results
in a divide-by-zero in llm_engine.py when running:
```
vllm serve tiiuae/falcon-mamba-7b-instruct
```

We're already guarding against None here so this guards against zero as
well.

I also tried setting `num_gpu_blocks` to `None` in
`determine_num_available_blocks` but a couple of different spots choked
on this.

Signed-off-by: Tyler Michael Smith <[email protected]>
    @tlrmchlsmth
    tlrmchlsmth committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    7e3261d View commit details
    Browse the repository at this point in the history