-
-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Model][Pixtral] Optimizations for input_processor_for_pixtral_hf #9514
Conversation
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
# Backward iteration for replacement without affecting known indices | ||
for placeholder_idx, replace_tokens in zip(reversed(placeholder_indices), | ||
reversed(replace_tokens_list)): | ||
new_token_ids[placeholder_idx:placeholder_idx + 1] = replace_tokens |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't know you could assign slices of a different size to a list, TIL!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work! You can merge this once you have tested this using the example script.
I tested with both the example script and a full eval on chartqa, thanks! I will work on making an integration test for this model since we can directly compare with HF |
…lm-project#9514) Signed-off-by: charlifu <[email protected]>
…lm-project#9514) Signed-off-by: Vinay Damodaran <[email protected]>
…lm-project#9514) Signed-off-by: Alvant <[email protected]>
…lm-project#9514) Signed-off-by: Amit Garg <[email protected]>
…lm-project#9514) Signed-off-by: qishuai <[email protected]>
…lm-project#9514) Signed-off-by: Sumit Dubey <[email protected]>
This PR improves the performance of the Pixtral input processor function a bit.
For
new_prompt
we split the prompt only once, process images and prompt parts together, and generate new tokens for each image more directly.For
new_token_ids
we now find all image tokens at once, process images and tokens together more efficiently, and replace tokens from end to start to avoid index issues.