Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Support thread-based async tokenizer pools #3449

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[Core] Support thread-based async tokenizer pools #3449

wants to merge 4 commits into from

Commits on Mar 19, 2024

  1. [Core] Support thread-based async tokenizer pools 

    vllm-project#2879 added support for using ray to offload tokenization from the asyncio event loop.

This PR extends that to support using a thread pool instead of ray, and makes that the default, with the default pool size determined based on the number of available CPU cores and the tensor parallel size.

The main thing to note is that separate tokenizer instances are used per thread. This is because officially the HF tokenizers are not thread-safe. In practice I think they are unless you're making use of padding/truncation, which we aren't currently but may want to soon (see for example vllm-project#3144).

Also includes some type hint additions to related parts of the code.

This replaces the original PR vllm-project#3206 from before vllm-project#2879 was reworked and merged.
    @njhill
    njhill committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    ebf6967 View commit details
    Browse the repository at this point in the history

  2. Address some review comments

    @njhill
    njhill committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    8810908 View commit details
    Browse the repository at this point in the history

  3. Update tests

    @njhill
    njhill committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    3ca96d1 View commit details
    Browse the repository at this point in the history

  4. Fix test

    @njhill
    njhill committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    9f3abb8 View commit details
    Browse the repository at this point in the history