Add compute capability 6.x support

[jasonacox]

UPDATED 23-Apr-2024

This proposal adds a script pascal.sh which allows users to add Pascal GPU (e.g. GTX 1060, Tesla P100) to vLLM

The script:

• Adds 6.0, 6.1 and 6.2 (compute capability) GPU architectures to the CMakeLists.txt and Dockerfile files

Run:

# Add Pascal Support
./pascal.sh

# You can now build from source with Pascal GPU support:
pip install -e .

# or build the Docker image with:
DOCKER_BUILDKIT=1 docker build . --target vllm-openai --tag vllm/vllm-openai

Notes:

• Pascal architectures are still supported by latest CUDA.
• I understand that adding 6.x support expands the wheel size beyond the limit so this provides a way for users of this older architecture to use vLLM.

Build and run example:

# Build From Source
git clone https://github.com/vllm-project/vllm.git
cd vllm
./pascal.sh
pip install -e .

# Run OpenAI API Compatible Server
python3 -m vllm.entrypoints.openai.api_server \
    --tensor-parallel-size 4 \
    --worker-use-ray \
    --host 0.0.0.0 \
    --port 8080  \
    --model mistralai/Mistral-7B-Instruct-v0.1 \
    --served-model-name mistralai/Mistral-7B-Instruct-v0.1 \
    --dtype float \
    --max-model-len 20000

Related: #963 #1284

Thank you for the great project!!! 🙏

[cduk]

Why do you delete some of the ROCM cards from the supported list?

[jasonacox]

Why do you delete some of the ROCM cards from the supported list?

Hi @cduk - This PR doesn't touch the ROCM cards. It only adds the Nvidia Pascal (capability 6) cards. Here are the diffs:

...

[cduk]

Maybe I'm mistaken. I was looking at this diff:

[Fuckingnameless]

what's the status on this?

[nkuhn-vmw]

Hello - very interested in this PR - as I also am running multiple P40s and would like to use vLLM

[youkaichao]

Thanks for the effort. You can keep this branch, but this PR is not necessary. Closed.

[jasonacox]

No problem. Thanks @youkaichao

Permanent solution would be: #4290