Skip to content

Commit

Permalink
refine and verify ipex-llm-serving-xpu docker document (#10615)
Browse files Browse the repository at this point in the history 
* refine serving on cpu/xpu

* minor fix

* replace localhost with 0.0.0.0 so that service can be accessed through ip address
  • Loading branch information
@liu-shaojun
liu-shaojun authored Apr 2, 2024
1 parent 89d780f commit 20a5e72
Show file tree
Hide file tree
Showing 2 changed files with 166 additions and 99 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/manually_build.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ on:
tag:
description: 'docker image tag (e.g. 2.1.0-SNAPSHOT)'
required: true
default: 'latest'
default: '2.1.0-SNAPSHOT'
type: string
workflow_call:
inputs:
Expand All @@ -32,7 +32,7 @@ on:
tag:
description: 'docker image tag (e.g. 2.1.0-SNAPSHOT)'
required: true
default: 'latest'
default: '2.1.0-SNAPSHOT'
type: string

env:
Expand Down
261 changes: 164 additions & 97 deletions docker/llm/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Here's a demonstration of how to navigate the tutorial in the explorer:
**3.3 Performance Benchmark**: We provide a benchmark tool help users to test all the benchmarks and record them in a result CSV.

```bash
cd /llm//benchmark/all-in-one
cd /llm/benchmark/all-in-one
```

Users can provide models and related information in config.yaml.
Expand Down Expand Up @@ -144,9 +144,9 @@ Then, execute `bash run-spr.sh`, which will generate output results in `results.
bash run-spr.sh
```

For further details and comprehensive functionality of the benchmark tool, please refer to the [all-in-one benchmark tool](https://github.com/intel-analytics/BigDL/tree/main/python/llm/dev/benchmark/all-in-one).
For further details and comprehensive functionality of the benchmark tool, please refer to the [all-in-one benchmark tool](https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/dev/benchmark/all-in-one).

Additionally, for examples related to Inference with Speculative Decoding, you can explore [Speculative-Decoding Examples](https://github.com/intel-analytics/BigDL/tree/main/python/llm/example/CPU/Speculative-Decoding).
Additionally, for examples related to Inference with Speculative Decoding, you can explore [Speculative-Decoding Examples](https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/CPU/Speculative-Decoding).



Expand Down Expand Up @@ -202,19 +202,26 @@ For example, if your model is Llama-2-7b-chat-hf and mounted on /llm/models, you
python chat.py --model-path /llm/models/Llama-2-7b-chat-hf
```

To run inference using `IPEX-LLM` using xpu, you could refer to this [documentation](https://github.com/intel-analytics/IPEX/tree/main/python/llm/example/GPU).
To run inference using `IPEX-LLM` using xpu, you could refer to this [documentation](https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/GPU).


## IPEX-LLM Serving on CPU
FastChat is an open platform for training, serving, and evaluating large language model based chatbots. You can find the detailed information at their [homepage](https://github.com/lm-sys/FastChat).

### Boot container
IPEX-LLM is integrated into FastChat so that user can use IPEX-LLM as a serving backend in the deployment.

Pull image:
```
### 1. Prepare ipex-llm-serving-cpu Docker Image

Run the following command:

```bash
docker pull intelanalytics/ipex-llm-serving-cpu:2.1.0-SNAPSHOT
```

You could use the following bash script to start the container. Please be noted that the CPU config is specified for Xeon CPUs, change it accordingly if you are not using a Xeon CPU.
### 2. Start ipex-llm-serving-cpu Docker Container

Please be noted that the CPU config is specified for Xeon CPUs, change it accordingly if you are not using a Xeon CPU.

```bash
export DOCKER_IMAGE=intelanalytics/ipex-llm-serving-cpu:2.1.0-SNAPSHOT
export CONTAINER_NAME=my_container
Expand All @@ -229,102 +236,132 @@ docker run -itd \
-v $MODEL_PATH:/llm/models \
$DOCKER_IMAGE
```
After the container is booted, you could get into the container through `docker exec`.

### Models
Access the container:
```
docker exec -it $CONTAINER_NAME bash
```

Using IPEX-LLM in FastChat does not impose any new limitations on model usage. Therefore, all Hugging Face Transformer models can be utilized in FastChat.
### 3. Serving with FastChat

FastChat determines the Model adapter to use through path matching. Therefore, in order to load models using IPEX-LLM, you need to make some modifications to the model's name.
To serve using the Web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the web server and model workers.

A special case is `ChatGLM` models. For these models, you do not need to do any changes after downloading the model and the `IPEX-LLM` backend will be used automatically.
- #### **Step 1: Launch the Controller**
```bash
python3 -m fastchat.serve.controller &
```

This controller manages the distributed workers.

### Start the service
- #### **Step 2: Launch the model worker(s)**

#### Serving with Web UI
Using IPEX-LLM in FastChat does not impose any new limitations on model usage. Therefore, all Hugging Face Transformer models can be utilized in FastChat.
```bash
source ipex-llm-init -t

To serve using the Web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the web server and model workers.
# Available low_bit format including sym_int4, sym_int8, bf16 etc.
python3 -m ipex_llm.serving.fastchat.ipex_llm_worker --model-path path/to/vicuna-7b-v1.5 --low-bit "sym_int4" --trust-remote-code --device "cpu" &
```
Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller.

##### Launch the Controller
```bash
python3 -m fastchat.serve.controller
```
- #### **Step 3: Launch Gradio web server or RESTful API server**
You can launch Gradio web server to serve your models using the web UI or launch RESTful API server to serve with HTTP.

This controller manages the distributed workers.
- **Option 1: Serving with Web UI**
```bash
python3 -m fastchat.serve.gradio_web_server &
```
This is the user interface that users will interact with.

##### Launch the model worker(s)
```bash
python3 -m ipex_llm.serving.model_worker --model-path lmsys/vicuna-7b-v1.3 --device cpu
```
Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller.
By following these steps, you will be able to serve your models using the web UI with `IPEX-LLM` as the backend. You can open your browser and chat with a model now.

> To run model worker using Intel GPU, simply change the --device cpu option to --device xpu
- **Option 2: Serving with OpenAI-Compatible RESTful APIs**

##### Launch the Gradio web server
Launch the RESTful API server

```bash
python3 -m fastchat.serve.gradio_web_server
```
```bash
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 &
```

This is the user interface that users will interact with.
Use curl for testing, an example could be:

By following these steps, you will be able to serve your models using the web UI with `IPEX-LLM` as the backend. You can open your browser and chat with a model now.
```bash
curl -X POST -H "Content-Type: application/json" -d '{
"model": "Llama-2-7b-chat-hf",
"prompt": "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun",
"n": 1,
"best_of": 1,
"use_beam_search": false,
"stream": false
}' http://YOUR_HTTP_HOST:8000/v1/completions
```
You can find more details here [Serving using IPEX-LLM and FastChat](https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/src/ipex_llm/serving/fastchat/README.md)

#### Serving with OpenAI-Compatible RESTful APIs
### 4. Serving with vLLM Continuous Batching
To fully utilize the continuous batching feature of the vLLM, you can send requests to the service using curl or other similar methods. The requests sent to the engine will be batched at token level. Queries will be executed in the same forward step of the LLM and be removed when they are finished instead of waiting for all sequences to be finished.

To start an OpenAI API server that provides compatible APIs using `IPEX-LLM` backend, you need three main components: an OpenAI API Server that serves the in-coming requests, model workers that host one or more models, and a controller to coordinate the web server and model workers.
- #### **Step 1: Launch the api_server**
```bash
#!/bin/bash
# You may also want to adjust the `--max-num-batched-tokens` argument, it indicates the hard limit
# of batched prompt length the server will accept
numactl -C 0-47 -m 0 python -m ipex_llm.vllm.entrypoints.openai.api_server \
--model /llm/models/Llama-2-7b-chat-hf/ \
--host 0.0.0.0 --port 8000 \
--load-format 'auto' --device cpu --dtype bfloat16 \
--max-num-batched-tokens 4096 &
```

First, launch the controller
- #### **Step 2: Use curl for testing, access the api server as follows:**

```bash
python3 -m fastchat.serve.controller
```
```bash
curl http://YOUR_HTTP_HOST:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/llm/models/Llama-2-7b-chat-hf/",
"prompt": "San Francisco is a",
"max_tokens": 128,
"temperature": 0
}' &
```

Then, launch the model worker(s):
You can find more details here: [Serving with vLLM Continuous Batching](https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/example/CPU/vLLM-Serving/README.md)

```bash
python3 -m ipex_llm.serving.model_worker --model-path lmsys/vicuna-7b-v1.3 --device cpu
```

Finally, launch the RESTful API server
## IPEX-LLM Serving on XPU

```bash
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```
FastChat is an open platform for training, serving, and evaluating large language model based chatbots. You can find the detailed information at their [homepage](https://github.com/lm-sys/FastChat).

IPEX-LLM is integrated into FastChat so that user can use IPEX-LLM as a serving backend in the deployment.

## IPEX-LLM Serving on XPU
### 1. Prepare ipex-llm-serving-xpu Docker Image

### Boot container
Run the following command:

Pull image:
```
```bash
docker pull intelanalytics/ipex-llm-serving-xpu:2.1.0-SNAPSHOT
```

### 2. Start ipex-llm-serving-xpu Docker Container

To map the `xpu` into the container, you need to specify `--device=/dev/dri` when booting the container.

An example could be:
```bash
#/bin/bash
export DOCKER_IMAGE=intelanalytics/ipex-llm-serving-cpu:2.1.0-SNAPSHOT
export DOCKER_IMAGE=intelanalytics/ipex-llm-serving-xpu:2.1.0-SNAPSHOT
export CONTAINER_NAME=my_container
export MODEL_PATH=/llm/models[change to your model path]
export SERVICE_MODEL_PATH=/llm/models/chatglm2-6b[a specified model path for running service]
docker run -itd \
--net=host \
--device=/dev/dri \
--memory="32G" \
--cpuset-cpus="0-47" \
--cpuset-mems="0" \
--name=$CONTAINER_NAME \
--shm-size="16g" \
-v $MODEL_PATH:/llm/models \
-e SERVICE_MODEL_PATH=$SERVICE_MODEL_PATH \
$DOCKER_IMAGE --service-model-path $SERVICE_MODEL_PATH
$DOCKER_IMAGE
```
Access the container:
```
docker exec -it $CONTAINER_NAME bash
```
You can assign specified model path to service-model-path to run the service while booting the container. Also you can manually run the service after entering container. Run `/opt/entrypoint.sh --help` in container to see more information. There are steps below describe how to run service in details as well.

To verify the device is successfully mapped into the container, run `sycl-ls` to check the result. In a machine with Arc A770, the sampled output is:
```bash
Expand All @@ -334,58 +371,88 @@ root@arda-arc12:/# sycl-ls
[opencl:gpu:2] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) A770 Graphics 3.0 [23.17.26241.33]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Arc(TM) A770 Graphics 1.3 [1.3.26241]
```
After the container is booted, you could get into the container through `docker exec`.

### Start the service

#### Serving with Web UI
### 3. Serving with FastChat

To serve using the Web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the web server and model workers.

##### Launch the Controller
```bash
python3 -m fastchat.serve.controller
```
- #### **Step 1: Launch the Controller**
```bash
python3 -m fastchat.serve.controller &
```

This controller manages the distributed workers.
This controller manages the distributed workers.

##### Launch the model worker(s)
```bash
python3 -m ipex_llm.serving.model_worker --model-path lmsys/vicuna-7b-v1.3 --device xpu
```
Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller.
- #### **Step 2: Launch the model worker(s)**

##### Launch the Gradio web server
Using IPEX-LLM in FastChat does not impose any new limitations on model usage. Therefore, all Hugging Face Transformer models can be utilized in FastChat.
```bash
# Available low_bit format including sym_int4, sym_int8, fp16 etc.
python3 -m ipex_llm.serving.fastchat.ipex_llm_worker --model-path /llm/models/Llama-2-7b-chat-hf/ --low-bit "sym_int4" --trust-remote-code --device "xpu" &
```
Wait until the process finishes loading the model and you see "Uvicorn running on ...". The model worker will register itself to the controller.

```bash
python3 -m fastchat.serve.gradio_web_server
```
- #### **Step 3: Launch Gradio web server or RESTful API server**
You can launch Gradio web server to serve your models using the web UI or launch RESTful API server to serve with HTTP.

This is the user interface that users will interact with.
- **Option 1: Serving with Web UI**
```bash
python3 -m fastchat.serve.gradio_web_server &
```
This is the user interface that users will interact with.

By following these steps, you will be able to serve your models using the web UI with `IPEX-LLM` as the backend. You can open your browser and chat with a model now.
By following these steps, you will be able to serve your models using the web UI with `IPEX-LLM` as the backend. You can open your browser and chat with a model now.

#### Serving with OpenAI-Compatible RESTful APIs
- **Option 2: Serving with OpenAI-Compatible RESTful APIs**

To start an OpenAI API server that provides compatible APIs using `IPEX-LLM` backend, you need three main components: an OpenAI API Server that serves the in-coming requests, model workers that host one or more models, and a controller to coordinate the web server and model workers.
Launch the RESTful API server

First, launch the controller
```bash
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 &
```

```bash
python3 -m fastchat.serve.controller
```
Use curl for testing, an example could be:

Then, launch the model worker(s):
```bash
curl -X POST -H "Content-Type: application/json" -d '{
"model": "Llama-2-7b-chat-hf",
"prompt": "Once upon a time, there existed a little girl who liked to have adventures. She wanted to go to places and meet new people, and have fun",
"n": 1,
"best_of": 1,
"use_beam_search": false,
"stream": false
}' http://YOUR_HTTP_HOST:8000/v1/completions
```
You can find more details here [Serving using IPEX-LLM and FastChat](https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/src/ipex_llm/serving/fastchat/README.md)

```bash
python3 -m ipex_llm.serving.model_worker --model-path lmsys/vicuna-7b-v1.3 --device xpu
```
### 4. Serving with vLLM Continuous Batching
To fully utilize the continuous batching feature of the vLLM, you can send requests to the service using curl or other similar methods. The requests sent to the engine will be batched at token level. Queries will be executed in the same forward step of the LLM and be removed when they are finished instead of waiting for all sequences to be finished.

Finally, launch the RESTful API server
- #### **Step 1: Launch the api_server**
```bash
#!/bin/bash
# You may also want to adjust the `--max-num-batched-tokens` argument, it indicates the hard limit
# of batched prompt length the server will accept
python -m ipex_llm.vllm.entrypoints.openai.api_server \
--model /llm/models/Llama-2-7b-chat-hf/ \
--host 0.0.0.0 --port 8000 \
--load-format 'auto' --device xpu --dtype bfloat16 \
--max-num-batched-tokens 4096 &
```

```bash
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```
- #### **Step 2: Use curl for testing, access the api server as follows:**

```bash
curl http://YOUR_HTTP_HOST:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/llm/models/Llama-2-7b-chat-hf/",
"prompt": "San Francisco is a",
"max_tokens": 128,
"temperature": 0
}' &
```
You can find more details here [Serving with vLLM Continuous Batching](https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/example/GPU/vLLM-Serving/README.md)

## IPEX-LLM Fine Tuning on CPU

Expand Down

0 comments on commit 20a5e72

Please sign in to comment.