Skip to content

Commit

Permalink
modify README
Browse files Browse the repository at this point in the history
  • Loading branch information
Zhengjin-Wang committed Oct 9, 2023
1 parent 3814abf commit a1aefdb
Showing 1 changed file with 25 additions and 1 deletion.
26 changes: 25 additions & 1 deletion docker/llm/inference/cpu/docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,32 @@ To run inference using `BigDL-LLM` using cpu, you could refer to this [documenta

chat.py can be used to initiate a conversation with a specified model. The file is under directory '/root'.

To run chat.py:
You can download models and bind the model directory from host machine to container when start a container.

Here is an example:
```bash
export DOCKER_IMAGE=intelanalytics/bigdl-llm-cpu:2.4.0-SNAPSHOT
export MODEL_PATH=/home/llm/models

sudo docker run -itd \
--net=host \
--cpuset-cpus="0-47" \
--cpuset-mems="0" \
--memory="32G" \
--name=CONTAINER_NAME \
--shm-size="16g" \
-v $MODEL_PATH:/llm/models/
$DOCKER_IMAGE

```

After entering the container through `docker exec`, you can run chat.py by:
```bash
cd /root
python chat.py --model-path YOUR_MODEL_PATH
```
In the example above, it can be:
```bash
cd /root
python chat.py --model-path /llm/models/MODEL_NAME
```

0 comments on commit a1aefdb

Please sign in to comment.