Is multiple gpu supported with non-persistent pipeline

[yaliqin]

Hi team,

From the code I can see the multiple gpu deployment is supported for persistent deployment with server method. Is it supported with pipeline method?

Thanks

[mrwyattii]

Yes we support multi-GPU support for the mii.pipeline: https://github.com/microsoft/DeepSpeed-MII#tensor-parallelism

You will need to launch script using the deepspeed launcher to enable model parallelism with the pipeline:
deepspeed --num_gpus 2 pipeline.py

Where pipeline.py could be:

import mii
pipe = mii.pipeline("mistralai/Mistral-7B-v0.1")
response = pipe("DeepSpeed is", max_new_tokens=128)
print(response)