How to deploy a RESTful API deepspeed MII on one node?

[shaoxuefeng]

Follow the README doc, I would like to deploy a RESTful API on one node,
But I got a ValueError: No slot '1' specified on host 'localhost' error:
the deploy python code :

import mii
from mii import DeploymentType

if __name__ == "__main__":
    HOST_FILE_PATH = "./hostfile"
    mii_configs = {
        "tensor_parallel": 8,
        "dtype": "fp16",
        "enable_restful_api": True,
        "restful_api_port": 8080,
        "skip_model_check": True,
        "enable_load_balancing": False,
        "replica_num": 1,
        "hostfile": HOST_FILE_PATH,
    }

    mii.deploy(task="text-generation",
               model="/workspace/workfile/Models/gptj-350m",
               deployment_name="codegen-350m",
               mii_config=mii_configs,
               deployment_type=DeploymentType.LOCAL)

And the hosfile:

localhost slots=8

According to the Deepspeed Issue, it seems we can't start with hosftile on only node. I even update deepspeed pkg to lastest master version, but it still not work.

deepspeed          0.8.3+unknown
deepspeed-mii      0.0.5+unknown

So, How can i start a a RESTful API deepspeed MII on one node?
Thank you!

[Wohoholo]

i have started with hostfile on only node(my machine has 2 gpu, but i only can deploy on one gpu).
configs:
tensor_parallel: 1
deploy_rank: 0
other params are the same as yours
my hostfile's content:
127.0.0.1 slots=2

and, by the way, u need to set your node passwordless login itself by ssh.
i want to know how to deploy on one node with multi gpu?

[Wohoholo]

i find some detail in source script.

1. config.py:

2. @root_validator
   

3. def auto_enable_load_balancing(cls, values):
   

4.     if values["enable_restful_api"] and not values["enable_load_balancing"]:
   

5.         logger.warn("Restful API is enabled, enabling Load Balancing")
   

6.         values["enable_load_balancing"] = True
   

7.     return values
   

it will make your "enable_load_balancing" become True.Then
server.py:

1. if mii_configs.enable_load_balancing:

3.         # Start replica instances
   

4.         for i, repl_config in enumerate(lb_config.replica_configs):
   

5.             hostfile = tempfile.NamedTemporaryFile(delete=False)
   

6.             hostfile.write(
   

7.                 f'{repl_config.hostname} slots={mii_configs.replica_num}\n'.encode())
   

8.             processes.append(
   

9.                 self._launch_deepspeed(
   

10.                     deployment_name,
    

11.                     model_name,
    

12.                     model_path,
    

13.                     ds_optimize,
    

14.                     ds_zero,
    

15.                     ds_config,
    

16.                     mii_configs,
    

17.                     hostfile.name,
    

18.                     repl_config.hostname,
    

19.                     repl_config.tensor_parallel_ports[0],
    

20.                     mii_configs.torch_dist_port + (100 * i) +
    

21.                     repl_config.gpu_indices[0],
    

22.                     repl_config.gpu_indices))
    

it will write a temp file with your "replica_num" but not your hostfile.
you can comment line 5, 6 and rewite line 17 to mii_configs.hostfile.
And "tensor_parallel" must be equal to length of parameter "deploy rank" in mii_configs.
Hope that will be helpful