MD Express: Dependence on nvidia-runtime to Launch MAP Containers

[bluna301]

Hi all,

I have been investigating the ability for MAPs to be launched and perform inference with or without GPU support; while the ideal deployment solution is an inference server with GPU support, knowing if MAPs can be launched in a CPU-only environment is useful in determining all possible deployment options.

It has previously been confirmed by the MONAI team that the MONAI Application Runner (MAR) was designed with the assumption of an available GPU and installation of NVIDIA Container Toolkit (MAR checks for this installation); thus, the MAR defaults to launch MAP containers using GPU and the nvidia-runtime in Docker.

The MONAI team provided a custom script to launch MAP containers directly with Docker. Within this script, the RUN_TIME environment variable represents the Docker runtime, whose value can be edited to have the MAP container perform inference with or without GPU support:

• --gpus=all or --runtime=nvidia to run MAP with GPU support
• "" or --runtime=runc to run MAP without GPU support (CPU only)

After confirming MAPs can be launched with and without GPU support, I was wondering if MDE would be able to support launching MAPs in a GPU agnostic environment. To test this, I removed the nvidia-runtime from my docker/daemon.json configuration file to see if MAPs can be launched in MDE with the default Docker runtime. Subsequently initiating a workflow in MDE produced the following error from the Task Manager, stating that Docker couldn’t find the nvidia-runtime:

2024-07-24 09:48:26 2024-07-24 13:48:26.7022|1002|ERROR|Monai.Deploy.WorkflowManager.TaskManager.Docker.DockerPlugin|EventId=1002,
EventName=ErrorDeployingContainer, @messageId=1795a7f8-0d20-4b82-8c94-e46e618a035a,
@applicationId=16988a78-87b5-4168-a5c3-2cfc2bab8e54, @correlationId=0808fb16-f911-4843-a540-e31ccae5c555,
@receivedTime=07/24/2024 13:48:24, correlationId=0808fb16-f911-4843-a540-e31ccae5c555,
messageId=1795a7f8-0d20-4b82-8c94-e46e618a035a, messageType=TaskDispatchEvent,
workflowInstanceId=7c072a1a-d5c6-492f-8c47-ee57636f9d42, taskId=liver,
executionId=2c2ae671-c867-45e6-921e-b1f9d6522af9|Error deploying Container.
Docker.DotNet.DockerApiException: Docker API responded with status code=BadRequest,
response={"message":"unknown or invalid runtime name: nvidia"}

Thus, like the MAR, MDE (specifically the Task Manager) seems to be reliant on NVIDIA Container Toolkit/nvidia-runtime. Is it possible to remove/modify this dependency within the Task Manager so that MAPs can be launched with or without the nvidia-runtime; i.e. with or without GPU?

Adding @mocsharp @MMelQin @vikashg for visibility.