DCGM will not run on GPU nodes with Bottlerocket OS

[NicolasGrosjeanProbayes]

Hello,

I use AWS Bottlerocket AMI 1.28.0-0ab4fab4 inside an EKS cluster and the dcgm-exporter pod fails with the following logs

Warning #2: dcgm-exporter doesn't have sufficient privileges to expose profiling metrics. To get profiling metrics with dcgm-exporter, use --cap-add SYS_ADMIN
time="2024-12-18T08:59:42Z" level=info msg="Starting dcgm-exporter"
Error: Failed to initialize NVML
time="2024-12-18T08:59:42Z" level=fatal msg="Error starting nv-hostengine: DCGM initialization error"

I have found #3992 but I don't understand how to apply the proposed solutions.

[arnaldo2792]

Hello @NicolasGrosjeanProbayes, let me provide some context. In the NVIDIA containers' stack, a container could get access to all the GPUs by just having the NVIDIA_VISIBLE_DEVICES environment variable set to all. This is a problem, because you as the administrator won't want to let containers bypass the restrictions that are set through Kubernetes directives (a.k.a nvidia.com/gpu: 1).

In Bottlerocket, we don't allow unprivileged containers (usually, those actually using the GPUs) to leverage on NVIDIA_VISIBLE_DEVICES=all. However, there are "privileged" containers (e.g. dcgm-exporter) that must have access to all GPUs without you the administrator granting individual access to every single one of them. For said containers, NVIDIA's documentation suggests to add the CAP_SYS_ADMIN capability to make them explicitly privileged so that they can have access to all the GPUs by setting NVIDIA_VISIBLE_DEVICES=all. Most of NVIDIA's container images (such as the dcgm-exporter) already ship with the NVIDIA_VISIBLE_DEVICES environment variable set to all, so the only configuration you should apply is adding the extra CAP_SYS_ADMIN capability as NVIDIA suggests:

└─> ❯ docker inspect nvidia/dcgm-exporter | jq '.[].Config.Env' | grep VISIBLE
  "NVIDIA_VISIBLE_DEVICES=all",