The nvidia-device-plugin label GPU node automatically?

[pytimer]

Hi,

I deploy the nvidia-device-plugin on my kubernetes cluster. But on my cluster, different nodes have different GPU type, i hope my pod can deploy the specify GPU type node.

Now i must label GPU on the node manually, i hope nvidia-device-plugin can label GPU type automatically.

Is it reasonable?

[flx42]

There is a plan in the Kubernetes community: kubernetes/community#2265
The problem with a per-node label is that it wouldn't work for heterogeneous nodes (different GPU types on the same node). In addition, injecting those labels from the device plugin today would require RBAC and the deployment would be more complex.

[pytimer]

Thanks for your reply.

I know. But now if i want to implement my case, Should i must be label GPU type on the node?

[cliffburdick]

@flx42 we're interested in this too. Can the nvidia agent on each node report to the kubelet or API server how many GPUs and of what type there are? The node label seems restrictive since it's static and the amount of each isn't tracked.

[Bharathkumarraju]

I did this with some terraform varible with this simple command, I did node-labeling

sed -i '/\/usr\/bin\/kubelet/ a \ \ --node-labels nodetype=${var.instance-label} \\' /etc/systemd/system/kubelet.service

[pytimer]

@Bharathkumarraju which program add --node-labels value to kubelet? Your own program or when you deploy this node?

Now i write a daemonset to do it, when i deploy a GPU pod, i have to choose a node through my program, this program add nodeSelector to yaml.

I think it is not good, but i don't know other methods to do it now.

[RenaudWasTaken]

This is now possible with the GPU feature discovery: https://github.com/NVIDIA/gpu-feature-discovery