GPU device plugin deployment issue (non default namespace)

[pawel-gacek]

Describe the bug
GPU device plugin will not work properly once NOT installed in default namespace. For the ClusterRoleBinding resource the ServiceAccount namespace is set to "default" once installed using kustomization tool regardless of namespace configured/used during GPU device plugin deployment:
https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/deployments/gpu_plugin/overlays/fractional_resources/gpu-manager-rolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: gpu-manager-rolebinding
subjects:

• kind: ServiceAccount
  name: gpu-manager-sa
  namespace: default ->>> here
  roleRef:
  kind: ClusterRole
  name: gpu-manager-role
  apiGroup: rbac.authorization.k8s.io

To Reproduce
Install GPU device plugin in non default namespace with kustomization.

Expected behavior
For ClusterRoleBinding resource (name gpu-manager-rolebinding) the ServiceAccount namespace is set to desired namespace.

System (please complete the following information):

• OS version: Ubuntu 22.04
• Kernel version: Linux 5.15
• Device plugins version: v0.30.0
• Hardware info:
  Xeon 8360Y,
  System Information
  Manufacturer: Intel Corporation
  Product Name: M50CYP2SBSTD
  Version: M50CYP2UR208

Thank you
Pawel

[tkatila]

Hi @pawel-gacek, yep, you are correct. This is a limitation of the deployment. We can't change the namespace name within the yaml file. The namespace is handled properly in our operator based deployment, though.

[pawel-gacek]

hi @tkatila got it thanks. Cause it may cause some issues in plugin operation as deployment itself works fine. Would be good if such limitation can be documented somewhere as I believe there are still kustomization based deployments in use. In our case we simply have not noticed that GPU plugin did not work properly until we have seen the GPU resource allocation failure for one of our workload.

[tkatila]

Sure. I'll add a note about it to the advanced deployments docs.

Off-topic: fractional resources is a sort of niche use case, how are you using it?

[pawel-gacek]

We do use GPU Aware Scheduler extender that requires fractional resources to be enabled with GPU dev plugin.