GPU plugin can be configured to register a monitoring resource for the nodes that have Intel GPUs on them. gpu.intel.com/i915_monitoring (or gpu.intel.com/xe_monitoring) is a singular resource on the nodes. A container requesting it, will get access to all the Intel GPUs (i915 or xe KMD device files) on the node. The idea behind this resource is to allow the container to monitor the GPUs. A container requesting the i915_monitoring resource would typically export data to some metrics consumer. An example for such a consumer is Prometheus.
For the monitoring applications, there are two possibilities: Intel XPU Manager and collectd. Intel XPU Manager is readily available as a container and with a deployment yaml. collectd has Intel GPU support in its 6.0 branch, but there are no public containers available for it.
To deploy XPU Manager to a cluster, one has to run the following kubectl:
$ kubectl apply -k https://github.com/intel/xpumanager/deployment/kubernetes/daemonset/base
This will deploy an XPU Manager daemonset to run on all the nodes having the i915_monitoring resource.
For deploying Prometheus to a cluster, see this page. One can also use Prometheus' helm chart.
Prometheus requires additional Kubernetes configuration so it can fetch GPU metrics. The following steps will add a Kubernetes Service and a ServiceMonitor components. The components instruct Prometheus how and where from to retrieve the metrics.
$ kubectl apply -f https://raw.githubusercontent.com/intel/xpumanager/master/deployment/kubernetes/monitoring/service-intel-xpum.yaml
$ kubectl apply -f https://raw.githubusercontent.com/intel/xpumanager/master/deployment/kubernetes/monitoring/servicemonitor-intel-xpum.yaml
With those components in place, one can query Intel GPU metrics from Prometheus with xpum_ prefix.