Break down Pending vs Terminating status

[rgaiacs]

When I look at Graphana, I see

I checked

sum(label_replace(kube_pod_status_phase{phase="Pending",pod=~"jupyter-.*"}, "repo", "$1", "pod", "jupyter-(.+)-[^-]+")) by (repo)

on Prometheus and I got

Based on the information that Prometheus provided, the information on Graphana is wrong because Graphana mentions 25 pending pods but Prometheus only reports 6.

I looked at Kubernetes and the number of pending pods is only 6 as reported by Prometheus.

But we have 19 "Terminating" pods:

My understanding is that Graphana is merging "Pending" and "Terminating". I looked at the expression used by Graphana

sum(kube_pod_status_phase{pod=~\"^jupyter-.*\", kubernetes_namespace!=\"jhub-ns\"}) by (phase)

This expression looks good to me. This means that the problem is at the metric exporting part.

In kubernetes/kube-state-metrics#1013, someone said

I have a pod in status Terminating but whith kube-state-metrics:v2.7.0 can not see kube_pod_status_phase{phase="Terminating"}

@sgibson91 and @manics can you help me to have the pod "Terminating" state exported? Thanks!

[manics]

https://github.com/kubernetes/kube-state-metrics/blob/b1c2e0c1cf897202fa10da7b622e883df8a7a66e/docs/metrics/workload/pod-metrics.md#useful-metrics-queries
suggests
count(kube_pod_deletion_timestamp) by (namespace, pod) * count(kube_pod_status_reason{reason="NodeLost"} == 0) by (namespace, pod)