Fix how kubectl top pod is called

[sed-i]

cmd = "/snap/microk8s/current/kubectl --kubeconfig /var/snap/microk8s/current/credentials/client.config top pod -n ${JUJU_MODEL_NAME} --no-headers".split()

This is way too brittle. Calling various microk8s commands differs between strictly confined and not. The amount of scraping needed to actually pull this in "properly" is enormous, and it's honestly a PITA to get any of these out without something running inside the cluster, but there is a middle ground between "literally call kubectl top pod" and run in k8s.

That middle ground is:
kubectl get --raw /apis/metrics.k8s.io/v1beta1 | jq

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nodes",
      "singularName": "",
      "namespaced": false,
      "kind": "NodeMetrics",
      "verbs": [
        "get",
        "list"
      ]
    },
    {
      "name": "pods",
      "singularName": "",
      "namespaced": true,
      "kind": "PodMetrics",
      "verbs": [
        "get",
        "list"
      ]
    }
  ]
}

You can drill into nodes or pods.
For kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq, you'll get a list of all of the objects. Including metadata, like the namespace, which can be used to filter.

    {                                                                                                                                                                                                                                               
      "metadata": {                                                                                                                                                                                                                                 
        "name": "coredns-597584b69b-tddzt",                                                                                                                                                                                                         
        "namespace": "kube-system",                                                                                                                                                                                                                 
        "creationTimestamp": "2023-02-26T02:08:55Z",                                                                                                                                                                                                
        "labels": {                                                                                                                                                                                                                                 
          "k8s-app": "kube-dns",                                                                                                                                                                                                                    
          "pod-template-hash": "597584b69b"                                                                                                                                                                                                         
        }                                                                                                                                                                                                                                           
      },                                                                                                                                                                                                                                            
      "timestamp": "2023-02-26T02:08:41Z",                                                                                                                                                                                                          
      "window": "17.843s",                                                                                                                                                                                                                          
      "containers": [                                                                                                                                                                                                                               
        {                                                                                                                                                                                                                                           
          "name": "coredns",                                                                                                                                                                                                                        
          "usage": {                                                                                                                                                                                                                                
            "cpu": "3491677n",                                                                                                                                                                                                                      
            "memory": "24656Ki"                                                                                                                                                                                                                     
          }                                                                                                                                                                                                                                         
        }                                                                                                                                                                                                                                           
      ]                                                                                                                                                                                                                                             
    }

All of the certs are in /snap/microk8s/current/certs-beta/. You can pretty much yank all of these in pure Python (no subprocess) with something like:

curl $KUBE_API/apis/apps/v1/deployments \
  --cacert /snap/microk8s/current/certs-beta/ca.crt \
  --cert /snap/microk8s/current/certs-beta/client.crt \
  --key /snap/microk8s/current/certs-beta/client.key

Yank the certs into the exporter on startup, and use requests or urllib or whatever you want to read it all and transform

Originally posted by @rbarry82 in #65 (comment)