SD kube-controller-manager + kubescheduler not being discovered

[KlavsKlavsen]

Prometheus servicediscovery says:
monitoring/kube-prometheus-stack-kube-controller-manager/0 (0 / 37 active targets)
monitoring/kube-prometheus-stack-kube-scheduler/0 (0 / 37 active targets)

all other targets are discovered just fine.

What happened?
I installed this using community helm chart v13.13.0 - and all other SD scrape jobs work fine.

Did you expect to see something different?
I did expect it to discover these two services.

How to reproduce it (as minimally and precisely as possible):
Setup is kops 1.18.3 installation of k8s 1.18.15

Environment

• Prometheus Operator version:
  quay.io/prometheus-operator/prometheus-operator:v0.45.0

• Kubernetes version information:
  1.18.15

• Kubernetes cluster kind:

kops 1.18.3

Used community helm chart: kube-prometheus-stack

• Prometheus Operator Logs:

nothing relevant. job is setup - it just does not match correctly for these two services it seems.

 apiVersion: v1                                                                                                                                                                                                                                        
 kind: Service                                                                                                                                                                                                                                         
 metadata:                                                                                                                                                                                                                                             
   annotations:                                                                                                                                                                                                                                        
     kubectl.kubernetes.io/last-applied-configuration: |                                                                                                                                                                                               
       {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"kube-prometheus-stack-kube-controller-manager","argocd.argoproj.io/instance":"kube-prometheus-stack","chart":"kube-prometheus-stack-13.13.0","heritage":"Helm 
 ","jobLabel":"kube-controller-manager","release":"kube-prometheus-stack"},"name":"kube-prometheus-stack-kube-controller-manager","namespace":"kube-system"},"spec":{"clusterIP":"None","ports":[{"name":"http-metrics","port":10252,"protocol":"TCP", 
 "targetPort":10252}],"selector":{"component":"kube-controller-manager"},"type":"ClusterIP"}}                                                                                                                                                          
   creationTimestamp: "2021-02-26T10:24:28Z"                                                                                                                                                                                                           
   labels:                                                                                                                                                                                                                                             
     app: kube-prometheus-stack-kube-controller-manager                                                                                                                                                                                                
     argocd.argoproj.io/instance: kube-prometheus-stack                                                                                                                                                                                                
     chart: kube-prometheus-stack-13.13.0                                                                                                                                                                                                              
     heritage: Helm                                                                                                                                                                                                                                    
     jobLabel: kube-controller-manager                                                                                                                                                                                                                 
     release: kube-prometheus-stack                                                                                                                                                                                                                    
   name: kube-prometheus-stack-kube-controller-manager                                                                                                                                                                                                 
   namespace: kube-system                                                                                                                                                                                                                              
   resourceVersion: "1106635"                                                                                                                                                                                                                          
   selfLink: /api/v1/namespaces/kube-system/services/kube-prometheus-stack-kube-controller-manager                                                                                                                                                     
   uid: 01b648aa-9695-45e1-9a63-a4006944a02d                                                                                                                                                                                                           
 spec:                                                                                                                                                                                                                                                 
   clusterIP: None                                                                                                                                                                                                                                     
   ports:                                                                                                                                                                                                                                              
   - name: http-metrics                                                                                                                                                                                                                                
     port: 10252                                                                                                                                                                                                                                       
     protocol: TCP                                                                                                                                                                                                                                     
     targetPort: 10252                                                                                                                                                                                                                                 
   selector:                                                                                                                                                                                                                                           
     component: kube-controller-manager                                                                                                                                                                                                                
   sessionAffinity: None                                                                                                                                                                                                                               
   type: ClusterIP                                                                                                                                                                                                                                     
 status:                                                                                                                                                                                                                                               
   loadBalancer: {} 

and kube-scheduler service:

 apiVersion: v1                                                                                                                                                                                                                                        
 kind: Service                                                                                                                                                                                                                                         
 metadata:                                                                                                                                                                                                                                             
   annotations:                                                                                                                                                                                                                                        
     kubectl.kubernetes.io/last-applied-configuration: |                                                                                                                                                                                               
       {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"kube-prometheus-stack-kube-scheduler","argocd.argoproj.io/instance":"kube-prometheus-stack","chart":"kube-prometheus-stack-13.13.0","heritage":"Helm","jobLab 
 el":"kube-scheduler","release":"kube-prometheus-stack"},"name":"kube-prometheus-stack-kube-scheduler","namespace":"kube-system"},"spec":{"clusterIP":"None","ports":[{"name":"http-metrics","port":10251,"protocol":"TCP","targetPort":10251}],"selec 
 tor":{"component":"kube-scheduler"},"type":"ClusterIP"}}                                                                                                                                                                                              
   creationTimestamp: "2021-02-26T10:24:28Z"                                                                                                                                                                                                           
   labels:                                                                                                                                                                                                                                             
     app: kube-prometheus-stack-kube-scheduler                                                                                                                                                                                                         
     argocd.argoproj.io/instance: kube-prometheus-stack                                                                                                                                                                                                
     chart: kube-prometheus-stack-13.13.0                                                                                                                                                                                                              
     heritage: Helm                                                                                                                                                                                                                                    
     jobLabel: kube-scheduler                                                                                                                                                                                                                          
     release: kube-prometheus-stack                                                                                                                                                                                                                    
   name: kube-prometheus-stack-kube-scheduler                                                                                                                                                                                                          
   namespace: kube-system                                                                                                                                                                                                                              
   resourceVersion: "1106659"                                                                                                                                                                                                                          
   selfLink: /api/v1/namespaces/kube-system/services/kube-prometheus-stack-kube-scheduler                                                                                                                                                              
   uid: 7dd3e985-d407-4e1f-8e1b-628cadababf4                                                                                                                                                                                                           
 spec:                                                                                                                                                                                                                                                 
   clusterIP: None                                                                                                                                                                                                                                     
   ports:                                                                                                                                                                                                                                              
   - name: http-metrics                                                                                                                                                                                                                                
     port: 10251                                                                                                                                                                                                                                       
     protocol: TCP                                                                                                                                                                                                                                     
     targetPort: 10251                                                                                                                                                                                                                                 
   selector:                                                                                                                                                                                                                                           
     component: kube-scheduler                                                                                                                                                                                                                         
   sessionAffinity: None                                                                                                                                                                                                                               
   type: ClusterIP                                                                                                                                                                                                                                     
 status:                                                                                                                                                                                                                                               
   loadBalancer: {} 

[paulfantom]

Possibly related to #718

[KlavsKlavsen]

After some tips from slack.. I was pointed to #718 - but as a commenter said - comparing labels and port names for Service and ServiceMonitor (both created by community helm chart) - they seem correct.
So the issue seems to be with the generated prometheus configuration for these two services ?

- job_name: monitoring/kube-prometheus-stack-kube-scheduler/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: kube-prometheus-stack-kube-scheduler
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: kube-prometheus-stack
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
  - source_labels:
    - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep

kube-controller-manager sd job:

- job_name: monitoring/kube-prometheus-stack-kube-controller-manager/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - kube-system
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    regex: kube-prometheus-stack-kube-controller-manager
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_release
    regex: kube-prometheus-stack
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: http-metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - source_labels:
    - __meta_kubernetes_service_label_jobLabel
    target_label: job
    regex: (.+)
    replacement: ${1}
  - target_label: endpoint
    replacement: http-metrics
  - source_labels:
    - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep

[KlavsKlavsen]

It turned out to be the Service (created by helm chart) not matching labels in pod (from kops 1.18 deployment). Closing and opening on helm chart :)