Unable to obtain kube-controller-manager and kube-scheduler monitoring information #913
Comments
|
I had same problem at following env:
I haven't resolve the problem but at least service label seems problem. exporter service require "k8s-app=kube-scheduler" label. But scheduler doesn't have label. After change selector k8s-app to component as above result, I service worked fine. But metrics was not collected by prometheus. servicemonitor is following. Install InformationI install helm: custom-values.yml |
|
I found the cause. I found following error on Prometheus targets panel: Prometheus tried to got metrics from http://172.31.24.19:10251/metrics. |
|
I finally found the resolution. Use 0.0.0.0 for scheduler and contoller-manager bind address. Then scheduler and controller manager ports are available. |
|
@xiaomuyi does that solve your issue? |
Using the latest version has been resolved |
|
https://coreos.com/operators/prometheus/docs/latest/user-guides/cluster-monitoring.html |
|
I have the same issue, altho my kubespray scheduler is only running on the https port. I'll need to dig into the helm chart at this point. Edit ... |
|
The issue is there still/again. Problems:
Is there a recommended approach to this issue? I see two approaches:
|
|
@okamototk @ksa-real I updated the kubeadm-config comfig map as below but still facing the same issue. DataClusterConfiguration:apiServer: ClusterStatus:apiEndpoints: Events: |
|
@syednadeembe you're using the apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
...
controllerManager:
extraArgs:
bind-address: 0.0.0.0
scheduler:
extraArgs:
bind-address: 0.0.0.0In any case, to verify that the setting has been applied, you can do |
|
the pod has the required flag and i can verify that while doing the pod describe: spec:
####################################################################### |
|
what is the recommended way to update kubeadm config file for a running cluster ? |
|
Resolved The default values.yml file which comes with prometheus helm release only scraps at http unless specified elsewise. helm fetch prometheus-community/prometheus-operator ### to get the default values that had been used kubeControllerManager: service: // Ran the helm upgrade |
|
Since setting This could be implemented with an HAProxy container run on each master node with a DaemonSet with the following configuration: defaults
mode http
timeout connect 5000ms
timeout client 5000ms
timeout server 5000ms
default-server maxconn 10
frontend kube-controller-manager
bind ${NODE_IP}:10257
http-request deny if !{ path /metrics }
default_backend kube-controller-manager
backend kube-controller-manager
server kube-controller-manager 127.0.0.1:10257 ssl verify none
frontend kube-scheduler
bind ${NODE_IP}:10259
http-request deny if !{ path /metrics }
default_backend kube-scheduler
backend kube-scheduler
server kube-scheduler 127.0.0.1:10259 ssl verify noneThis would listen on port 10257 on the node's IP address for requests to the The env:
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIPThe proxy Pod must run in the |
|
@weibeld should not you need to pass bearer token with cluster role binding on scrapping metrics endpoint. ? |
@weibeld Sounds like a fantastic solution - albeit a big more complex ofcourse :) As I understand it - we need a daemonset running haproxy - with the config you mentioned above - and a modified service definition from Prometheus - to make it try to reach the nodes kubecontrolermanager and kube-scheduler pods - via node-ip instead |
|
@KlavsKlavsen I used Kustomize, since all the other services in the cluster were deployed with Kustomize. But it really doesn't matter how you deploy the DaemonSet, best if you just deploy it like any other application in your cluster, e.g. Helm if you usually use Helm. Once you have the DaemonSet, you can scrape the exposed ports as a normal Prometheus target, i.e. either with a Service and a ServiceMonitor, or with a PodMonitor. Here's the complete configuration I used: apiVersion: v1
kind: ConfigMap
metadata:
name: metrics-proxy-master
data:
haproxy.cfg: |
defaults
mode http
timeout connect 5000ms
timeout client 5000ms
timeout server 5000ms
default-server maxconn 10
frontend kube-controller-manager
bind ${NODE_IP}:10257
http-request deny if !{ path /metrics }
default_backend kube-controller-manager
backend kube-controller-manager
server kube-controller-manager 127.0.0.1:10257 ssl verify none
frontend kube-scheduler
bind ${NODE_IP}:10259
http-request deny if !{ path /metrics }
default_backend kube-scheduler
backend kube-scheduler
server kube-scheduler 127.0.0.1:10259 ssl verify none
frontend etcd
bind ${NODE_IP}:2381
http-request deny if !{ path /metrics }
default_backend etcd
backend etcd
server etcd 127.0.0.1:2381
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: metrics-proxy-master
spec:
selector:
matchLabels:
app: metrics-proxy-master
template:
metadata:
labels:
app: metrics-proxy-master
spec:
hostNetwork: true
serviceAccountName: metrics-proxy
nodeSelector:
node-role.kubernetes.io/master: ""
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
containers:
- name: haproxy
image: haproxy:2.2.8
env:
- name: NODE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
ports:
- name: kube-ctrl-mgr # Port names are limited to 15 characters
containerPort: 10257
- name: kube-scheduler
containerPort: 10259
- name: etcd
containerPort: 2381
volumeMounts:
- mountPath: /usr/local/etc/haproxy
name: haproxy-config
volumes:
- configMap:
name: metrics-proxy-master
name: haproxy-configThen you can scrape the metrics of, for example, kube-controller-manager with a PodMonitor: apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: kube-controller-manager
spec:
selector:
matchLabels:
app: metrics-proxy-master
namespaceSelector:
matchNames:
- monitoring
podMetricsEndpoints:
- port: kube-ctrl-mgr
bearerTokenSecret:
name: metrics-endpoint-reader
key: token
relabelings:
- targetLabel: node
sourceLabels: [__meta_kubernetes_pod_node_name]
- targetLabel: job
replacement: kube-controller-managerNote that the metrics endpoints of kube-controller-manager and kube-scheduler require RBAC authorisation through a ServiceAccount token with at least read permissions for the |
Yes, this is required for kube-controller-manager and kube-scheduler. You can create a Secret with a ServiceAccount token with the appropriate permissions like this: apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-endpoint-reader
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metrics-endpoint-reader
rules:
- nonResourceURLs: [/metrics]
verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metrics-endpoint-reader
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metrics-endpoint-reader
subjects:
- kind: ServiceAccount
name: metrics-endpoint-reader
namespace: monitoring
---
apiVersion: v1
kind: Secret
metadata:
name: metrics-endpoint-reader
annotations:
kubernetes.io/service-account.name: metrics-endpoint-reader
type: kubernetes.io/service-account-tokenThen you can reference the created Secret by its name (e.g. |
|
@weibeld what are the roles/RBACs for the metrics-proxy service-account? thanks for putting this awesome solution together. |
What did you do?
I use helm to deploy prometheus-operator to monitor binary kubernetes clusters. However, it is not possible to obtain kube-controller-manager and kube-scheduler component data at present.
What did you expect to see?
I want to know how prometheus-operator monitors kubernetes Masters.
What did you see instead? Under which circumstances?
Look at the EP created by prometheus-operator and find that the ENDPOINTS field is .
Service port listening 0.0.0.0.
Environment
Kubernetes cluster kind:
Binary manual installation
Manifests:
Prometheus Operator Logs:
The text was updated successfully, but these errors were encountered: