Incompatible with some kubelet args #402
Labels
kind/support
Categorizes issue or PR as a support question.
Comments
|
Metrics server needs a minute to scrape all nodes in clusters for metrics. Is the problem still persisting? Could you provide more metrics server logs? |
# kubectl top no
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master1 598m 3% 17935Mi 55%
master2 398m 2% 24380Mi 76%
master3 408m 2% 7793Mi 23%
node-1 281m 1% 6966Mi 21%
node-2 89m 0% 8755Mi 54%
node-3 425m 2% 23215Mi 71%
node-4 165m 1% 24233Mi 75% # kubectl get po -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
adminserver-rc-8894899f4-mb5qd 1/1 Running 3 31d 10.244.137.86 master1 <none> <none>
calico-kube-controllers-58596dfcf7-9g2ms 1/1 Running 1 12d 192.168.40.161 master2 <none> <none>
calico-node-86mnf 2/2 Running 21 41d 192.168.40.224 node-4 <none> <none>
calico-node-d4xpn 2/2 Running 0 15h 192.168.40.162 master3 <none> <none>
calico-node-g67fm 2/2 Running 12 41d 192.168.40.160 master1 <none> <none>
calico-node-knm82 2/2 Running 12 15d 192.168.40.210 node-1 <none> <none>
calico-node-kqbl5 2/2 Running 21 41d 192.168.40.234 node-3 <none> <none>
calico-node-lxkt5 2/2 Running 0 7d17h 192.168.40.235 node-2 <none> <none>
calico-node-xfpqj 2/2 Running 9 41d 192.168.40.161 master2 <none> <none>
cjbdemodeployment-76557b5dd7-g69k2 1/1 Running 2 30d 10.244.137.69 master1 <none> <none>
cloudapiserver-7595458584-wdqmx 1/1 Running 0 6d20h 10.244.247.25 node-2 <none> <none>
coredns-bcwxj 1/1 Running 0 15h 10.244.136.1 master3 <none> <none>
coredns-shkm8 1/1 Running 7 49d 10.244.137.79 master1 <none> <none>
coredns-xnngb 1/1 Running 8 58d 10.244.180.34 master2 <none> <none>
harbor-ui-rc-77f696b975-gbqjt 1/1 Running 5 7d19h 10.244.136.21 master3 <none> <none>
kube-apiserver-master1 1/1 Running 342 153d 192.168.40.160 master1 <none> <none>
kube-apiserver-master2 1/1 Running 80 153d 192.168.40.161 master2 <none> <none>
kube-apiserver-master3 1/1 Running 8 153d 192.168.40.162 master3 <none> <none>
kube-controller-manager-master1 1/1 Running 14 153d 192.168.40.160 master1 <none> <none>
kube-controller-manager-master2 1/1 Running 10 153d 192.168.40.161 master2 <none> <none>
kube-controller-manager-master3 1/1 Running 7 153d 192.168.40.162 master3 <none> <none>
kube-proxy-4gbs5 1/1 Running 1 41d 192.168.40.234 node-3 <none> <none>
kube-proxy-7t5d8 1/1 Running 6 15d 192.168.40.210 node-1 <none> <none>
kube-proxy-d42f4 1/1 Running 9 41d 192.168.40.160 master1 <none> <none>
kube-proxy-k28rd 1/1 Running 0 15h 192.168.40.162 master3 <none> <none>
kube-proxy-ml82s 1/1 Running 0 7d17h 192.168.40.235 node-2 <none> <none>
kube-proxy-vbbt5 1/1 Running 9 41d 192.168.40.224 node-4 <none> <none>
kube-proxy-vssbv 1/1 Running 4 41d 192.168.40.161 master2 <none> <none>
kube-scheduler-master1 1/1 Running 3 29d 192.168.40.160 master1 <none> <none>
kube-scheduler-master2 1/1 Running 4 29d 192.168.40.161 master2 <none> <none>
kube-scheduler-master3 1/1 Running 8 29d 192.168.40.162 master3 <none> <none>
metrics-server-fbcd89d77-f2p96 1/1 Running 0 15h 10.244.139.175 node-3 <none> <none>
mysql-rc-797bb5c96-k4m6x 1/1 Running 12 15d 10.244.137.97 master1 <none> <none>
postgresql-f6f94755d-zxmwf 1/1 Running 0 7d15h 10.244.247.39 node-2 <none> <none>
redis-rc-5cdd78555-mrhlc 1/1 Running 0 22h 10.244.217.127 node-4 <none> <none>
registry-rc-5677c8695d-qjmv8 1/1 Running 2 7d19h 10.244.136.2 master3 <none> <none>
virtlet-78x78 2/3 Running 6 3d15h 192.168.40.162 master3 <none> <none>
virtlet-nl6qb 3/3 Running 3 3d15h 192.168.40.210 node-1 <none> <none>
# kubectl get po -n kube-system -o wide | wc -l
39
# kubectl top po -n kube-system
NAME CPU(cores) MEMORY(bytes)
adminserver-rc-8894899f4-mb5qd 0m 59Mi
calico-kube-controllers-58596dfcf7-9g2ms 1m 26Mi
calico-node-86mnf 23m 175Mi
calico-node-g67fm 32m 124Mi
calico-node-lxkt5 19m 232Mi
calico-node-xfpqj 24m 130Mi
cjbdemodeployment-76557b5dd7-g69k2 1m 369Mi
cloudapiserver-7595458584-wdqmx 1m 1299Mi
coredns-shkm8 5m 24Mi
coredns-xnngb 4m 21Mi
kube-apiserver-master1 40m 561Mi
kube-apiserver-master2 30m 544Mi
kube-controller-manager-master1 35m 95Mi
kube-controller-manager-master2 1m 22Mi
kube-proxy-d42f4 34m 47Mi
kube-proxy-ml82s 57m 142Mi
kube-proxy-vbbt5 1m 67Mi
kube-proxy-vssbv 70m 48Mi
kube-scheduler-master1 2m 28Mi
kube-scheduler-master2 2m 28Mi
mysql-rc-797bb5c96-k4m6x 4m 1694Mi
postgresql-f6f94755d-zxmwf 1m 205Mi
redis-rc-5cdd78555-mrhlc 2m 17Mi
# kubectl top po -n kube-system | wc -l
24
metrics-server logs I0103 08:40:27.255356 1 manager.go:95] Scraping metrics from 7 sources
I0103 08:40:27.263570 1 manager.go:120] Querying source: kubelet_summary:master3
I0103 08:40:27.279592 1 manager.go:120] Querying source: kubelet_summary:master1
I0103 08:40:27.281588 1 manager.go:120] Querying source: kubelet_summary:master2
I0103 08:40:27.294574 1 manager.go:120] Querying source: kubelet_summary:node-3
I0103 08:40:27.297590 1 manager.go:120] Querying source: kubelet_summary:node-2
I0103 08:40:27.301574 1 manager.go:120] Querying source: kubelet_summary:node-4
I0103 08:40:27.310601 1 manager.go:120] Querying source: kubelet_summary:node-1
I0103 08:40:27.358839 1 manager.go:148] ScrapeMetrics: time: 103.41204ms, nodes: 7, pods: 45
E0103 08:40:40.521920 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-fbcd89d77-f2p96: no metrics known for pod
E0103 08:40:40.521965 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-78x78: no metrics known for pod
E0103 08:40:40.521993 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/registry-rc-5677c8695d-qjmv8: no metrics known for pod
E0103 08:40:40.522017 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-kqbl5: no metrics known for pod
E0103 08:40:40.522043 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-knm82: no metrics known for pod
E0103 08:40:40.522065 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-master3: no metrics known for pod
E0103 08:40:40.522089 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-nl6qb: no metrics known for pod
E0103 08:40:40.522109 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/harbor-ui-rc-77f696b975-gbqjt: no metrics known for pod
E0103 08:40:40.522131 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-master3: no metrics known for pod
E0103 08:40:40.522153 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-4gbs5: no metrics known for pod
E0103 08:40:40.522174 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod
E0103 08:40:40.522199 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-7t5d8: no metrics known for pod
E0103 08:40:40.522224 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-bcwxj: no metrics known for pod
E0103 08:40:40.522244 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-master3: no metrics known for pod
E0103 08:40:40.522271 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-k28rd: no metrics known for pod
E0103 08:40:44.606154 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/harbor-ui-rc-77f696b975-gbqjt: no metrics known for pod
E0103 08:40:44.606199 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-master3: no metrics known for pod
E0103 08:40:44.606224 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-4gbs5: no metrics known for pod
E0103 08:40:44.606244 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod
E0103 08:40:44.606271 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-7t5d8: no metrics known for pod
E0103 08:40:44.606299 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-bcwxj: no metrics known for pod
E0103 08:40:44.606321 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-master3: no metrics known for pod
E0103 08:40:44.606347 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-k28rd: no metrics known for pod
E0103 08:40:44.606380 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-fbcd89d77-f2p96: no metrics known for pod
E0103 08:40:44.606401 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-78x78: no metrics known for pod
E0103 08:40:44.606426 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/registry-rc-5677c8695d-qjmv8: no metrics known for pod
E0103 08:40:44.606456 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-kqbl5: no metrics known for pod
E0103 08:40:44.606478 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-knm82: no metrics known for pod
E0103 08:40:44.606499 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-master3: no metrics known for pod
E0103 08:40:44.606520 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-nl6qb: no metrics known for pod
E0103 08:41:15.184900 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-kqbl5: no metrics known for pod
E0103 08:41:15.184948 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-nl6qb: no metrics known for pod
E0103 08:41:15.184973 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-knm82: no metrics known for pod
E0103 08:41:15.184996 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-master3: no metrics known for pod
E0103 08:41:15.185024 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-master3: no metrics known for pod
E0103 08:41:15.185046 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/harbor-ui-rc-77f696b975-gbqjt: no metrics known for pod
E0103 08:41:15.185066 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-master3: no metrics known for pod
E0103 08:41:15.185092 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-4gbs5: no metrics known for pod
E0103 08:41:15.185112 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod
E0103 08:41:15.185134 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-7t5d8: no metrics known for pod
E0103 08:41:15.185153 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-bcwxj: no metrics known for pod
E0103 08:41:15.185179 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-k28rd: no metrics known for pod
E0103 08:41:15.185211 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-fbcd89d77-f2p96: no metrics known for pod
E0103 08:41:15.185230 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-78x78: no metrics known for pod
E0103 08:41:15.185249 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/registry-rc-5677c8695d-qjmv8: no metrics known for pod
E0103 08:41:18.389278 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/harbor-ui-rc-77f696b975-gbqjt: no metrics known for pod
E0103 08:41:18.389320 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-controller-manager-master3: no metrics known for pod
E0103 08:41:18.389343 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-4gbs5: no metrics known for pod
E0103 08:41:18.389364 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod
E0103 08:41:18.389386 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-7t5d8: no metrics known for pod
E0103 08:41:18.389407 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/coredns-bcwxj: no metrics known for pod
E0103 08:41:18.389428 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-scheduler-master3: no metrics known for pod
E0103 08:41:18.389452 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-proxy-k28rd: no metrics known for pod
E0103 08:41:18.389484 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/metrics-server-fbcd89d77-f2p96: no metrics known for pod
E0103 08:41:18.389509 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-78x78: no metrics known for pod
E0103 08:41:18.389535 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/registry-rc-5677c8695d-qjmv8: no metrics known for pod
E0103 08:41:18.389560 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-kqbl5: no metrics known for pod
E0103 08:41:18.389582 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-knm82: no metrics known for pod
E0103 08:41:18.389603 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/kube-apiserver-master3: no metrics known for pod
E0103 08:41:18.389627 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/virtlet-nl6qb: no metrics known for pod
I0103 08:41:27.255320 1 manager.go:95] Scraping metrics from 7 sources
I0103 08:41:27.269555 1 manager.go:120] Querying source: kubelet_summary:node-1
I0103 08:41:27.269577 1 manager.go:120] Querying source: kubelet_summary:master1
I0103 08:41:27.282571 1 manager.go:120] Querying source: kubelet_summary:master2
I0103 08:41:27.295604 1 manager.go:120] Querying source: kubelet_summary:node-2
I0103 08:41:27.300552 1 manager.go:120] Querying source: kubelet_summary:master3
I0103 08:41:27.302562 1 manager.go:120] Querying source: kubelet_summary:node-4
I0103 08:41:27.303607 1 manager.go:120] Querying source: kubelet_summary:node-3
I0103 08:41:27.354555 1 manager.go:148] ScrapeMetrics: time: 99.16302ms, nodes: 7, pods: 45
the pod on master3 ,node-1 ,node-3 can not scrape, # kubectl get po -n kube-system -o wide | grep master3
calico-node-d4xpn 2/2 Running 0 15h 192.168.40.162 master3 <none> <none>
coredns-bcwxj 1/1 Running 0 15h 10.244.136.1 master3 <none> <none>
harbor-ui-rc-77f696b975-gbqjt 1/1 Running 5 7d19h 10.244.136.21 master3 <none> <none>
kube-apiserver-master3 1/1 Running 8 153d 192.168.40.162 master3 <none> <none>
kube-controller-manager-master3 1/1 Running 7 153d 192.168.40.162 master3 <none> <none>
kube-proxy-k28rd 1/1 Running 0 15h 192.168.40.162 master3 <none> <none>
kube-scheduler-master3 1/1 Running 8 29d 192.168.40.162 master3 <none> <none>
registry-rc-5677c8695d-qjmv8 1/1 Running 2 7d19h 10.244.136.2 master3 <none> <none>
# kubectl top po -n kube-system calico-node-d4xpn
Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/calico-node-d4xpn" not found
# kubectl top po -n kube-system coredns-bcwxj
Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/coredns-bcwxj" not found
# kubectl get po -n kube-system -o wide | grep node-1
calico-node-knm82 2/2 Running 12 15d 192.168.40.210 node-1 <none> <none>
kube-proxy-7t5d8 1/1 Running 6 15d 192.168.40.210 node-1 <none> <none>
# kubectl top po -n kube-system calico-node-knm82
Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/calico-node-knm82" not found
# kubectl top po -n kube-system kube-proxy-7t5d8
Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/kube-proxy-7t5d8" not found# kubectl get po -n kube-system -o wide | grep node-2
calico-node-lxkt5 2/2 Running 0 7d17h 192.168.40.235 node-2 <none> <none>
kube-proxy-ml82s 1/1 Running 0 7d17h 192.168.40.235 node-2 <none> <none>
# kubectl top po -n kube-system calico-node-lxkt5
NAME CPU(cores) MEMORY(bytes)
calico-node-lxkt5 25m 226Mi
on master3,node-1,node-3 ,the kubelet args,i add |
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
imagePullSecrets:
- name: myregistrykey
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
- name: time
hostPath:
path: /etc/localtime
- name: metrics-ca-config
configMap:
name: metrics-ca-config
containers:
- name: metrics-server
image: kylincloud2.hub/kube-system/metrics-server:v0.3.6
command:
- /metrics-server
args:
- --v=2
- --kubelet-preferred-address-types=InternalIP
- --kubelet-insecure-tls=true
- --requestheader-client-ca-file=/opt/front-proxy-ca.pem
imagePullPolicy: IfNotPresent
volumeMounts:
- name: tmp-dir
mountPath: /tmp
- name: metrics-ca-config
mountPath: /opt
- mountPath: /etc/localtime
name: time# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"archive", BuildDate:"2019-05-29T08:20:47Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/arm64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"archive", BuildDate:"2019-05-29T08:20:47Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/arm64"}
metrics-server version 0.3.6 |
|
Ok, metrics-server doesn't report errors about scraping those nodes. This would mean that scrapes succeed but return no data. Can you show results from summary api of nodes without CRI? |
root@master1:~# kubectl get po -n kube-system -o wide | grep calico
calico-kube-controllers-58596dfcf7-9g2ms 1/1 Running 2 14d 192.168.40.161 master2 <none> <none>
calico-node-86mnf 2/2 Running 23 43d 192.168.40.224 node-4 <none> <none>
calico-node-d4xpn 2/2 Running 2 43h 192.168.40.162 master3 <none> <none>
calico-node-g67fm 2/2 Running 14 43d 192.168.40.160 master1 <none> <none>
calico-node-knm82 2/2 Running 14 17d 192.168.40.210 node-1 <none> <none>
calico-node-lxkt5 2/2 Running 2 8d 192.168.40.235 node-2 <none> <none>
calico-node-rlqhs 2/2 Running 0 6m29s 192.168.40.234 node-3 <none> <none>
calico-node-xfpqj 2/2 Running 11 43d 192.168.40.161 master2 <none> <none>
root@master1:~# kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-d4xpn
Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/calico-node-d4xpn" not found
root@master1:~# kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-86mnf
{"kind":"PodMetrics","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"name":"calico-node-86mnf","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-86mnf","creationTimestamp":"2020-01-04T05:21:01Z"},"timestamp":"2020-01-04T05:17:51Z","window":"30s","containers":[{"name":"install-cni","usage":{"cpu":"458954n","memory":"59768Ki"}},{"name":"calico-node","usage":{"cpu":"29264119n","memory":"97716Ki"}}]}
master3(with cri) pods is null root@master1:~# kubectl get --raw /api/v1/nodes/master3/proxy/stats/summary
{
"node": {
"nodeName": "master3",
"systemContainers": [
{
"name": "pods",
"startTime": "2020-01-03T03:58:43Z",
"cpu": {
"time": "2020-01-04T05:30:23Z",
"usageNanoCores": 157788715,
"usageCoreNanoSeconds": 9960515390080
},
"memory": {
"time": "2020-01-04T05:30:23Z",
"availableBytes": 32240369664,
"usageBytes": 2456485888,
"workingSetBytes": 1958084608,
"rssBytes": 634388480,
"pageFaults": 0,
"majorPageFaults": 0
},
"userDefinedMetrics": null
},
{
"name": "kubelet",
"startTime": "2020-01-03T03:58:43Z",
"userDefinedMetrics": null
}
],
"startTime": "2020-01-03T03:28:23Z",
"cpu": {
"time": "2020-01-04T05:30:30Z",
"usageNanoCores": 285715089,
"usageCoreNanoSeconds": 25971640137980
},
"memory": {
"time": "2020-01-04T05:30:30Z",
"availableBytes": 29598810112,
"usageBytes": 5932580864,
"workingSetBytes": 4599644160,
"rssBytes": 1302200320,
"pageFaults": 20002695,
"majorPageFaults": 21409
},
"network": {
"time": "2020-01-04T05:30:30Z",
"name": "",
"interfaces": [
{
"name": "enp1s0f2",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "calia6014d419f1",
"rxBytes": 115406573,
"rxErrors": 0,
"txBytes": 51077836,
"txErrors": 0
},
{
"name": "dummy0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "virbr1-nic",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "enp1s0f3",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "virbr1",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "enp1s0f0",
"rxBytes": 1896846395,
"rxErrors": 0,
"txBytes": 1554194404,
"txErrors": 0
},
{
"name": "kube-ipvs0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali1f12a6c61be",
"rxBytes": 65804751,
"rxErrors": 0,
"txBytes": 81627234,
"txErrors": 0
},
{
"name": "cali24592a0a449",
"rxBytes": 6192,
"rxErrors": 0,
"txBytes": 8593,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 48566664,
"rxErrors": 0,
"txBytes": 148852706,
"txErrors": 0
},
{
"name": "enp1s0f1",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
},
"fs": {
"time": "2020-01-04T05:30:30Z",
"availableBytes": 1650124169216,
"capacityBytes": 1798376165376,
"usedBytes": 148251996160,
"inodesFree": 175179511,
"inodes": 175708416,
"inodesUsed": 528905
},
"runtime": {
"imageFs": {
"time": "2020-01-04T05:30:32Z",
"usedBytes": 148251996160,
"inodesUsed": 528905
}
},
"rlimit": {
"time": "2020-01-04T05:30:32Z",
"maxpid": 32768,
"curproc": 1355
}
},
"pods": []
}
node-4(without cri) pods is not null root@master1:~# kubectl get --raw /api/v1/nodes/node-4/proxy/stats/summary
{
"node": {
"nodeName": "node-4",
"systemContainers": [
{
"name": "kubelet",
"startTime": "2020-01-03T03:20:12Z",
"userDefinedMetrics": null
},
{
"name": "runtime",
"startTime": "2020-01-03T03:20:12Z",
"userDefinedMetrics": null
},
{
"name": "pods",
"startTime": "2020-01-03T03:20:12Z",
"cpu": {
"time": "2020-01-04T05:27:29Z",
"usageNanoCores": 70548670,
"usageCoreNanoSeconds": 16278654476180
},
"memory": {
"time": "2020-01-04T05:27:29Z",
"availableBytes": 13847207936,
"usageBytes": 20246360064,
"workingSetBytes": 19866341376,
"rssBytes": 19436134400,
"pageFaults": 0,
"majorPageFaults": 0
},
"userDefinedMetrics": null
}
],
"startTime": "2020-01-03T03:19:19Z",
"cpu": {
"time": "2020-01-04T05:27:29Z",
"usageNanoCores": 172764362,
"usageCoreNanoSeconds": 27891280039640
},
"memory": {
"time": "2020-01-04T05:27:29Z",
"availableBytes": 11636699136,
"usageBytes": 23287717888,
"workingSetBytes": 22076850176,
"rssBytes": 20128161792,
"pageFaults": 27777864,
"majorPageFaults": 3756
},
"network": {
"time": "2020-01-04T05:27:29Z",
"name": "",
"interfaces": [
{
"name": "calica560fcd2e9",
"rxBytes": 121177160,
"rxErrors": 0,
"txBytes": 20234908,
"txErrors": 0
},
{
"name": "cali98bebba79da",
"rxBytes": 657175199,
"rxErrors": 0,
"txBytes": 335835244,
"txErrors": 0
},
{
"name": "enp1s0f2",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "dummy0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali5a0b6d12df4",
"rxBytes": 10578,
"rxErrors": 0,
"txBytes": 8803,
"txErrors": 0
},
{
"name": "enp1s0f3",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "caliecaaedc2aa7",
"rxBytes": 346162,
"rxErrors": 0,
"txBytes": 870159,
"txErrors": 0
},
{
"name": "calia3d07af5b54",
"rxBytes": 99947319,
"rxErrors": 0,
"txBytes": 102993834,
"txErrors": 0
},
{
"name": "calib32f68a1729",
"rxBytes": 16559708,
"rxErrors": 0,
"txBytes": 98060208,
"txErrors": 0
},
{
"name": "enp1s0f0",
"rxBytes": 3283897734,
"rxErrors": 0,
"txBytes": 3324298401,
"txErrors": 0
},
{
"name": "kube-ipvs0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali7305fa7cdba",
"rxBytes": 99866306,
"rxErrors": 0,
"txBytes": 100785221,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 489056546,
"rxErrors": 0,
"txBytes": 820367853,
"txErrors": 0
},
{
"name": "enp1s0f1",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
},
"fs": {
"time": "2020-01-04T05:27:29Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 88167452672,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"runtime": {
"imageFs": {
"time": "2020-01-04T05:27:29Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 15058696737,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
}
},
"rlimit": {
"time": "2020-01-04T05:27:35Z",
"maxpid": 32768,
"curproc": 1394
}
},
"pods": [
{
"podRef": {
"name": "web-84dd6df4b9-zkclr",
"namespace": "istio",
"uid": "9336abbc-2787-11ea-b8e3-00073e906e4b"
},
"startTime": "2019-12-26T02:21:19Z",
"containers": [
{
"name": "web",
"startTime": "2020-01-03T03:20:55Z",
"cpu": {
"time": "2020-01-04T05:27:33Z",
"usageNanoCores": 281527,
"usageCoreNanoSeconds": 30610108080
},
"memory": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 3988828160,
"usageBytes": 109318144,
"workingSetBytes": 107171840,
"rssBytes": 98439168,
"pageFaults": 2341,
"majorPageFaults": 35
},
"rootfs": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 61440,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 16
},
"logs": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 36864,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
},
{
"name": "istio-proxy",
"startTime": "2020-01-03T03:20:56Z",
"cpu": {
"time": "2020-01-04T05:27:27Z",
"usageNanoCores": 8142872,
"usageCoreNanoSeconds": 1143207882140
},
"memory": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1014878208,
"usageBytes": 101339136,
"workingSetBytes": 58863616,
"rssBytes": 45826048,
"pageFaults": 14324,
"majorPageFaults": 462
},
"rootfs": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 36864,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 10
},
"logs": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 31080448,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:35Z",
"usageNanoCores": 9138152,
"usageCoreNanoSeconds": 1174081004200
},
"memory": {
"time": "2020-01-04T05:27:35Z",
"availableBytes": 5002387456,
"usageBytes": 211976192,
"workingSetBytes": 167354368,
"rssBytes": 144384000,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:27Z",
"name": "eth0",
"rxBytes": 100784754,
"rxErrors": 0,
"txBytes": 99865857,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 100784754,
"rxErrors": 0,
"txBytes": 99865857,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "default-token-hft25"
},
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "istio-certs"
},
{
"time": "2020-01-04T05:26:44Z",
"availableBytes": 16856764416,
"capacityBytes": 16856772608,
"usedBytes": 8192,
"inodesFree": 4115421,
"inodes": 4115423,
"inodesUsed": 2,
"name": "istio-envoy"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 31215616,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 26
}
},
{
"podRef": {
"name": "calico-node-86mnf",
"namespace": "kube-system",
"uid": "21bcf17e-0ce8-11ea-b492-00073e9087b4"
},
"startTime": "2019-11-22T05:15:53Z",
"containers": [
{
"name": "install-cni",
"startTime": "2020-01-03T03:20:36Z",
"cpu": {
"time": "2020-01-04T05:27:30Z",
"usageNanoCores": 344124,
"usageCoreNanoSeconds": 46625916480
},
"memory": {
"time": "2020-01-04T05:27:30Z",
"usageBytes": 165068800,
"workingSetBytes": 61300736,
"rssBytes": 106496,
"pageFaults": 2616035,
"majorPageFaults": 26
},
"rootfs": {
"time": "2020-01-04T05:27:30Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 61440,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 16
},
"logs": {
"time": "2020-01-04T05:27:30Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 32768,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
},
{
"name": "calico-node",
"startTime": "2020-01-03T03:20:34Z",
"cpu": {
"time": "2020-01-04T05:27:34Z",
"usageNanoCores": 45279592,
"usageCoreNanoSeconds": 2463260516140
},
"memory": {
"time": "2020-01-04T05:27:34Z",
"usageBytes": 114135040,
"workingSetBytes": 103915520,
"rssBytes": 48988160,
"pageFaults": 3773534,
"majorPageFaults": 352
},
"rootfs": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 327680,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 105
},
"logs": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 32768,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:26Z",
"usageNanoCores": 16365859,
"usageCoreNanoSeconds": 2509432748800
},
"memory": {
"time": "2020-01-04T05:27:26Z",
"usageBytes": 275316736,
"workingSetBytes": 161329152,
"rssBytes": 44941312,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:32Z",
"name": "",
"interfaces": [
{
"name": "calica560fcd2e9",
"rxBytes": 121180800,
"rxErrors": 0,
"txBytes": 20235375,
"txErrors": 0
},
{
"name": "cali98bebba79da",
"rxBytes": 657201661,
"rxErrors": 0,
"txBytes": 335875709,
"txErrors": 0
},
{
"name": "enp1s0f2",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "dummy0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali5a0b6d12df4",
"rxBytes": 10578,
"rxErrors": 0,
"txBytes": 8803,
"txErrors": 0
},
{
"name": "enp1s0f3",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "caliecaaedc2aa7",
"rxBytes": 346162,
"rxErrors": 0,
"txBytes": 870159,
"txErrors": 0
},
{
"name": "calia3d07af5b54",
"rxBytes": 99948178,
"rxErrors": 0,
"txBytes": 102995009,
"txErrors": 0
},
{
"name": "calib32f68a1729",
"rxBytes": 16559708,
"rxErrors": 0,
"txBytes": 98060208,
"txErrors": 0
},
{
"name": "enp1s0f0",
"rxBytes": 3284317114,
"rxErrors": 0,
"txBytes": 3324651101,
"txErrors": 0
},
{
"name": "kube-ipvs0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali7305fa7cdba",
"rxBytes": 99867165,
"rxErrors": 0,
"txBytes": 100786461,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 489097585,
"rxErrors": 0,
"txBytes": 820394407,
"txErrors": 0
},
{
"name": "enp1s0f1",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "calico-node-token-hdngt"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 454656,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 121
}
},
{
"podRef": {
"name": "webterminal-6565496776-w4kx2",
"namespace": "istio",
"uid": "9335921d-2787-11ea-b8e3-00073e906e4b"
},
"startTime": "2019-12-26T02:21:19Z",
"containers": [
{
"name": "logs",
"startTime": "2020-01-03T03:20:57Z",
"cpu": {
"time": "2020-01-04T05:27:19Z",
"usageNanoCores": 0,
"usageCoreNanoSeconds": 72214220
},
"memory": {
"time": "2020-01-04T05:27:19Z",
"availableBytes": 506687488,
"usageBytes": 5312512,
"workingSetBytes": 5312512,
"rssBytes": 4464640,
"pageFaults": 376,
"majorPageFaults": 0
},
"rootfs": {
"time": "2020-01-04T05:27:19Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 14
},
"logs": {
"time": "2020-01-04T05:27:19Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 36864,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
},
{
"name": "webterminal",
"startTime": "2020-01-03T03:20:56Z",
"cpu": {
"time": "2020-01-04T05:27:22Z",
"usageNanoCores": 0,
"usageCoreNanoSeconds": 80776720
},
"memory": {
"time": "2020-01-04T05:27:22Z",
"availableBytes": 503918592,
"usageBytes": 19369984,
"workingSetBytes": 8081408,
"rssBytes": 6500352,
"pageFaults": 392,
"majorPageFaults": 2
},
"rootfs": {
"time": "2020-01-04T05:27:22Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 14
},
"logs": {
"time": "2020-01-04T05:27:22Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 36864,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
},
{
"name": "istio-proxy",
"startTime": "2020-01-03T03:20:58Z",
"cpu": {
"time": "2020-01-04T05:27:33Z",
"usageNanoCores": 9011775,
"usageCoreNanoSeconds": 1142902798000
},
"memory": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1012293632,
"usageBytes": 75988992,
"workingSetBytes": 61448192,
"rssBytes": 49422336,
"pageFaults": 13281,
"majorPageFaults": 232
},
"rootfs": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 36864,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 10
},
"logs": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 37216256,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:24Z",
"usageNanoCores": 8576830,
"usageCoreNanoSeconds": 1143168325560
},
"memory": {
"time": "2020-01-04T05:27:24Z",
"availableBytes": 2019971072,
"usageBytes": 103800832,
"workingSetBytes": 77770752,
"rssBytes": 60424192,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:24Z",
"name": "eth0",
"rxBytes": 102992059,
"rxErrors": 0,
"txBytes": 99945945,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 102992059,
"rxErrors": 0,
"txBytes": 99945945,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:13Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "default-token-hft25"
},
{
"time": "2020-01-04T05:26:06Z",
"availableBytes": 16856764416,
"capacityBytes": 16856772608,
"usedBytes": 8192,
"inodesFree": 4115421,
"inodes": 4115423,
"inodesUsed": 2,
"name": "istio-envoy"
},
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "istio-certs"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 37416960,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 38
}
},
{
"podRef": {
"name": "elasticsearch2-5cc88657b9-lcsxg",
"namespace": "logging",
"uid": "a142d255-2863-11ea-b8e3-00073e906e4b"
},
"startTime": "2019-12-27T04:41:27Z",
"containers": [
{
"name": "elasticsearch",
"startTime": "2020-01-03T03:24:42Z",
"cpu": {
"time": "2020-01-04T05:27:28Z",
"usageNanoCores": 22688305,
"usageCoreNanoSeconds": 5574364426960
},
"memory": {
"time": "2020-01-04T05:27:28Z",
"availableBytes": 1073893376,
"usageBytes": 19007987712,
"workingSetBytes": 18926104576,
"rssBytes": 18797940736,
"pageFaults": 241988,
"majorPageFaults": 10969
},
"rootfs": {
"time": "2020-01-04T05:27:28Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 86016,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 15
},
"logs": {
"time": "2020-01-04T05:27:28Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 65536,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:33Z",
"usageNanoCores": 83645829,
"usageCoreNanoSeconds": 5702627996600
},
"memory": {
"time": "2020-01-04T05:27:33Z",
"availableBytes": 1020923904,
"usageBytes": 19075493888,
"workingSetBytes": 18979074048,
"rssBytes": 18797981696,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:30Z",
"name": "eth0",
"rxBytes": 335836414,
"rxErrors": 0,
"txBytes": 657186466,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 335836414,
"rxErrors": 0,
"txBytes": 657186466,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 1585023787008,
"capacityBytes": 1762321661952,
"usedBytes": 24576,
"inodesFree": 108965862,
"inodes": 109289472,
"inodesUsed": 9,
"name": "es2-config-volume"
},
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "elasticsearch-token-lm5x5"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:29Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 176128,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 24
}
},
{
"podRef": {
"name": "alertmanager-main-0",
"namespace": "istio-system",
"uid": "933d1e2c-2787-11ea-b8e3-00073e906e4b"
},
"startTime": "2019-12-26T02:21:19Z",
"containers": [
{
"name": "config-reloader",
"startTime": "2020-01-03T03:20:46Z",
"cpu": {
"time": "2020-01-04T05:27:27Z",
"usageNanoCores": 0,
"usageCoreNanoSeconds": 1003554140
},
"memory": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 6565888,
"usageBytes": 6889472,
"workingSetBytes": 3919872,
"rssBytes": 2228224,
"pageFaults": 639,
"majorPageFaults": 25
},
"rootfs": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 12
},
"logs": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 36864,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
},
{
"name": "alertmanager",
"startTime": "2020-01-03T03:20:45Z",
"cpu": {
"time": "2020-01-04T05:27:30Z",
"usageNanoCores": 1716133,
"usageCoreNanoSeconds": 200407796020
},
"memory": {
"time": "2020-01-04T05:27:30Z",
"usageBytes": 33624064,
"workingSetBytes": 22462464,
"rssBytes": 14168064,
"pageFaults": 7804,
"majorPageFaults": 95
},
"rootfs": {
"time": "2020-01-04T05:27:30Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 12
},
"logs": {
"time": "2020-01-04T05:27:30Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 311296,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:25Z",
"usageNanoCores": 1215366,
"usageCoreNanoSeconds": 201428860520
},
"memory": {
"time": "2020-01-04T05:27:25Z",
"usageBytes": 41095168,
"workingSetBytes": 26963968,
"rssBytes": 16433152,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:32Z",
"name": "eth0",
"rxBytes": 20235375,
"rxErrors": 0,
"txBytes": 121180800,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 20235375,
"rxErrors": 0,
"txBytes": 121180800,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-04T05:27:13Z",
"availableBytes": 1584609984512,
"capacityBytes": 1762321661952,
"usedBytes": 4096,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 3,
"name": "alertmanager-main-db"
},
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "alertmanager-main-token-x2xkr"
},
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856768512,
"capacityBytes": 16856772608,
"usedBytes": 4096,
"inodesFree": 4115418,
"inodes": 4115423,
"inodesUsed": 5,
"name": "config-volume"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:30Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 442368,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 27
}
},
{
"podRef": {
"name": "kube-proxy-vbbt5",
"namespace": "kube-system",
"uid": "17df212a-0ce8-11ea-b492-00073e9087b4"
},
"startTime": "2019-11-22T05:15:37Z",
"containers": [
{
"name": "kube-proxy",
"startTime": "2020-01-03T03:20:34Z",
"cpu": {
"time": "2020-01-04T05:27:31Z",
"usageNanoCores": 532085,
"usageCoreNanoSeconds": 4428633982280
},
"memory": {
"time": "2020-01-04T05:27:31Z",
"usageBytes": 84955136,
"workingSetBytes": 57352192,
"rssBytes": 29069312,
"pageFaults": 2815,
"majorPageFaults": 239
},
"rootfs": {
"time": "2020-01-04T05:27:31Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 53248,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 19
},
"logs": {
"time": "2020-01-04T05:27:31Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 172032,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:30Z",
"usageNanoCores": 754397,
"usageCoreNanoSeconds": 4428649459860
},
"memory": {
"time": "2020-01-04T05:27:30Z",
"usageBytes": 86409216,
"workingSetBytes": 58265600,
"rssBytes": 29110272,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:22Z",
"name": "",
"interfaces": [
{
"name": "calica560fcd2e9",
"rxBytes": 121165168,
"rxErrors": 0,
"txBytes": 20233509,
"txErrors": 0
},
{
"name": "cali98bebba79da",
"rxBytes": 657148046,
"rxErrors": 0,
"txBytes": 335829387,
"txErrors": 0
},
{
"name": "enp1s0f2",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "dummy0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali5a0b6d12df4",
"rxBytes": 10578,
"rxErrors": 0,
"txBytes": 8803,
"txErrors": 0
},
{
"name": "enp1s0f3",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "caliecaaedc2aa7",
"rxBytes": 346162,
"rxErrors": 0,
"txBytes": 870159,
"txErrors": 0
},
{
"name": "calia3d07af5b54",
"rxBytes": 99945047,
"rxErrors": 0,
"txBytes": 102990991,
"txErrors": 0
},
{
"name": "calib32f68a1729",
"rxBytes": 16557953,
"rxErrors": 0,
"txBytes": 98059876,
"txErrors": 0
},
{
"name": "enp1s0f0",
"rxBytes": 3283858418,
"rxErrors": 0,
"txBytes": 3324245888,
"txErrors": 0
},
{
"name": "kube-ipvs0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali7305fa7cdba",
"rxBytes": 99864034,
"rxErrors": 0,
"txBytes": 100782514,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 489049290,
"rxErrors": 0,
"txBytes": 820334065,
"txErrors": 0
},
{
"name": "enp1s0f1",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "kube-proxy-token-b6h4h"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:31Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 225280,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 19
}
},
{
"podRef": {
"name": "fluentd-es-cr2l7",
"namespace": "logging",
"uid": "db5810df-dddd-11e9-8b17-00073e906e4b"
},
"startTime": "2019-09-23T08:40:19Z",
"containers": [
{
"name": "fluentd-es",
"startTime": "2020-01-03T03:20:51Z",
"cpu": {
"time": "2020-01-04T05:27:34Z",
"usageNanoCores": 6329225,
"usageCoreNanoSeconds": 627562552740
},
"memory": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 280829952,
"usageBytes": 252796928,
"workingSetBytes": 243458048,
"rssBytes": 220872704,
"pageFaults": 10069,
"majorPageFaults": 81
},
"rootfs": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 65536,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 18
},
"logs": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 69632,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:30Z",
"usageNanoCores": 6184810,
"usageCoreNanoSeconds": 627554074340
},
"memory": {
"time": "2020-01-04T05:27:30Z",
"availableBytes": 279449600,
"usageBytes": 254177280,
"workingSetBytes": 244838400,
"rssBytes": 220909568,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:16Z",
"name": "eth0",
"rxBytes": 8803,
"rxErrors": 0,
"txBytes": 10578,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 8803,
"rxErrors": 0,
"txBytes": 10578,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 1585023795200,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965862,
"inodes": 109289472,
"inodesUsed": 21,
"name": "config-volume"
},
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "fluentd-es-token-hcpt5"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:34Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 180224,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 39
}
},
{
"podRef": {
"name": "prometheus-operator-5b7cf77679-49s6s",
"namespace": "istio-system",
"uid": "933238cd-2787-11ea-b8e3-00073e906e4b"
},
"startTime": "2019-12-26T02:21:19Z",
"containers": [
{
"name": "prometheus-operator",
"startTime": "2020-01-03T03:20:53Z",
"cpu": {
"time": "2020-01-04T05:27:32Z",
"usageNanoCores": 437946,
"usageCoreNanoSeconds": 64882837520
},
"memory": {
"time": "2020-01-04T05:27:32Z",
"usageBytes": 50626560,
"workingSetBytes": 31584256,
"rssBytes": 26521600,
"pageFaults": 13641,
"majorPageFaults": 171
},
"rootfs": {
"time": "2020-01-04T05:27:32Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 32768,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 9
},
"logs": {
"time": "2020-01-04T05:27:32Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 331776,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:24Z",
"usageNanoCores": 2170670,
"usageCoreNanoSeconds": 64895030680
},
"memory": {
"time": "2020-01-04T05:27:24Z",
"usageBytes": 51544064,
"workingSetBytes": 32501760,
"rssBytes": 26562560,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:33Z",
"name": "eth0",
"rxBytes": 98060208,
"rxErrors": 0,
"txBytes": 16559708,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 98060208,
"rxErrors": 0,
"txBytes": 16559708,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "prometheus-operator-token-g6qtl"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:32Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 364544,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 9
}
},
{
"podRef": {
"name": "node-exporter-mfs9r",
"namespace": "istio-system",
"uid": "a7c9ed1a-e0e5-11e9-8b17-00073e906e4b"
},
"startTime": "2019-09-27T05:13:42Z",
"containers": [
{
"name": "kube-rbac-proxy",
"startTime": "2020-01-03T03:20:35Z",
"cpu": {
"time": "2020-01-04T05:27:31Z",
"usageNanoCores": 114776,
"usageCoreNanoSeconds": 25422415440
},
"memory": {
"time": "2020-01-04T05:27:31Z",
"usageBytes": 30232576,
"workingSetBytes": 14573568,
"rssBytes": 10424320,
"pageFaults": 7045,
"majorPageFaults": 125
},
"rootfs": {
"time": "2020-01-04T05:27:31Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 12
},
"logs": {
"time": "2020-01-04T05:27:31Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 28672,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
},
{
"name": "node-exporter",
"startTime": "2020-01-03T03:20:34Z",
"cpu": {
"time": "2020-01-04T05:27:26Z",
"usageNanoCores": 5324898,
"usageCoreNanoSeconds": 297120132520
},
"memory": {
"time": "2020-01-04T05:27:26Z",
"usageBytes": 92327936,
"workingSetBytes": 82845696,
"rssBytes": 76128256,
"pageFaults": 6239,
"majorPageFaults": 86
},
"rootfs": {
"time": "2020-01-04T05:27:26Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 45056,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 12
},
"logs": {
"time": "2020-01-04T05:27:26Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 40960,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:24Z",
"usageNanoCores": 7447484,
"usageCoreNanoSeconds": 322557805780
},
"memory": {
"time": "2020-01-04T05:27:24Z",
"usageBytes": 123011072,
"workingSetBytes": 97869824,
"rssBytes": 86589440,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:35Z",
"name": "",
"interfaces": [
{
"name": "calica560fcd2e9",
"rxBytes": 121185036,
"rxErrors": 0,
"txBytes": 20236923,
"txErrors": 0
},
{
"name": "cali98bebba79da",
"rxBytes": 657378385,
"rxErrors": 0,
"txBytes": 335880908,
"txErrors": 0
},
{
"name": "enp1s0f2",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "dummy0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali5a0b6d12df4",
"rxBytes": 10578,
"rxErrors": 0,
"txBytes": 8803,
"txErrors": 0
},
{
"name": "enp1s0f3",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "caliecaaedc2aa7",
"rxBytes": 346162,
"rxErrors": 0,
"txBytes": 870159,
"txErrors": 0
},
{
"name": "calia3d07af5b54",
"rxBytes": 99949486,
"rxErrors": 0,
"txBytes": 102996652,
"txErrors": 0
},
{
"name": "calib32f68a1729",
"rxBytes": 16559708,
"rxErrors": 0,
"txBytes": 98060208,
"txErrors": 0
},
{
"name": "enp1s0f0",
"rxBytes": 3284369430,
"rxErrors": 0,
"txBytes": 3324864794,
"txErrors": 0
},
{
"name": "kube-ipvs0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "cali7305fa7cdba",
"rxBytes": 99868473,
"rxErrors": 0,
"txBytes": 100788234,
"txErrors": 0
},
{
"name": "tunl0",
"rxBytes": 489103385,
"rxErrors": 0,
"txBytes": 820557095,
"txErrors": 0
},
{
"name": "enp1s0f1",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "node-exporter-token-48g4n"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:31Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 159744,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 24
}
},
{
"podRef": {
"name": "redis-rc-5cdd78555-mrhlc",
"namespace": "kube-system",
"uid": "b432e669-2d09-11ea-b8e3-00073e906e4b"
},
"startTime": "2020-01-02T02:40:10Z",
"containers": [
{
"name": "redis-app",
"startTime": "2020-01-03T03:20:54Z",
"cpu": {
"time": "2020-01-04T05:27:27Z",
"usageNanoCores": 1201280,
"usageCoreNanoSeconds": 105543482180
},
"memory": {
"time": "2020-01-04T05:27:27Z",
"usageBytes": 22634496,
"workingSetBytes": 19472384,
"rssBytes": 8843264,
"pageFaults": 6040,
"majorPageFaults": 53
},
"rootfs": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 28672,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 8
},
"logs": {
"time": "2020-01-04T05:27:27Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 49152,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 323547
},
"userDefinedMetrics": null
}
],
"cpu": {
"time": "2020-01-04T05:27:18Z",
"usageNanoCores": 1255203,
"usageCoreNanoSeconds": 105549985000
},
"memory": {
"time": "2020-01-04T05:27:18Z",
"usageBytes": 23699456,
"workingSetBytes": 20537344,
"rssBytes": 8880128,
"pageFaults": 0,
"majorPageFaults": 0
},
"network": {
"time": "2020-01-04T05:27:18Z",
"name": "eth0",
"rxBytes": 870159,
"rxErrors": 0,
"txBytes": 346162,
"txErrors": 0,
"interfaces": [
{
"name": "tunl0",
"rxBytes": 0,
"rxErrors": 0,
"txBytes": 0,
"txErrors": 0
},
{
"name": "eth0",
"rxBytes": 870159,
"rxErrors": 0,
"txBytes": 346162,
"txErrors": 0
}
]
},
"volume": [
{
"time": "2020-01-03T03:21:12Z",
"availableBytes": 16856760320,
"capacityBytes": 16856772608,
"usedBytes": 12288,
"inodesFree": 4115414,
"inodes": 4115423,
"inodesUsed": 9,
"name": "default-token-rk8nk"
}
],
"ephemeral-storage": {
"time": "2020-01-04T05:27:29Z",
"availableBytes": 1584609857536,
"capacityBytes": 1762321661952,
"usedBytes": 77824,
"inodesFree": 108965925,
"inodes": 109289472,
"inodesUsed": 8
}
}
]
}
metrics-server logs root@master1:~# kubectl -n kube-system logs metrics-server-fbcd89d77-f2p96
I0104 13:16:24.067654 1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0104 13:16:29.244160 1 manager.go:95] Scraping metrics from 0 sources
I0104 13:16:29.244249 1 manager.go:148] ScrapeMetrics: time: 6.06µs, nodes: 0, pods: 0
I0104 13:16:29.287349 1 secure_serving.go:116] Serving securely on [::]:443
I0104 13:17:29.244320 1 manager.go:95] Scraping metrics from 7 sources
I0104 13:17:29.256617 1 manager.go:120] Querying source: kubelet_summary:node-2
I0104 13:17:29.259553 1 manager.go:120] Querying source: kubelet_summary:master3
I0104 13:17:29.261593 1 manager.go:120] Querying source: kubelet_summary:master1
I0104 13:17:29.276647 1 manager.go:120] Querying source: kubelet_summary:node-3
I0104 13:17:29.279565 1 manager.go:120] Querying source: kubelet_summary:master2
I0104 13:17:29.285574 1 manager.go:120] Querying source: kubelet_summary:node-1
I0104 13:17:29.290582 1 manager.go:120] Querying source: kubelet_summary:node-4
I0104 13:17:29.377422 1 manager.go:148] ScrapeMetrics: time: 133.01824ms, nodes: 7, pods: 50
I0104 13:19:29.326403 1 manager.go:148] ScrapeMetrics: time: 81.9234ms, nodes: 7, pods: 50
E0104 13:19:30.215156 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod
E0104 13:19:30.215206 1 reststorage.go:133] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod "kube-system/calico-node-d4xpn"
I0104 13:20:29.244392 1 manager.go:95] Scraping metrics from 7 sources
I0104 13:20:29.251603 1 manager.go:120] Querying source: kubelet_summary:master1
I0104 13:20:29.259707 1 manager.go:120] Querying source: kubelet_summary:master2
I0104 13:20:29.260070 1 manager.go:120] Querying source: kubelet_summary:node-4
I0104 13:20:29.267838 1 manager.go:120] Querying source: kubelet_summary:node-2
I0104 13:20:29.278666 1 manager.go:120] Querying source: kubelet_summary:node-3
I0104 13:20:29.290737 1 manager.go:120] Querying source: kubelet_summary:node-1
I0104 13:20:29.292723 1 manager.go:120] Querying source: kubelet_summary:master3
I0104 13:20:29.353339 1 manager.go:148] ScrapeMetrics: time: 108.8629ms, nodes: 7, pods: 50
E0104 13:20:42.768164 1 reststorage.go:160] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod
E0104 13:20:42.768212 1 reststorage.go:133] unable to fetch pod metrics for pod kube-system/calico-node-d4xpn: no metrics known for pod "kube-system/calico-node-d4xpn"
I0104 13:21:29.244328 1 manager.go:95] Scraping metrics from 7 sources
I0104 13:21:29.264645 1 manager.go:120] Querying source: kubelet_summary:node-1
I0104 13:21:29.265552 1 manager.go:120] Querying source: kubelet_summary:master3
I0104 13:21:29.268548 1 manager.go:120] Querying source: kubelet_summary:node-3
I0104 13:21:29.271589 1 manager.go:120] Querying source: kubelet_summary:node-4
I0104 13:21:29.283628 1 manager.go:120] Querying source: kubelet_summary:master1
I0104 13:21:29.290135 1 manager.go:120] Querying source: kubelet_summary:master2
I0104 13:21:29.294189 1 manager.go:120] Querying source: kubelet_summary:node-2
I0104 13:21:29.323354 1 manager.go:148] ScrapeMetrics: time: 78.94962ms, nodes: 7, pods: 50
calcio on node-4(without cri) root@master1:~# kubectl top po -n kube-system --v=8 calico-node-86mnf
I0104 13:33:28.107117 30098 loader.go:359] Config loaded from file /root/.kube/config
I0104 13:33:28.109975 30098 round_trippers.go:416] GET https://192.168.40.121:6443/api?timeout=32s
I0104 13:33:28.110005 30098 round_trippers.go:423] Request Headers:
I0104 13:33:28.110031 30098 round_trippers.go:426] Accept: application/json, */*
I0104 13:33:28.110050 30098 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (linux/arm64) kubernetes/66049e3
I0104 13:33:28.135527 30098 round_trippers.go:441] Response Status: 200 OK in 25 milliseconds
I0104 13:33:28.135564 30098 round_trippers.go:444] Response Headers:
I0104 13:33:28.135582 30098 round_trippers.go:447] Audit-Id: 66aa397d-bee7-4764-a722-193ebce32772
I0104 13:33:28.135601 30098 round_trippers.go:447] Content-Type: application/json
I0104 13:33:28.135617 30098 round_trippers.go:447] Content-Length: 137
I0104 13:33:28.135633 30098 round_trippers.go:447] Date: Sat, 04 Jan 2020 05:33:28 GMT
I0104 13:33:28.135692 30098 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.40.160:6443"}]}
I0104 13:33:28.136154 30098 round_trippers.go:416] GET https://192.168.40.121:6443/apis?timeout=32s
I0104 13:33:28.136176 30098 round_trippers.go:423] Request Headers:
I0104 13:33:28.136194 30098 round_trippers.go:426] Accept: application/json, */*
I0104 13:33:28.136212 30098 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (linux/arm64) kubernetes/66049e3
I0104 13:33:28.137685 30098 round_trippers.go:441] Response Status: 200 OK in 1 milliseconds
I0104 13:33:28.137711 30098 round_trippers.go:444] Response Headers:
I0104 13:33:28.137731 30098 round_trippers.go:447] Audit-Id: b754f195-4a41-465d-b98a-77ec61f2d459
I0104 13:33:28.137745 30098 round_trippers.go:447] Content-Type: application/json
I0104 13:33:28.137758 30098 round_trippers.go:447] Date: Sat, 04 Jan 2020 05:33:28 GMT
I0104 13:33:28.138111 30098 request.go:942] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"},{"groupVersion":"apps/v1beta2","version":"v1beta2"},{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authenticati [truncated 4730 chars]
I0104 13:33:28.139191 30098 round_trippers.go:416] GET https://192.168.40.121:6443/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-86mnf
I0104 13:33:28.139219 30098 round_trippers.go:423] Request Headers:
I0104 13:33:28.139238 30098 round_trippers.go:426] Accept: application/json, */*
I0104 13:33:28.139256 30098 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (linux/arm64) kubernetes/66049e3
I0104 13:33:28.143331 30098 round_trippers.go:441] Response Status: 200 OK in 4 milliseconds
I0104 13:33:28.143361 30098 round_trippers.go:444] Response Headers:
I0104 13:33:28.143381 30098 round_trippers.go:447] Date: Sat, 04 Jan 2020 05:27:10 GMT
I0104 13:33:28.143398 30098 round_trippers.go:447] Content-Length: 461
I0104 13:33:28.143412 30098 round_trippers.go:447] Audit-Id: e6cf4397-e36f-4a1b-853f-8edbd08962f1
I0104 13:33:28.143425 30098 round_trippers.go:447] Content-Type: application/json
I0104 13:33:28.143587 30098 request.go:942] Response Body: {"kind":"PodMetrics","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"name":"calico-node-86mnf","namespace":"kube-system","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-86mnf","creationTimestamp":"2020-01-04T05:27:10Z"},"timestamp":"2020-01-04T05:23:55Z","window":"30s","containers":[{"name":"install-cni","usage":{"cpu":"503984n","memory":"59672Ki"}},{"name":"calico-node","usage":{"cpu":"25781998n","memory":"111620Ki"}}]}
NAME CPU(cores) MEMORY(bytes)
calico-node-86mnf 27m 167Mi calico on master3(with cri) root@master1:~# kubectl top po -n kube-system --v=8 calico-node-d4xpn
I0104 13:33:39.333864 30127 loader.go:359] Config loaded from file /root/.kube/config
I0104 13:33:39.336686 30127 round_trippers.go:416] GET https://192.168.40.121:6443/api?timeout=32s
I0104 13:33:39.336714 30127 round_trippers.go:423] Request Headers:
I0104 13:33:39.336733 30127 round_trippers.go:426] Accept: application/json, */*
I0104 13:33:39.336752 30127 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (linux/arm64) kubernetes/66049e3
I0104 13:33:39.362382 30127 round_trippers.go:441] Response Status: 200 OK in 25 milliseconds
I0104 13:33:39.362433 30127 round_trippers.go:444] Response Headers:
I0104 13:33:39.362468 30127 round_trippers.go:447] Audit-Id: afdcf714-e2a5-4ee3-a7a4-1485a53655e0
I0104 13:33:39.362485 30127 round_trippers.go:447] Content-Type: application/json
I0104 13:33:39.362500 30127 round_trippers.go:447] Content-Length: 137
I0104 13:33:39.362515 30127 round_trippers.go:447] Date: Sat, 04 Jan 2020 05:33:39 GMT
I0104 13:33:39.362806 30127 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.40.160:6443"}]}
I0104 13:33:39.363417 30127 round_trippers.go:416] GET https://192.168.40.121:6443/apis?timeout=32s
I0104 13:33:39.363441 30127 round_trippers.go:423] Request Headers:
I0104 13:33:39.363459 30127 round_trippers.go:426] Accept: application/json, */*
I0104 13:33:39.363477 30127 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (linux/arm64) kubernetes/66049e3
I0104 13:33:39.365006 30127 round_trippers.go:441] Response Status: 200 OK in 1 milliseconds
I0104 13:33:39.365031 30127 round_trippers.go:444] Response Headers:
I0104 13:33:39.365049 30127 round_trippers.go:447] Audit-Id: 164ed330-f081-4c19-89e9-f34693d11770
I0104 13:33:39.365066 30127 round_trippers.go:447] Content-Type: application/json
I0104 13:33:39.365082 30127 round_trippers.go:447] Date: Sat, 04 Jan 2020 05:33:39 GMT
I0104 13:33:39.365459 30127 request.go:942] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"},{"groupVersion":"apps/v1beta2","version":"v1beta2"},{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authenticati [truncated 4730 chars]
I0104 13:33:39.366408 30127 round_trippers.go:416] GET https://192.168.40.121:6443/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/calico-node-d4xpn
I0104 13:33:39.366438 30127 round_trippers.go:423] Request Headers:
I0104 13:33:39.366456 30127 round_trippers.go:426] Accept: application/json, */*
I0104 13:33:39.366474 30127 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (linux/arm64) kubernetes/66049e3
I0104 13:33:39.370315 30127 round_trippers.go:441] Response Status: 404 Not Found in 3 milliseconds
I0104 13:33:39.370341 30127 round_trippers.go:444] Response Headers:
I0104 13:33:39.370357 30127 round_trippers.go:447] Audit-Id: a0c5d307-ad75-4903-9de5-d5abc774fdb1
I0104 13:33:39.370373 30127 round_trippers.go:447] Content-Type: application/json
I0104 13:33:39.370388 30127 round_trippers.go:447] Date: Sat, 04 Jan 2020 05:27:21 GMT
I0104 13:33:39.370404 30127 round_trippers.go:447] Content-Length: 278
I0104 13:33:39.370549 30127 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"podmetrics.metrics.k8s.io \"kube-system/calico-node-d4xpn\" not found","reason":"NotFound","details":{"name":"kube-system/calico-node-d4xpn","group":"metrics.k8s.io","kind":"podmetrics"},"code":404}
I0104 13:33:39.371571 30127 helpers.go:196] server response object: [{
"metadata": {},
"status": "Failure",
"message": "podmetrics.metrics.k8s.io \"kube-system/calico-node-d4xpn\" not found",
"reason": "NotFound",
"details": {
"name": "kube-system/calico-node-d4xpn",
"group": "metrics.k8s.io",
"kind": "podmetrics"
},
"code": 404
}]
F0104 13:33:39.371632 30127 helpers.go:114] Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/calico-node-d4xpn" not found
what is the problem？ |
|
Closing per Kubernetes issue triage policy GitHub is not the right place for support requests. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
when kubelet add the args
--container-runtime=remote --container-runtime-endpoint=unix:///run/criproxy.sock --image-service-endpoint=unix:///run/criproxy.sock --enable-controller-attac h-detach=false --feature-gates=RuntimeClass=falsekubectl top po error
kubectl top po harbor-ui-rc-77f696b975-gbqjt Error from server (NotFound): podmetrics.metrics.k8s.io "kube-system/harbor-ui-rc-77f696b975-gbqjt" not foundmetrics-server logs
The text was updated successfully, but these errors were encountered: