Fix Grafana dashboards on k8s 1.16

staging/prometheus-operator/Chart.yaml

@@ -11,7 +11,7 @@ name: prometheus-operator
 sources:
   - https://github.com/coreos/prometheus-operator
   - https://coreos.com/operators/prometheus
-version: 5.19.7
+version: 5.20.0
 appVersion: 0.31.1
 home: https://github.com/coreos/prometheus-operator
 icon: https://raw.githubusercontent.com/prometheus/prometheus.github.io/master/assets/prometheus_logo-cb55bb5c346.png

staging/prometheus-operator/OWNERS

@@ -1,8 +1,6 @@
 approvers:
 - gianrubio
 - vsliouniaev
-- anothertobi
 reviewers:
 - gianrubio
 - vsliouniaev
-- anothertobi

staging/prometheus-operator/README.md

@@ -117,15 +117,18 @@ The following tables list the configurable parameters of the prometheus-operator
 | `defaultRules.rules.kubernetesResources` | Create Kubernetes Resources  default rules| `true` |
 | `defaultRules.rules.kubernetesStorage` | Create Kubernetes Storage  default rules| `true` |
 | `defaultRules.rules.kubernetesSystem` | Create Kubernetes System  default rules| `true` |
-| `defaultRules.rules.node` | Create Node  default rules| `true` |
+| `defaultRules.rules.node` | Create Node default rules | `true` |
+| `defaultRules.rules.network` | Create networking default rules | `true` |
 | `defaultRules.rules.PrometheusOperator` | Create Prometheus Operator  default rules| `true` |
 | `defaultRules.rules.prometheus` | Create Prometheus  default rules| `true` |
+| `defaultRules.rules.time` | Create time default rules | `true` |
 | `defaultRules.labels` | Labels for default rules for monitoring the cluster | `{}` |
 | `defaultRules.annotations` | Annotations for default rules for monitoring the cluster | `{}` |
 | `additionalPrometheusRules` | *DEPRECATED* Will be removed in a future release.  Please use **additionalPrometheusRulesMap** instead.  List of `prometheusRule` objects to create. See https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec. | `[]` |
 | `additionalPrometheusRulesMap` | Map of `prometheusRule` objects to create with the key used as the name of the rule spec. If defined, this will take precedence over `additionalPrometheusRules`. See https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec. | `nil` |
 | `global.rbac.create` | Create RBAC resources | `true` |
 | `global.rbac.pspEnabled` | Create pod security policy resources | `true` |
+| `kubeTargetVersionOverride` | Provide a target gitVersion of K8S, in case .Capabilites.KubeVersion is not available (e.g. `helm template`) |`""`|
 | `global.imagePullSecrets` | Reference to one or more secrets to be used when pulling images | `[]` |
 
 ### Prometheus Operator

staging/prometheus-operator/ci/test-values.yaml

@@ -28,15 +28,18 @@ defaultRules:
     kubeApiserver: true
     kubePrometheusNodeAlerting: true
     kubePrometheusNodeRecording: true
-    kubeScheduler: true
     kubernetesAbsent: true
     kubernetesApps: true
     kubernetesResources: true
     kubernetesStorage: true
     kubernetesSystem: true
+    kubeScheduler: true
+    network: true
     node: true
-    prometheusOperator: true
     prometheus: true
+    prometheusOperator: true
+    time: true
+
   ## Labels for default rules
   labels: {}
   ## Annotations for default rules

staging/prometheus-operator/hack/README.md

@@ -8,15 +8,15 @@ Currently following imported:
 
 - [coreos/kube-prometheus rules set](https://github.com/coreos/kube-prometheus/master/manifests/prometheus-rules.yaml)
   - In order to modify these rules:
-    - prepare and merge PR into [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/rules)
+    - prepare and merge PR into [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/rules) master and/or release branch
     - run import inside your fork of [coreos/kube-prometheus](https://github.com/coreos/kube-prometheus/tree/master)
 
      ```bash
      jb update
      make generate-in-docker
      ```
 
-    - prepare and merge PR with imported changes into coreos/prometheus-operator
+    - prepare and merge PR with imported changes into coreos/kube-prometheus master and/or release branch
     - run sync_prometheus_rules.py inside your fork of this repo
     - send PR with changes to this repo
 - [etcd-io/etc rules set](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/etcd3_alert.rules.yml)
@@ -31,17 +31,17 @@ This script generates grafana dashboards from json files, splitting them to sepa
 
 Currently following imported:
 
-- [coreos/prometheus-operator dashboards](https://github.com/coreos/kube-prometheus/manifests/grafana-deployment.yaml)
+- [coreos/kube-prometheus dashboards](https://github.com/coreos/kube-prometheus/manifests/grafana-deployment.yaml)
   - In order to modify these dashboards:
-    - prepare and merge PR into [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards)
+    - prepare and merge PR into [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/dashboards) master and/or release branch
     - run import inside your fork of [coreos/kube-prometheus](https://github.com/coreos/kube-prometheus/tree/master)
 
      ```bash
      jb update
      make generate-in-docker
      ```
 
-    - prepare and merge PR with imported changes into coreos/prometheus-operator
+    - prepare and merge PR with imported changes into coreos/kube-prometheus master and/or release branch
     - run sync_grafana_dashboards.py inside your fork of this repo
     - send PR with changes to this repo
 - [etcd-io/etc dashboard](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/grafana.json)

staging/prometheus-operator/hack/sync_grafana_dashboards.py

@@ -27,27 +27,51 @@ def new_representer(dumper, data):
 charts = [
     {
         'source': 'https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/grafana-dashboardDefinitions.yaml',
+        'destination': '../templates/grafana/dashboards-1.14',
+        'type': 'yaml',
+        'min_kubernetes': '1.14.0-0'
+    },
+    {
+        'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/grafana.json',
+        'destination': '../templates/grafana/dashboards-1.14',
+        'type': 'json',
+        'min_kubernetes': '1.14.0-0'
+    },
+    {
+        'source': 'https://raw.githubusercontent.com/coreos/kube-prometheus/release-0.1/manifests/grafana-dashboardDefinitions.yaml',
         'destination': '../templates/grafana/dashboards',
         'type': 'yaml',
+        'min_kubernetes': '1.10.0-0',
+        'max_kubernetes': '1.14.0-0'
     },
     {
         'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/grafana.json',
         'destination': '../templates/grafana/dashboards',
         'type': 'json',
+        'min_kubernetes': '1.10.0-0',
+        'max_kubernetes': '1.14.0-0'
     },
 ]
 
 # Additional conditions map
 condition_map = {
     'grafana-coredns-k8s': ' .Values.coreDns.enabled',
     'etcd': ' .Values.kubeEtcd.enabled',
+    'apiserver': ' .Values.kubeApiServer.enabled',
+    'controller-manager': ' .Values.kubeControllerManager.enabled',
+    'kubelet': ' .Values.kubelet.enabled',
+    'proxy': ' .Values.kubeProxy.enabled',
+    'scheduler': ' .Values.kubeScheduler.enabled',
+    'node-rsrc-use': ' .Values.nodeExporter.enabled',
+    'node-cluster-rsrc-use': ' .Values.nodeExporter.enabled'
 }
 
 # standard header
 header = '''# Generated from '%(name)s' from %(url)s
 # Do not change in-place! In order to change this file first read following link:
 # https://github.com/helm/charts/tree/master/stable/prometheus-operator/hack
-{{- if and .Values.grafana.enabled .Values.grafana.defaultDashboardsEnabled%(condition)s }}
+{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
+{{- if and (semverCompare ">=%(min_kubernetes)s" $kubeTargetVersion) (semverCompare "<%(max_kubernetes)s" $kubeTargetVersion) .Values.grafana.enabled .Values.grafana.defaultDashboardsEnabled%(condition)s }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -83,12 +107,14 @@ def yaml_str_repr(struct, indent=2):
     return text
 
 
-def write_group_to_file(resource_name, content, url, destination):
+def write_group_to_file(resource_name, content, url, destination, min_kubernetes, max_kubernetes):
     # initialize header
     lines = header % {
         'name': resource_name,
         'url': url,
         'condition': condition_map.get(resource_name, ''),
+        'min_kubernetes': min_kubernetes,
+        'max_kubernetes': max_kubernetes
     }
 
     filename_struct = {resource_name + '.json': (LiteralStr(content))}
@@ -121,22 +147,23 @@ def main():
             print('Skipping the file, response code %s not equals 200' % response.status_code)
             continue
         raw_text = response.text
+        if ('max_kubernetes' not in chart): chart['max_kubernetes']="9.9.9-9"
         if chart['type'] == 'yaml':
             yaml_text = yaml.load(raw_text)
             groups = yaml_text['items']
             for group in groups:
                 for resource, content in group['data'].items():
-                    write_group_to_file(resource.replace('.json', ''), content, chart['source'], chart['destination'])
+                    write_group_to_file(resource.replace('.json', ''), content, chart['source'], chart['destination'], chart['min_kubernetes'], chart['max_kubernetes'])
         elif chart['type'] == 'json':
             json_text = json.loads(raw_text)
             # is it already a dashboard structure or is it nested (etcd case)?
             flat_structure = bool(json_text.get('annotations'))
             if flat_structure:
                 resource = path.basename(chart['source']).replace('.json', '')
-                write_group_to_file(resource, json.dumps(json_text, indent=4), chart['source'], chart['destination'])
+                write_group_to_file(resource, json.dumps(json_text, indent=4), chart['source'], chart['destination'], chart['min_kubernetes'], chart['max_kubernetes'])
             else:
                 for resource, content in json_text.items():
-                    write_group_to_file(resource.replace('.json', ''), json.dumps(content, indent=4), chart['source'], chart['destination'])
+                    write_group_to_file(resource.replace('.json', ''), json.dumps(content, indent=4), chart['source'], chart['destination'], chart['min_kubernetes'], chart['max_kubernetes'])
     print("Finished")
 
 

staging/prometheus-operator/hack/sync_prometheus_rules.py

@@ -26,11 +26,25 @@ def new_representer(dumper, data):
 charts = [
     {
         'source': 'https://raw.githubusercontent.com/coreos/kube-prometheus/master/manifests/prometheus-rules.yaml',
-        'destination': '../templates/prometheus/rules'
+        'destination': '../templates/prometheus/rules-1.14',
+        'min_kubernetes': '1.14.0-0'
     },
     {
         'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/etcd3_alert.rules.yml',
-        'destination': '../templates/prometheus/rules'
+        'destination': '../templates/prometheus/rules-1.14',
+        'min_kubernetes': '1.14.0-0'
+    },
+    {
+        'source': 'https://raw.githubusercontent.com/coreos/kube-prometheus/release-0.1/manifests/prometheus-rules.yaml',
+        'destination': '../templates/prometheus/rules',
+        'min_kubernetes': '1.10.0-0',
+        'max_kubernetes': '1.14.0-0'
+    },
+    {
+        'source': 'https://raw.githubusercontent.com/etcd-io/etcd/master/Documentation/op-guide/etcd3_alert.rules.yml',
+        'destination': '../templates/prometheus/rules',
+        'min_kubernetes': '1.10.0-0',
+        'max_kubernetes': '1.14.0-0'
     },
 ]
 
@@ -47,7 +61,11 @@ def new_representer(dumper, data):
     'kubernetes-resources': ' .Values.defaultRules.rules.kubernetesResources',
     'kubernetes-storage': ' .Values.defaultRules.rules.kubernetesStorage',
     'kubernetes-system': ' .Values.defaultRules.rules.kubernetesSystem',
+    'node-exporter.rules': ' .Values.nodeExporter.enabled .Values.defaultRules.rules.node',
+    'node-exporter': ' .Values.nodeExporter.enabled .Values.defaultRules.rules.node',
     'node.rules': ' .Values.nodeExporter.enabled .Values.defaultRules.rules.node',
+    'node-network': ' .Values.defaultRules.rules.network',
+    'node-time': ' .Values.defaultRules.rules.time',
     'prometheus-operator': ' .Values.defaultRules.rules.prometheusOperator',
     'prometheus.rules': ' .Values.defaultRules.rules.prometheus',
     'kubernetes-apps': ' .Values.kubeStateMetrics.enabled .Values.defaultRules.rules.kubernetesApps',
@@ -88,7 +106,8 @@ def new_representer(dumper, data):
 header = '''# Generated from '%(name)s' group from %(url)s
 # Do not change in-place! In order to change this file first read following link:
 # https://github.com/helm/charts/tree/master/stable/prometheus-operator/hack
-{{- if and .Values.defaultRules.create%(condition)s }}%(init_line)s
+{{- $kubeTargetVersion := default .Capabilities.KubeVersion.GitVersion .Values.kubeTargetVersionOverride }}
+{{- if and (semverCompare ">=%(min_kubernetes)s" $kubeTargetVersion) (semverCompare "<%(max_kubernetes)s" $kubeTargetVersion) .Values.defaultRules.create%(condition)s }}%(init_line)s
 apiVersion: {{ printf "%%s/v1" (.Values.prometheusOperator.crdApiGroup | default "monitoring.coreos.com") }}
 kind: PrometheusRule
 metadata:
@@ -173,7 +192,7 @@ def add_rules_conditions(rules, indent=4):
     return rules
 
 
-def write_group_to_file(group, url, destination):
+def write_group_to_file(group, url, destination, min_kubernetes, max_kubernetes):
     fix_expr(group['rules'])
 
     # prepare rules string representation
@@ -193,6 +212,8 @@ def write_group_to_file(group, url, destination):
         'url': url,
         'condition': condition_map.get(group['name'], ''),
         'init_line': init_line,
+        'min_kubernetes': min_kubernetes,
+        'max_kubernetes': max_kubernetes
     }
 
     # rules themselves
@@ -225,10 +246,11 @@ def main():
             continue
         raw_text = response.text
         yaml_text = yaml.load(raw_text)
+        if ('max_kubernetes' not in chart): chart['max_kubernetes']="9.9.9-9"
         # etcd workaround, their file don't have spec level
         groups = yaml_text['spec']['groups'] if yaml_text.get('spec') else yaml_text['groups']
         for group in groups:
-            write_group_to_file(group, chart['source'], chart['destination'])
+            write_group_to_file(group, chart['source'], chart['destination'], chart['min_kubernetes'], chart['max_kubernetes'])
     print("Finished")