feat: enable customresource metrics by configuration

docs/README.md

@@ -78,6 +78,10 @@ sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod,
   * on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1)
 ```
 
+## Metrics from Custom Resources
+
+See [Custom Resource State Metrics](customresourcestate-metrics.md) for experimental support for custom resources.
+
 ## CLI Arguments
 
 Additionally, options for `kube-state-metrics` can be passed when executing as a CLI, or in a kubernetes / openshift environment. More information can be found here: [CLI Arguments](cli-arguments.md)

docs/cli-arguments.md

@@ -25,40 +25,42 @@ spec:
 ```txt
 $ kube-state-metrics -h
 Usage of ./kube-state-metrics:
-      --add_dir_header                        If true, adds the file directory to the header of the log messages
-      --alsologtostderr                       log to standard error as well as files
-      --apiserver string                      The URL of the apiserver to use as a master
-      --enable-gzip-encoding                  Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.
-  -h, --help                                  Print Help text
-      --host string                           Host to expose metrics on. (default "::")
-      --kubeconfig string                     Absolute path to the kubeconfig file
-      --log_backtrace_at traceLocation        when logging hits line file:N, emit a stack trace (default :0)
-      --log_dir string                        If non-empty, write log files in this directory
-      --log_file string                       If non-empty, use this log file
-      --log_file_max_size uint                Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
-      --logtostderr                           log to standard error instead of files (default true)
-      --metric-allowlist string               Comma-separated list of metrics to be exposed. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
-      --metric-annotations-allowlist string   Comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '*' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[*]').
-      --metric-denylist string                Comma-separated list of metrics not to be enabled. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
-      --metric-labels-allowlist string        Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '*' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[*]').
-      --metric-opt-in-list string             Comma-separated list of metrics which are opt-in and not enabled by default. This is in addition to the metric allow- and denylists
-      --namespaces string                     Comma-separated list of namespaces to be enabled. Defaults to ""
-      --namespaces-denylist string            Comma-separated list of namespaces not to be enabled. If namespaces and namespaces-denylist are both set, only namespaces that are excluded in namespaces-denylist will be used.
-      --one_output                            If true, only write logs to their native severity level (vs also writing to each lower severity level)
-      --pod string                            Name of the pod that contains the kube-state-metrics container. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
-      --pod-namespace string                  Name of the namespace of the pod specified by --pod. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
-      --port int                              Port to expose metrics on. (default 8080)
-      --resources string                      Comma-separated list of Resources to be enabled. Defaults to "certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
-      --shard int32                           The instances shard nominal (zero indexed) within the total number of shards. (default 0)
-      --skip_headers                          If true, avoid header prefixes in the log messages
-      --skip_log_headers                      If true, avoid headers when opening log files
-      --stderrthreshold severity              logs at or above this threshold go to stderr (default 2)
-      --telemetry-host string                 Host to expose kube-state-metrics self metrics on. (default "::")
-      --telemetry-port int                    Port to expose kube-state-metrics self metrics on. (default 8081)
-      --tls-config string                     Path to the TLS configuration file
-      --total-shards int                      The total number of shards. Sharding is disabled when total shards is set to 1. (default 1)
-      --use-apiserver-cache                   Sets resourceVersion=0 for ListWatch requests, using cached resources from the apiserver instead of an etcd quorum read.
-  -v, --v Level                               number for the log level verbosity
-      --version                               kube-state-metrics build version information
-      --vmodule moduleSpec                    comma-separated list of pattern=N settings for file-filtered logging
+      --add_dir_header                             If true, adds the file directory to the header of the log messages
+      --alsologtostderr                            log to standard error as well as files
+      --apiserver string                           The URL of the apiserver to use as a master
+      --custom-resource-state-config string        Inline Custom Resource State Metrics config YAML (experimental)
+      --custom-resource-state-config-file string   Path to a Custom Resource State Metrics config file (experimental)
+      --enable-gzip-encoding                       Gzip responses when requested by clients via 'Accept-Encoding: gzip' header.
+  -h, --help                                       Print Help text
+      --host string                                Host to expose metrics on. (default "::")
+      --kubeconfig string                          Absolute path to the kubeconfig file
+      --log_backtrace_at traceLocation             when logging hits line file:N, emit a stack trace (default :0)
+      --log_dir string                             If non-empty, write log files in this directory
+      --log_file string                            If non-empty, use this log file
+      --log_file_max_size uint                     Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
+      --logtostderr                                log to standard error instead of files (default true)
+      --metric-allowlist string                    Comma-separated list of metrics to be exposed. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
+      --metric-annotations-allowlist string        Comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them (Example: '=namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...)'. A single '*' can be provided per resource instead to allow any annotations, but that has severe performance implications (Example: '=pods=[*]').
+      --metric-denylist string                     Comma-separated list of metrics not to be enabled. This list comprises of exact metric names and/or regex patterns. The allowlist and denylist are mutually exclusive.
+      --metric-labels-allowlist string             Comma-separated list of additional Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them (Example: '=namespaces=[k8s-label-1,k8s-label-n,...],pods=[app],...)'. A single '*' can be provided per resource instead to allow any labels, but that has severe performance implications (Example: '=pods=[*]').
+      --metric-opt-in-list string                  Comma-separated list of metrics which are opt-in and not enabled by default. This is in addition to the metric allow- and denylists
+      --namespaces string                          Comma-separated list of namespaces to be enabled. Defaults to ""
+      --namespaces-denylist string                 Comma-separated list of namespaces not to be enabled. If namespaces and namespaces-denylist are both set, only namespaces that are excluded in namespaces-denylist will be used.
+      --one_output                                 If true, only write logs to their native severity level (vs also writing to each lower severity level)
+      --pod string                                 Name of the pod that contains the kube-state-metrics container. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
+      --pod-namespace string                       Name of the namespace of the pod specified by --pod. When set, it is expected that --pod and --pod-namespace are both set. Most likely this should be passed via the downward API. This is used for auto-detecting sharding. If set, this has preference over statically configured sharding. This is experimental, it may be removed without notice.
+      --port int                                   Port to expose metrics on. (default 8080)
+      --resources string                           Comma-separated list of Resources to be enabled. Defaults to "certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments"
+      --shard int32                                The instances shard nominal (zero indexed) within the total number of shards. (default 0)
+      --skip_headers                               If true, avoid header prefixes in the log messages
+      --skip_log_headers                           If true, avoid headers when opening log files
+      --stderrthreshold severity                   logs at or above this threshold go to stderr (default 2)
+      --telemetry-host string                      Host to expose kube-state-metrics self metrics on. (default "::")
+      --telemetry-port int                         Port to expose kube-state-metrics self metrics on. (default 8081)
+      --tls-config string                          Path to the TLS configuration file
+      --total-shards int                           The total number of shards. Sharding is disabled when total shards is set to 1. (default 1)
+      --use-apiserver-cache                        Sets resourceVersion=0 for ListWatch requests, using cached resources from the apiserver instead of an etcd quorum read.
+  -v, --v Level                                    number for the log level verbosity
+      --version                                    kube-state-metrics build version information
+      --vmodule moduleSpec                         comma-separated list of pattern=N settings for file-filtered logging
 ```

docs/customresourcestate-metrics.md

@@ -0,0 +1,223 @@
+# Custom Resource State Metrics
+
+This section describes how to add metrics based on the state of a custom resource without writing a custom resource 
+registry and running your own build of KSM.
+
+## Configuration
+
+A YAML configuration file described below is required to define your custom resources and the fields to turn into metrics.
+
+Two flags can be used:
+
+ * `--custom-resource-state-config "inline yaml (see example)"` or
+ * `--custom-resource-state-config-file /path/to/config.yaml`
+
+If both flags are provided, the inline configuration will take precedence.
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: kube-state-metrics
+  namespace: kube-system
+spec:
+  template:
+    spec:
+      containers:
+      - name: kube-state-metrics
+        args:
+          - --custom-resource-state-config
+          # in YAML files, | allows a multi-line string to be passed as a flag value
+          # see https://yaml-multiline.info
+          -  |
+              spec:
+                resources:
+                  - groupVersionKind:
+                      group: myteam.io
+                      version: "v1"
+                      kind: Foo
+                    metrics:
+                      - name: active_count
+                        help: "Count of active Foo"
+                        ...
+```
+
+### Examples
+
+The examples in this section will use the following custom resource:
+
+```yaml
+kind: Foo
+apiVersion: myteam.io/vl
+metadata:
+    annotations:
+        bar: baz
+        qux: quxx
+    labels:
+        foo: bar
+    name: foo
+spec:
+    order:
+        - id: 1
+          value: true
+        - id: 3
+          value: false
+    replicas: 1
+status:
+    active:
+        type-a: 1
+        type-b: 3
+    conditions:
+        - name: a
+          value: 45
+        - name: b
+          value: 66
+    sub:
+        type-a:
+            active: 1
+            ready: 2
+        type-b:
+            active: 3
+            ready: 4
+    uptime: 43.21
+```
+
+#### Single Values
+
+The config:
+
+```yaml
+kind: CustomResourceStateMetrics
+spec:
+  resources:
+    - groupVersionKind:
+        group: myteam.io
+        kind: "Foo"
+        version: "v1"
+      metrics:
+        - name: "uptime"
+          help: "Foo uptime"
+          each:
+            path: [status, uptime]
+```
+
+Produces the metric:
+
+```prometheus
+kube_myteam_io_v1_Foo_uptime 43.21
+```
+
+#### Multiple Metrics/Kitchen Sink
+
+```yaml
+kind: CustomResourceStateMetrics
+spec:
+  resources:
+    - groupVersionKind:
+        group: myteam.io
+        kind: "Foo"
+        version: "v1"
+      # labels can be added to all metrics from a resource
+      commonLabels:
+        crd_type: "foo"
+      labelsFromPath:
+        name: [metadata, name]
+      metrics:
+        - name: "ready_count"
+          help: "Number Foo Bars ready"
+          each:
+            # targeting an object or array will produce a metric for each element
+            # labelsFromPath and value are relative to this path
+            path: [status, sub]
+
+            # if path targets an object, the object key will be used as label value
+            labelFromKey: type
+            # label values can be resolved specific to this path 
+            labelsFromPath:
+              active: [active]
+            # The actual field to use as metric value. Should be a number.
+            value: [ready]
+          commonLabels:
+            custom_metric: "yes"
+          labelsFromPath:
+            # whole objects may be copied into labels by prefixing with "*"
+            # *anything will be copied into labels, with the highest sorted * strings first
+            "*": [metadata, labels]
+            "**": [metadata, annotations]
+
+            # or specific fields may be copied. these fields will always override values from *s
+            name: [metadata, name]
+            foo: [metadata, labels, foo]
+```
+
+Produces the following metrics:
+
+```prometheus
+kube_myteam_io_v1_Foo_active_count{active="1",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-a"} 1
+kube_myteam_io_v1_Foo_active_count{active="3",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-b"} 3
+```
+
+### Naming
+
+The default metric names are prefixed to avoid collisions with other metrics.
+By default, a namespace of `kube` and a subsystem based on your custom resource's group+version+kind is used.
+You can override these with the namespace and subsystem fields.
+
+```yaml
+kind: CustomResourceStateMetrics
+spec:
+  resources:
+    - groupVersionKind: ...
+      namespace: myteam
+      subsystem: foos
+      metrics:
+        - name: uptime
+          ...
+```
+
+Produces:
+```prometheus
+myteam_foos_uptime 43.21
+```
+
+To omit namespace and/or subsystem altogether, set them to `_`.
+
+### Logging
+
+If a metric path is registered but not found on a custom resource, an error will be logged. For some resources,
+this may produce a lot of noise. The error log [verbosity][vlog] for a metric or resource can be set with `errorLogV` on
+the resource or metric:
+
+```yaml
+kind: CustomResourceStateMetrics
+spec:
+  resources:
+    - groupVersionKind: ...
+      errorLogV: 0  # 0 = default for errors
+      metrics:
+        - name: uptime
+          errorLogV: 10  # only log at high verbosity
+```
+
+[vlog]: https://github.com/go-logr/logr#why-v-levels
+
+### Path Syntax
+
+Paths are specified as a list of strings. Each string is a path segment, resolved dynamically against the data of the custom resource.
+If any part of a path is missing, the result is nil.
+
+Examples:
+
+```yaml
+# simple path lookup
+[spec, replicas]                         # spec.replicas == 1
+
+# indexing an array
+[spec, order, "0", value]                # spec.order[0].value = true
+
+# finding an element in a list by key=value  
+[status, conditions, "[name=a]", value]  # status.conditions[0].value = 45
+
+# if the value to be matched is a number or boolean, the value is compared as a number or boolean  
+[status, conditions, "[value=66]", name]  # status.conditions[1].name = "b"
+```