WIP: Validate manifests using kubeval

[kakkoyun]

Signed-off-by: Kemal Akkoyun [email protected]

Add kubeval as a validation method for generated manifests.

Work in progress: Still entertaining the idea. However, I'd like to hear what maintainers think.

[brancz]

Neat! How does this work with CRDs?

[paulfantom]

How does this work with CRDs?

Sadly it doesn't. There is an open issue in kubeval project to support non-native objects: instrumenta/kubeval#47. This means having kubeval is a good step for fast feedback regarding native objects, but we will still need to load all objects into apiserver and check it this way.

[paulfantom]

Idea LGTM

[yannh]

Hi there 👋 I've been working on kubeconform ( https://github.com/yannh/kubeconform/ ) - which is very similar to kubeval, and adds support for CRDs. Maybe I can help?

[kakkoyun]

@yannh That would be awesome. Project needs a good validation strategy since we are moving away from ksonnet. I'll be checking the kubeconform. We could use it for other projects as well. Tell me if I can help in any way.

[yannh]

Mhhh one thing I'm missing is a stable interface to use kubeconform as a library. I'll try to work on that soon.
Edit: I've added an interface that should be easy enough to use.

[yannh]

The approach of Kubeval/Kubeconform to do validation is to use Json Schemas, generated from the OpenAPI Spec files. For Kubeconform to be able to validate Custom Resources, the Json Schemas need to be accessible either via HTTP or the local filesystem. What would be the best approach here - commit the JSON Schemas to this repo? Note that the only tool I have at the moment to convert from OpenAPI to Json Schema is in Python.

[yannh]

Alright, started some work here #785, this is still WIP.
I validated manifests/ using JSON schemas generated from the CRDs here: https://github.com/prometheus-operator/prometheus-operator/tree/master/example/prometheus-operator-crd , in strict mode against schemas for Kubernetes 1.18.0:

$ /home/yann/go/src/github.com/yannh/kubeconform/bin/kubeconform -kubernetes-version 1.18.0 -strict -summary -schema-location 'https://kubernetesjsonschema.dev' -schema-location '/home/yann/go/src/github.com/prometheus-operator/kube-prometheus/manifests/schemas/alertmanager_{{ .ResourceAPIVersion }}.json' manifests/*.yaml
manifests/alertmanager-serviceMonitor.yaml - ServiceMonitor alertmanager is invalid: Additional property endpoints is not allowed - Additional property selector is not allowed
manifests/grafana-serviceMonitor.yaml - ServiceMonitor grafana is invalid: Additional property endpoints is not allowed - Additional property selector is not allowed
manifests/kube-state-metrics-serviceMonitor.yaml - ServiceMonitor kube-state-metrics is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property selector is not allowed
manifests/node-exporter-serviceMonitor.yaml - ServiceMonitor node-exporter is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed - Additional property jobLabel is not allowed
manifests/prometheus-adapter-serviceMonitor.yaml - ServiceMonitor prometheus-adapter is invalid: Additional property endpoints is not allowed - Additional property selector is not allowed
manifests/prometheus-operator-serviceMonitor.yaml - ServiceMonitor prometheus-operator is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed
manifests/prometheus-prometheus.yaml - Prometheus k8s is invalid: Additional property alerting is not allowed - Additional property serviceMonitorSelector is not allowed - Additional property probeNamespaceSelector is not allowed - Additional property ruleSelector is not allowed - Additional property podMonitorSelector is not allowed - Additional property probeSelector is not allowed - Additional property serviceMonitorNamespaceSelector is not allowed - Additional property podMonitorNamespaceSelector is not allowed
manifests/prometheus-serviceMonitorApiserver.yaml - ServiceMonitor kube-apiserver is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed
manifests/prometheus-serviceMonitorCoreDNS.yaml - ServiceMonitor coredns is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed
manifests/prometheus-serviceMonitorKubeControllerManager.yaml - ServiceMonitor kube-controller-manager is invalid: Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed - Additional property endpoints is not allowed
manifests/prometheus-serviceMonitorKubeScheduler.yaml - ServiceMonitor kube-scheduler is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed
manifests/prometheus-serviceMonitor.yaml - ServiceMonitor prometheus is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed
manifests/prometheus-serviceMonitorKubelet.yaml - ServiceMonitor kubelet is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed
manifests/prometheus-rules.yaml - PrometheusRule prometheus-k8s-rules is invalid: Additional property groups is not allowed
Summary: 53 resources found in 53 files - Valid: 39, Invalid: 14, Errors: 0, Skipped: 0

[paulfantom]

manifests/prometheus-prometheus.yaml - Prometheus k8s is invalid: Additional property alerting is not allowed - Additional property serviceMonitorSelector is not allowed - Additional property probeNamespaceSelector is not allowed - Additional property ruleSelector is not allowed - Additional property podMonitorSelector is not allowed - Additional property probeSelector is not allowed - Additional property serviceMonitorNamespaceSelector is not allowed - Additional property podMonitorNamespaceSelector is not allowed

This is not right as those properties are allowed by Prometheus CRD: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusspec

I checked a few others and it seems like either Kubeconform has some false positives in its output or prometheus-operator has invalid schemas. 🤔

PS. @yannh awesome project!

[yannh]

Kubeconform is (I think) rather solid, but my script to convert OpenAPI to JSON schema is a bit brittle :) I will investigate these in details soon.

[yannh]

My command line was wrong, I forgot I hard hardcoded the path to alertmanager_{{ .ResourceAPIVersion }}.json while debugging.
This works now:

$ ~/go/src/github.com/yannh/kubeconform/bin/kubeconform -kubernetes-version 1.18.0 -summary -schema-location 'https://kubernetesjsonschema.dev' -schema-location '/home/yann/go/src/github.com/prometheus-operator/kube-prometheus/manifests/schemas/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' manifests/*.yaml
Summary: 53 resources found in 53 files - Valid: 53, Invalid: 0, Errors: 0, Skipped: 0

[kakkoyun]

Closing this one for the sake of #785. Let's continue the discussion on that PR. Amazing work @yannh, thanks a lot.