Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Validate manifests using kubeval #775

Closed
wants to merge 1 commit into from

Conversation

kakkoyun
Copy link
Member

Signed-off-by: Kemal Akkoyun [email protected]

Add kubeval as a validation method for generated manifests.

Work in progress: Still entertaining the idea. However, I'd like to hear what maintainers think.

@brancz
Copy link
Collaborator

brancz commented Nov 11, 2020

Neat! How does this work with CRDs?

@paulfantom
Copy link
Member

paulfantom commented Nov 12, 2020

How does this work with CRDs?

Sadly it doesn't. There is an open issue in kubeval project to support non-native objects: instrumenta/kubeval#47. This means having kubeval is a good step for fast feedback regarding native objects, but we will still need to load all objects into apiserver and check it this way.

Copy link
Member

@paulfantom paulfantom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idea LGTM

@yannh
Copy link
Contributor

yannh commented Nov 13, 2020

Hi there 👋 I've been working on kubeconform ( https://github.com/yannh/kubeconform/ ) - which is very similar to kubeval, and adds support for CRDs. Maybe I can help?

@kakkoyun
Copy link
Member Author

@yannh That would be awesome. Project needs a good validation strategy since we are moving away from ksonnet. I'll be checking the kubeconform. We could use it for other projects as well. Tell me if I can help in any way.

@yannh
Copy link
Contributor

yannh commented Nov 13, 2020

Mhhh one thing I'm missing is a stable interface to use kubeconform as a library. I'll try to work on that soon.
Edit: I've added an interface that should be easy enough to use.

@yannh
Copy link
Contributor

yannh commented Nov 15, 2020

The approach of Kubeval/Kubeconform to do validation is to use Json Schemas, generated from the OpenAPI Spec files. For Kubeconform to be able to validate Custom Resources, the Json Schemas need to be accessible either via HTTP or the local filesystem. What would be the best approach here - commit the JSON Schemas to this repo? Note that the only tool I have at the moment to convert from OpenAPI to Json Schema is in Python.

@yannh
Copy link
Contributor

yannh commented Nov 15, 2020

Alright, started some work here #785, this is still WIP.
I validated manifests/ using JSON schemas generated from the CRDs here: https://github.com/prometheus-operator/prometheus-operator/tree/master/example/prometheus-operator-crd , in strict mode against schemas for Kubernetes 1.18.0:

$ /home/yann/go/src/github.com/yannh/kubeconform/bin/kubeconform -kubernetes-version 1.18.0 -strict -summary -schema-location 'https://kubernetesjsonschema.dev' -schema-location '/home/yann/go/src/github.com/prometheus-operator/kube-prometheus/manifests/schemas/alertmanager_{{ .ResourceAPIVersion }}.json' manifests/*.yaml
manifests/alertmanager-serviceMonitor.yaml - ServiceMonitor alertmanager is invalid: Additional property endpoints is not allowed - Additional property selector is not allowed
manifests/grafana-serviceMonitor.yaml - ServiceMonitor grafana is invalid: Additional property endpoints is not allowed - Additional property selector is not allowed
manifests/kube-state-metrics-serviceMonitor.yaml - ServiceMonitor kube-state-metrics is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property selector is not allowed
manifests/node-exporter-serviceMonitor.yaml - ServiceMonitor node-exporter is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed - Additional property jobLabel is not allowed
manifests/prometheus-adapter-serviceMonitor.yaml - ServiceMonitor prometheus-adapter is invalid: Additional property endpoints is not allowed - Additional property selector is not allowed
manifests/prometheus-operator-serviceMonitor.yaml - ServiceMonitor prometheus-operator is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed
manifests/prometheus-prometheus.yaml - Prometheus k8s is invalid: Additional property alerting is not allowed - Additional property serviceMonitorSelector is not allowed - Additional property probeNamespaceSelector is not allowed - Additional property ruleSelector is not allowed - Additional property podMonitorSelector is not allowed - Additional property probeSelector is not allowed - Additional property serviceMonitorNamespaceSelector is not allowed - Additional property podMonitorNamespaceSelector is not allowed
manifests/prometheus-serviceMonitorApiserver.yaml - ServiceMonitor kube-apiserver is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed
manifests/prometheus-serviceMonitorCoreDNS.yaml - ServiceMonitor coredns is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed
manifests/prometheus-serviceMonitorKubeControllerManager.yaml - ServiceMonitor kube-controller-manager is invalid: Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed - Additional property endpoints is not allowed
manifests/prometheus-serviceMonitorKubeScheduler.yaml - ServiceMonitor kube-scheduler is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed
manifests/prometheus-serviceMonitor.yaml - ServiceMonitor prometheus is invalid: Additional property selector is not allowed - Additional property endpoints is not allowed
manifests/prometheus-serviceMonitorKubelet.yaml - ServiceMonitor kubelet is invalid: Additional property endpoints is not allowed - Additional property jobLabel is not allowed - Additional property namespaceSelector is not allowed - Additional property selector is not allowed
manifests/prometheus-rules.yaml - PrometheusRule prometheus-k8s-rules is invalid: Additional property groups is not allowed
Summary: 53 resources found in 53 files - Valid: 39, Invalid: 14, Errors: 0, Skipped: 0

@paulfantom
Copy link
Member

manifests/prometheus-prometheus.yaml - Prometheus k8s is invalid: Additional property alerting is not allowed - Additional property serviceMonitorSelector is not allowed - Additional property probeNamespaceSelector is not allowed - Additional property ruleSelector is not allowed - Additional property podMonitorSelector is not allowed - Additional property probeSelector is not allowed - Additional property serviceMonitorNamespaceSelector is not allowed - Additional property podMonitorNamespaceSelector is not allowed

This is not right as those properties are allowed by Prometheus CRD: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusspec

I checked a few others and it seems like either Kubeconform has some false positives in its output or prometheus-operator has invalid schemas. 🤔

PS. @yannh awesome project!

@yannh
Copy link
Contributor

yannh commented Nov 16, 2020

Kubeconform is (I think) rather solid, but my script to convert OpenAPI to JSON schema is a bit brittle :) I will investigate these in details soon.

@yannh
Copy link
Contributor

yannh commented Nov 16, 2020

My command line was wrong, I forgot I hard hardcoded the path to alertmanager_{{ .ResourceAPIVersion }}.json while debugging.
This works now:

$ ~/go/src/github.com/yannh/kubeconform/bin/kubeconform -kubernetes-version 1.18.0 -summary -schema-location 'https://kubernetesjsonschema.dev' -schema-location '/home/yann/go/src/github.com/prometheus-operator/kube-prometheus/manifests/schemas/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' manifests/*.yaml
Summary: 53 resources found in 53 files - Valid: 53, Invalid: 0, Errors: 0, Skipped: 0

@kakkoyun
Copy link
Member Author

Closing this one for the sake of #785. Let's continue the discussion on that PR. Amazing work @yannh, thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants