issue #47 : introduces --skip-crd-schema-miss flag

[emirozer]

Hi,

I made a quick fix for this issue, let me know if you are okay with this approach or not :)

Thx for your time!

[trnl]

Nice one!

[trnl]

@garethr can you take a look?

[bmhatfield]

Friendly bump on this PR - we've just run into exactly this

[Stono]

Pretty please merge this!

[garethr]

Thanks, I think allowing for skipping missing schemas is a good first step towards better support for CRDs. Thanks a lot for taking a first run at this.

Unfortunately I think the flag is misleading here. It flag won't just affect CRDs, but any resource for which the schema isn't loaded. So in the case of say a dropped network connection preventing the loading of a schema this would also allow it to pass.

I think it's also probably worth issuing a warning about the schemas not being validated.

Let me know if you would like to update this PR, otherwise I'm happy to finish off.

[emirozer]

@garethr thanks for the feedback! I have several questions, would appreciate whenever you have time you can give some answers to:

• in order to differentiate between a schema not being available on the target and a network issue, would it be ok to rely on golang.org/x/net/context and like have context of fetching the schema but with a timeout (timeout to cover the possible network issues) ?

• an extra step would be to accept user input kind strings and build a list of them to skip validation for, is this overkill ? (think like --skip-crd-kind-list="SealedSecret, WhatEver")

[garethr]

@emirozer I'm pretty easy on the implementation, so whatever you think is best to the first question. I tend to obsess over the user interface a bit more, as I'd prefer to keep that stable and clear, while the implementation can always evolve under the hood.

I'm fine with step one being a flag which ignores network failures too. Renaming what you have to --ignore-missing-schemas maybe?

Having an explicit list of resources to skip validation would be good as well, but could come later if preferred. I'd suggest changing the name here as well, even if the main usecase might be for CRDs it would work with any resource types. Maybe --skip-kind?

Thanks for taking a look, a number of folks have mentioned this.

[emirozer]

You are definitely much better at naming flags than I am that's for sure haha :D
OK, let me do some thinking and i will get back to you, at the very least I will add warn level log line about the skip and turn the flag name into --ignore-missing-schemas before we merge anything

EDIT: ok so that's done - I will see if I can find a smart/clean way of handling network failures

[alikhil]

It seems like it's ready to merge. @garethr can you take a look please?

[bmhatfield]

Adding another bump - this remains an important workaround that's much needed.

[emirozer]

The further change to determine potential network issues, if I'm not mistaken, seem to be has to handled in gojsonschema dependency. I tried to do a work around but it was quite ugly so i didn't push it. I rebased on master and fixed conflicts. I think we should just merge this maybe..

[mikespokefire]

@emirozer Looks like there has been a regression for us with the latest changes 3 days ago... So given these manifests:

# service.atcloud.io.manifest
apiVersion: atcloud.io/v1
kind: Service
metadata:
  name: solsson-http-echo
  labels:
    app: solsson-http-echo
spec:
  hosts:
  - solsson-http-echo.REDACTED
  description: Simulate app specific failure scenarios
  git-revision: REDACTED
  git-url: REDACTED
  name: Solsson-Http-Echo

# sidecar.networking.istio.io.v1alpha3.manifest
apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: solsson-http-echo
  labels:
    app: solsson-http-echo
spec:
  workloadSelector:
    labels:
      app: solsson-http-echo
  egress:
  - hosts:
    - solsson-http-echo/*

With this kubeval patch prior to 3 days ago we were getting the following correct output:

$ kubeval --ignore-missing-schemas service.atcloud.io.manifest
Skipping missing schema validation!
The document service.atcloud.io.manifest contains a valid Service

$ kubeval --ignore-missing-schemas sidecar.networking.istio.io.v1alpha3.manifest
Skipping missing schema validation!
The document sidecar.networking.istio.io.v1alpha3.manifest contains a valid Sidecar

Now with the latest changes in the tip of this PR we are getting the following output, we should be seeing them as successfully skipped:

$ kubeval --ignore-missing-schemas service.atcloud.io.manifest
Skipping missing schema validation!
1 error occurred:
	* Failed initalizing schema https://kubernetesjsonschema.dev/master-standalone/service-atcloud-v1.json: Could not read schema from HTTP, response status is 404 Not Found

$ kubeval --ignore-missing-schemas sidecar.networking.istio.io.v1alpha3.manifest
Skipping missing schema validation!
1 error occurred:
	* Failed initalizing schema https://kubernetesjsonschema.dev/master-standalone/sidecar-networking-v1alpha3.json: Could not read schema from HTTP, response status is 404 Not Found

[emirozer]

Yep @mikespokefire , thx for the heads up, regression caused by recent changes to master. I will push an update now

[mikespokefire]

@emirozer thanks for that fix, can confirm it fixed that issue, but now we have a slightly different issue as a result of that change. We are now getting some false positives when there should be errors, take the following manifest:

# deployment.manifest
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: solsson-http-echo
  labels:
    app: solsson-http-echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: solsson-http-echo
  template:
    metadata:
      labels:
        app: solsson-http-echo
    spec:
      containers:
      - name: master
        image: REDACTED
        resources:
          limits_memory: 1024Mi
          requests_cpu: 50m
          requests_memory: 32Mi
        ports:
        - name: http-web
          containerPort: 80

When running kubeval --strict deployment.manifest, we get the following error as expected:

$ kubeval --strict deployment.manifest
The document deployment.manifest contains an invalid Deployment
---> limits_memory: Additional property limits_memory is not allowed
---> requests_cpu: Additional property requests_cpu is not allowed
---> requests_memory: Additional property requests_memory is not allowed

However, when running kubeval --strict --ignore-missing-schemas deployment.manifest then everything looks fine when it shouldn't be:

$ kubeval --strict --ignore-missing-schemas deployment.manifest
Warning: Set to ignore missing schemas!
The document deployment.manifest contains a valid Deployment

[jeffg-hpe]

I think your unit tests for this change are also significantly affected by #141

[garethr]

Apologies for taking a while to get to merging this in, lots of travel and new job, etc. I've just added docs and acceptance tests as well and merged to master. I'll get a new version of kubeval released as well with the change.

Thanks to @emirozer for the PR and everyone else for the reviews.

[mikespokefire]

@garethr Thanks for merging this. We are still seeing #127 (comment) as an issue even with the 0.11.0 release. Would you prefer I raise it as a separate issue rather than trying to track it in here?

[ian-howell]

@mikespokefire Your issue is fixed in #150

$ kubeval --strict --ignore-missing-schemas /tmp/deployment.manifest
Warning: Set to ignore missing schemas
The document /tmp/deployment.manifest contains an invalid Deployment
---> limits_memory: Additional property limits_memory is not allowed
---> requests_cpu: Additional property requests_cpu is not allowed
---> requests_memory: Additional property requests_memory is not allowed