Introduce validations through CEL for Gateway and GatewayClass. #2226

gauravkghildiyal · 2023-07-24T05:14:19Z

What type of PR is this?

/kind feature

What this PR does / why we need it:
Add support for CEL based validation for Gateway and GatewayClass. This allows us to have more complex validations than those supported by the OpenAPIv3 spec without the need of an additional validation webhook. Upcoming PRs will gradually extend this to HTTPRoutes (and other remaining resources) as well.

References:

Notes for the reviewers:

The tests included in this PR perform error string comparisons. I understand that this may not be the most ideal choice but it helps establish some confidence that the validations are in fact working correctly.
Tests aren't currently being run parallely for easier debugging when reading the test output. The tests itself barely take anytime so we may not have the need to run them parallely at the moment.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Introduce validations through CEL for Gateway and GatewayClass.

howardjohn · 2023-07-24T15:27:16Z

config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml

@@ -78,6 +78,9 @@ spec:
                minLength: 1
                pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$
                type: string
+                x-kubernetes-validations:


Isn't this stuff only on by default in k8s 1.25+? Is that the new min version, or is this silently ignored on old versions?

Gateway API already promises to support the trailing 5 k8s versions, so I believe we'd need to release a second set of CRDs for k8s <=1.24 as long as we support those versions. Based on the k8s release cycle, I think that's going to be at least another ~6 months.

It is annoying to maintain yet another set of CRDs, but on the other hand, I think a significant amount of usage is moving towards 1.25+, and there's a chance we may be able to entirely remove the webhook on those versions in favor of CEL, which would be a significant win.

I wish we could get away with saying v1.25 is the minimum supported K8s version for GA.

Before we commit to double CRDs, can someone confirm its not gracefully handled on old versions? I assume not, but worth checking

Thanks for bringing this up!

Yes so I did check things with 1.23 and I didn't find any problem. To be specific, I was able to create these CRDs which have this new x-kubernetes-validations field. It got ignored.

@shaneutt at a minimum I'd like to make CEL the default installation method and then add a note like "if you're running Kubernetes 1.24 or older, installing the webhook is recommended as well".

@gauravkghildiyal that's great that we might be able to continue to use the same sets of CRDs for this!

Chatted with @jpbetz, it looks like CRDs with x-kubernetes-validations will go through starting in 1.23 thanks to kubernetes/kubernetes#108030. The actual validation will be limited to 1.25+ (unless clusters have had CEL manually enabled earlier).

So that means that once 1.28 is out, we will be both mandating an minimum of 1.23 for the CRDs that include x-kubernetes-validations and still in compliance with our "5 versions" support policy.

Including another set of CRDs for older versions seems like it would be more work that it's worth then.

shaneutt · 2023-07-24T22:43:10Z

(just a note that this has some implications on #1514)

youngnick

LGTM with some small changes to remove "should" in validation messages (to me, failing validation implies "must" :) )

apis/v1beta1/gateway_types.go

robscott

Thanks @gauravkghildiyal!

/approve

robscott · 2023-07-26T18:42:26Z

hack/run-cel-validation-test.sh

+# TODO(gauravkghildiyal): Make this a "verify" script (so that it runs along
+# with the verification presubmit) once we are confident about these tests.


@gauravkghildiyal Can you make an issue to track this so we don't lose track of it?

Sure I created #2239 to track

robscott · 2023-07-26T18:55:53Z

cel-validation-test/gateway_test.go

@@ -0,0 +1,470 @@
+/*


Tiny nit: I'm not really sure where this should live, but I'd rather avoid another top-level directory. I agree that we should probably keep it away from the other validation tests in /apis since we probably want to try to make /apis as targeted/focused as possible.

Some other options:

/hack/cel-validation: Most of our presubmits/things resembling tests live in /hack right now, which doesn't feel perfect, but it would at least live alongside the rest of the presubmit logic.

/pkg/test/cel: I think it's likely that we'll have additional go tests over time, this feels like it would be a fairly natural place for them to live.

At this point I think I'm leaning towards 2 here, but open to alternatives.

Yeah, this is a great point.

I see that the PR has been updated already with moving everything into /hack, but tbh I think that it's better if we put the Go code in /pkg/test/cel like @robscott suggested.

Oh definitely this should be changed. I was hoping to get feedback regarding this so thanks a lot for the suggestion.

I've choosen option /hack/cel-validation. (Because we have unit tests within pkg and it wasn't very neat since we run our unit tests using go test ./pkg/... which would then also run our e2e tests. This further suggests that perhaps that was not the right choice)

Oh, that's also a good point - this is good enough for now, and can be changed later. Thanks.

Whoops I noticed our comments collided there :)
I kinda totally get why hack/cel-validation may not seem ideal, but yeah we could definitely change it later

shaneutt

I think it's great that we're adding this, appreciate the hard work on this.

I'm pretty much ready to say let's go, as things look generally good:

/approve

Just to be clear, we've manually tested this as well right? I know we've added some automated tests, but we've performed some manual deployments of CRDs and exercised things a bit on an implementation using those CRDs, e.t.c.?

k8s-ci-robot · 2023-07-26T19:00:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gauravkghildiyal, robscott, shaneutt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [robscott,shaneutt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gauravkghildiyal · 2023-07-27T03:48:48Z

Just to be clear, we've manually tested this as well right? I know we've added some automated tests, but we've performed some manual deployments of CRDs and exercised things a bit on an implementation using those CRDs, e.t.c.?

That's a very valid concern and I completely share the sentiment. Yes I have manually tested it to an extent. But yes this could be critical and all the testing we could get here is welcomed. (I'll continue testing on the side whenever I can and quickly revert back if something seems off)

youngnick · 2023-07-27T03:50:42Z

I'm happy to LGTM this, but I'll also hold so that we can decide if we want to merge now or after v0.8.0.

/lgtm
/hold

robscott

Thanks @gauravkghildiyal! A couple more things and then I think we're good to merge this.

robscott · 2023-07-27T16:12:01Z

hack/cel-validation/gateway_test.go

+			wantErrors: []string{"certificateRefs must be set and not empty when TLSModeType is Terminate"},
+		},
+		{
+			desc: "certificatedRefs set with tls protocol and TLS terminate mode",


Suggested change

desc: "certificatedRefs set with tls protocol and TLS terminate mode",

desc: "certificateRefs set with tls protocol and TLS terminate mode",

robscott · 2023-07-27T16:29:57Z

hack/cel-validation/gateway_test.go

+						// TODO(gauravkghildiyal): Figure out a sensible way to check
+						// validity of IP addresses. Admission webhook uses golang IP
+						// parsing which may not be directly translateable to regex matching
+						// in CEL. At the moment, this value will not result in an error.


I think we should resolve this prior to merging + add more valid and invalid test cases around this. As far as I can tell, our only option here is regex, this one appears to work well, but open to alternatives: https://regex101.com/r/Kzijkp/1.

robscott · 2023-07-27T20:29:34Z

Chatted with @shaneutt and I think we're good to merge this. @gauravkghildiyal's pretty busy right now so I'm going to file a follow up PR shortly that addresses the last bits of feedback I have.

/hold cancel

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 24, 2023

k8s-ci-robot requested review from robscott and shaneutt July 24, 2023 05:14

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jul 24, 2023

howardjohn reviewed Jul 24, 2023

View reviewed changes

gauravkghildiyal force-pushed the cel-validation branch 2 times, most recently from 714d560 to a1bd3fc Compare July 26, 2023 03:22

youngnick reviewed Jul 26, 2023

View reviewed changes

apis/v1beta1/gateway_types.go Outdated Show resolved Hide resolved

apis/v1beta1/gateway_types.go Outdated Show resolved Hide resolved

gauravkghildiyal added 2 commits July 26, 2023 10:57

Add CEL validation for Gateway and GatewayClass

301765e

Run 'make generate'

81d0210

gauravkghildiyal force-pushed the cel-validation branch from a1bd3fc to 3b8e2a1 Compare July 26, 2023 18:03

robscott reviewed Jul 26, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 26, 2023

shaneutt approved these changes Jul 26, 2023

View reviewed changes

gauravkghildiyal force-pushed the cel-validation branch from 3b8e2a1 to 2d8ce7e Compare July 27, 2023 03:14

Add tests to verify CEL validation for Gateway and GatewayClass

2210ded

gauravkghildiyal force-pushed the cel-validation branch from 2d8ce7e to 2210ded Compare July 27, 2023 03:32

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 27, 2023

k8s-ci-robot assigned youngnick Jul 27, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 27, 2023

robscott reviewed Jul 27, 2023

View reviewed changes

robscott mentioned this pull request Jul 27, 2023

v0.8.0 API Review #2245

Closed

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 27, 2023

k8s-ci-robot merged commit 3ae9fa7 into kubernetes-sigs:main Jul 27, 2023

robscott mentioned this pull request Jul 27, 2023

Completing CEL validation for Gateway #2246

Merged

gauravkghildiyal deleted the cel-validation branch July 30, 2023 02:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce validations through CEL for Gateway and GatewayClass. #2226

Introduce validations through CEL for Gateway and GatewayClass. #2226

gauravkghildiyal commented Jul 24, 2023

howardjohn Jul 24, 2023

robscott Jul 24, 2023

shaneutt Jul 24, 2023

howardjohn Jul 24, 2023

gauravkghildiyal Jul 24, 2023

robscott Jul 24, 2023

robscott Jul 24, 2023

youngnick Jul 26, 2023

shaneutt commented Jul 24, 2023

youngnick left a comment

robscott left a comment

robscott Jul 26, 2023

gauravkghildiyal Jul 27, 2023

robscott Jul 26, 2023

youngnick Jul 27, 2023

gauravkghildiyal Jul 27, 2023

youngnick Jul 27, 2023

gauravkghildiyal Jul 27, 2023 •

edited

Loading

shaneutt left a comment

k8s-ci-robot commented Jul 26, 2023

gauravkghildiyal commented Jul 27, 2023

youngnick commented Jul 27, 2023

robscott left a comment

robscott Jul 27, 2023

robscott Jul 27, 2023

robscott commented Jul 27, 2023

		# TODO(gauravkghildiyal): Make this a "verify" script (so that it runs along
		# with the verification presubmit) once we are confident about these tests.

	desc: "certificatedRefs set with tls protocol and TLS terminate mode",
	desc: "certificateRefs set with tls protocol and TLS terminate mode",

Introduce validations through CEL for Gateway and GatewayClass. #2226

Introduce validations through CEL for Gateway and GatewayClass. #2226

Conversation

gauravkghildiyal commented Jul 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaneutt commented Jul 24, 2023

youngnick left a comment

Choose a reason for hiding this comment

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gauravkghildiyal Jul 27, 2023 • edited Loading

Choose a reason for hiding this comment

shaneutt left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 26, 2023

gauravkghildiyal commented Jul 27, 2023

youngnick commented Jul 27, 2023

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robscott commented Jul 27, 2023

gauravkghildiyal Jul 27, 2023 •

edited

Loading