From a6fb501f48fa175668047f558e100d0f9389d811 Mon Sep 17 00:00:00 2001 From: Kevin Delgado Date: Wed, 11 Jan 2023 17:08:08 +0000 Subject: [PATCH 1/3] Update field validation KEP for 1.27 GA graduation --- .../sig-api-machinery/2885.yaml | 2 + .../README.md | 64 +++++++++---------- .../kep.yaml | 8 +-- 3 files changed, 38 insertions(+), 36 deletions(-) diff --git a/keps/prod-readiness/sig-api-machinery/2885.yaml b/keps/prod-readiness/sig-api-machinery/2885.yaml index 6673b70ae63..2ce6e53d5d8 100644 --- a/keps/prod-readiness/sig-api-machinery/2885.yaml +++ b/keps/prod-readiness/sig-api-machinery/2885.yaml @@ -3,3 +3,5 @@ alpha: approver: "@deads2k" beta: approver: "@deads2k" +stable: + approver: "@deads2k" diff --git a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md index 5c21990c662..d30b6a38ebf 100644 --- a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md +++ b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md @@ -105,6 +105,8 @@ tags, and then generate with `hack/update-toc.sh`. - [Graduation Criteria](#graduation-criteria) - [Alpha](#alpha) - [Beta](#beta) + - [GA](#ga) + - [Version Skew Strategy](#version-skew-strategy) - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) - [Feature Enablement and Rollback](#feature-enablement-and-rollback) - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) @@ -313,11 +315,9 @@ client-side validation, albeit one that is error-prone and not officially supported). Long-term, we want to favor using out-of-tree solutions for client-side -validation, though this idea is still in its infancy. - -The [kubeval](https://www.kubeval.com/) project is an example of an out-of-tree solution that does this, and -we will look into expanding its support of open API to v3, and investigate -whether it makes sense as a permanent solution to client-side validation. +validation. The [kubeconform](https://github.com/yannh/kubeconform) project is an example of an out-of-tree solution that does this, and +we recommend using this or similar tools to validate manifests offline going +forward. ##### Aligning json and yaml errors @@ -615,6 +615,11 @@ It tests the cross product of all valid permutations along the dimensions of: With field validation on by default in beta, we will modify [test/e2e/kubectl/kubectl.go](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubectl/kubectl.go) to ensure that kubectl defaults to using server side field validation and detects unknown/duplicate fields as expected. +[GA] +We will introduce field validation specific e2e/conformance tests to submit +requests directly against the API server for both built-in and custom resources +to test that duplicate and unknown fields are appropriately detected. + ### Graduation Criteria ### Version Skew Strategy -If applicable, how will the component handle version skew with other -components? What are the guarantees? Make sure this is in the test plan. +Following the graduation of server-side field validation to GA, there will be a +period of time when a newer client (which expects server-side field validation locked in GA) will send +requests to an older server that could have server-side field validation +disabled. Until version skew makes it incompatible for a client to hit a server +with field validation disabled, kubectl will need to continue to have +client-side validation available. -Consider the following in developing a version skew strategy for this -enhancement: -- Does this enhancement involve coordinating behavior in the control plane and - in the kubelet? How does an n-2 kubelet without this feature available behave - when this feature is used? -- Will any other components on the node change? For example, changes to CSI, - CRI or CNI may require updating that component before the kubelet. ---> +After this skew (2-3 releases), client-side +validation will be removed from kubectl entirely. (See section on [Out-of-Tree +Alternatives to Client Side +Validation](#out-of-tree-alternatives-to-client-side-validation) for +alternatives to client-side validation) ## Production Readiness Review Questionnaire @@ -749,6 +747,7 @@ Pick one of these and delete the rest. - [x] Feature gate (also fill in values in `kep.yaml`) - Feature gate name: ServerSideFieldValidation - Components depending on the feature gate: kube-apiserver + - Note: feature gate removed upon graduation to GA (1.27) - [x] Other - Describe the mechanism: query parameter - Will enabling / disabling the feature require downtime of the control @@ -851,8 +850,9 @@ Recall that end users cannot usually observe component logs or access metrics. ###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? -Users should expect to see an increase in request latency and memory usage -(~20-25% increase) for requests that desire server side schema validation. +Users should expect to see an increase in request latency (~5% increase) and memory usage +(~10% increase) for requests that desire server side schema validation (json and +yaml only, no change for protobuf). Cluster operators can also use cpu usage monitoring to determine whether increased usage of server-side schema validation dramatically increases CPU usage after feature enablement. diff --git a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml index 3e69f60e4bb..88387ace7c8 100644 --- a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml +++ b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml @@ -18,18 +18,18 @@ see-also: replaces: # The target maturity stage in the current dev cycle for this KEP. -stage: beta +stage: stable # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.25" +latest-milestone: "v1.27" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: alpha: "v1.23" beta: "v1.25" - stable: "v1.26" + stable: "v1.27" # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled @@ -37,7 +37,7 @@ feature-gates: - name: ServerSideFieldValidation components: - kube-apiserver -disable-supported: true +disable-supported: false # The following PRR answers are required at beta release metrics: From b478f8147f649fc922d3c82e35673439e831d022 Mon Sep 17 00:00:00 2001 From: Kevin Delgado Date: Fri, 13 Jan 2023 18:01:00 +0000 Subject: [PATCH 2/3] Address feedback --- .../README.md | 49 +++++++++++++------ .../kep.yaml | 3 +- 2 files changed, 37 insertions(+), 15 deletions(-) diff --git a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md index d30b6a38ebf..1c331ae438f 100644 --- a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md +++ b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md @@ -315,9 +315,13 @@ client-side validation, albeit one that is error-prone and not officially supported). Long-term, we want to favor using out-of-tree solutions for client-side -validation. The [kubeconform](https://github.com/yannh/kubeconform) project is an example of an out-of-tree solution that does this, and -we recommend using this or similar tools to validate manifests offline going -forward. +validation. The [kubeconform](https://github.com/yannh/kubeconform) project is an example of an out-of-tree solution that does this. + +Although likely to never be a perfect representation of the validation that the +server performs, adding better support for these out-of-tree solutions that +better utilize the progress made server-side with regards to performance, +openapi v3, etc is definitely an avenue of future work worth exploring by +sig-api-machinery. ##### Aligning json and yaml errors @@ -669,8 +673,11 @@ Below are some examples to consider, in addition to the aforementioned [maturity #### GA - [x] Wait two releases (1.25 and 1.26) with field validation enabled by default - and receive minimal bug reports pertaining to the feature. -- [] More rigorous e2e/conformance testing added to invoke field validation directly + and receive minimal bug reports pertaining to the feature. As a data point, + GKE clusters running field validation by default in 1.25 are seeing less CPU + and memory overall vs 1.24, supporting the notion that this feature has not + significantly impacted performance. +- [] Add conformance testing to invoke field validation directly against the API server