From 6255030edf5a14119822760400aa87074d76a811 Mon Sep 17 00:00:00 2001 From: Cole Mickens Date: Wed, 31 Jan 2018 14:39:49 -0800 Subject: [PATCH] respond to feedback, bring in alignment with PodNodeSelector docs --- .../en/docs/admin/admission-controllers.md | 204 +++++++++++------- 1 file changed, 122 insertions(+), 82 deletions(-) diff --git a/content/en/docs/admin/admission-controllers.md b/content/en/docs/admin/admission-controllers.md index fad8438005fc0..8dace1c9bd3e9 100644 --- a/content/en/docs/admin/admission-controllers.md +++ b/content/en/docs/admin/admission-controllers.md @@ -1,5 +1,5 @@ --- -reviewers: +approvers: - lavalamp - davidopp - derekwaynecarr @@ -9,7 +9,8 @@ reviewers: title: Using Admission Controllers --- -{{< toc >}} +* TOC +{:toc} ## What are they? @@ -21,7 +22,7 @@ is authenticated and authorized. The controllers consist of the administrator. In that list, there are two special controllers: MutatingAdmissionWebhook and ValidatingAdmissionWebhook. These execute the mutating and validating (respectively) [admission control -webhooks](/docs/admin/extensible-admission-controllers/#external-admission-webhooks) +webhooks](/docs/admin/extensible-admission-controllers.md#external-admission-webhooks) which are configured in the API. Admission controllers may be "validating", "mutating", or both. Mutating @@ -30,7 +31,8 @@ controllers may modify the objects they admit; validating controllers may not. The admission control process proceeds in two phases. In the first phase, mutating admission controllers are run. In the second phase, validating admission controllers are run. Note again that some of the controllers are -both. +both. In both phases, the controllers are run in the order specified by the +`--admission-control` flag of `kube-apiserver`. If any of the controllers in either phase reject the request, the entire request is rejected immediately and an error is returned to the end-user. @@ -52,35 +54,27 @@ support all the features you expect. ## How do I turn on an admission controller? -The Kubernetes API server flag `enable-admission-plugins` takes a comma-delimited list of admission control plugins to invoke prior to modifying objects in the cluster. -For example, the following command line enables the `NamespaceLifecycle` and the `LimitRanger` -admission control plugins: +The Kubernetes API server supports a flag, `admission-control` that takes a comma-delimited, +ordered list of admission control choices to invoke prior to modifying objects in the cluster. +For example, the following command line turns on the `NamespaceLifecycle` and the `LimitRanger` +admission controller: ```shell -kube-apiserver --enable-admission-plugins=NamespaceLifecyle,LimitRanger ... +kube-apiserver --admission-control=NamespaceLifecyle,LimitRanger ... ``` -{{< note >}} **Note**: Depending on the way your Kubernetes cluster is deployed and how the API server is started, you may need to apply the settings in different ways. For example, you may have to modify the systemd unit file if the API server is deployed as a systemd service, you may modify the manifest file for the API server if Kubernetes is deployed in a self-hosted way. -{{< /note >}} - -## How do I turn off an admission controller? - -The Kubernetes API server flag `disable-admission-plugins` takes a comma-delimited list of admission control plugins to be disabled, even if they are in the list of plugins enabled by default. - -```shell -kube-apiserver --disable-admission-plugins=PodNodeSelector,AlwaysDeny ... -``` +{: .note} ## What does each admission controller do? -### AlwaysAdmit (DEPRECATED) +### AlwaysAdmit -Use this admission controller by itself to pass-through all requests. AlwaysAdmit is DEPRECATED as no real meaning. +Use this admission controller by itself to pass-through all requests. ### AlwaysPullImages @@ -92,9 +86,9 @@ scheduled onto the right node), without any authorization check against the imag is enabled, images are always pulled prior to starting containers, which means valid credentials are required. -### AlwaysDeny (DEPRECATED) +### AlwaysDeny -Rejects all requests. AlwaysDeny is DEPRECATED as no real meaning. +Rejects all requests. Used for testing. ### DefaultStorageClass @@ -140,9 +134,11 @@ enabling this admission controller. ### EventRateLimit (alpha) -This admission controller mitigates the problem where the API server gets flooded by +This admission controller is introduced in v1.9 to mitigate the problem where the API server gets flooded by event requests. The cluster admin can specify event rate limits by: + * Ensuring that `eventratelimit.admission.k8s.io/v1alpha1=true` is included in the + `--runtime-config` flag for the API server; * Enabling the `EventRateLimit` admission controller; * Referencing an `EventRateLimit` configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: @@ -184,16 +180,19 @@ for more details. ### ExtendedResourceToleration -This plug-in facilitates creation of dedicated nodes with extended resources. +This plug-in is introduced in v1.9 to facilitate creation of dedicated nodes with extended resources. If operators want to create dedicated nodes with extended resources (like GPUs, FPGAs etc.), they are expected to -[taint the node](/docs/concepts/configuration/taint-and-toleration/#example-use-cases) with the extended resource -name as the key. This admission controller, if enabled, automatically +taint the node with the extended resource name as the key. This admission controller, if enabled, automatically adds tolerations for such taints to pods requesting extended resources, so users don't have to manually add these tolerations. ### ImagePolicyWebhook -The ImagePolicyWebhook admission controller allows a backend webhook to make admission decisions. +The ImagePolicyWebhook admission controller allows a backend webhook to make admission decisions. You enable this admission controller by setting the admission-control option as follows: + +```shell +--admission-control=ImagePolicyWebhook +``` #### Configuration File Format @@ -224,8 +223,7 @@ plugins: ... ``` -The ImagePolicyWebhook config file must reference a [kubeconfig](docs/tasks/access-application-cluster/configure-access-multiple-clusters/) -formatted file which sets up the connection to the backend. It is required that the backend communicate over TLS. +The ImagePolicyWebhook config file must reference a [kubeconfig](/docs/concepts/cluster-administration/authenticate-across-clusters-kubeconfig/) formatted file which sets up the connection to the backend. It is required that the backend communicate over TLS. The kubeconfig file's cluster field must point to the remote service, and the user field must contain the returned authorizer. @@ -244,7 +242,7 @@ users: client-certificate: /path/to/cert.pem # cert for the webhook admission controller to use client-key: /path/to/key.pem # key matching the cert ``` -For additional HTTP configuration, refer to the [kubeconfig](docs/tasks/access-application-cluster/configure-access-multiple-clusters/) documentation. +For additional HTTP configuration, refer to the [kubeconfig](/docs/concepts/cluster-administration/authenticate-across-clusters-kubeconfig/) documentation. #### Request Payloads @@ -316,23 +314,24 @@ In any case, the annotations are provided by the user and are not validated by K ### Initializers (alpha) +This admission controller is introduced in v1.7. The admission controller determines the initializers of a resource based on the existing `InitializerConfiguration`s. It sets the pending initializers by modifying the metadata of the resource to be created. -For more information, please check [Dynamic Admission Control](/docs/admin/extensible-admission-controllers/). +For more information, please check [Dynamic Admission Control](/docs/admin/extensible-admission-controllers.md). ### InitialResources (experimental) This admission controller observes pod creation requests. If a container omits compute resource requests and limits, then the admission controller auto-populates a compute resource request based on historical usage of containers running the same image. If there is not enough data to make a decision the Request is left unchanged. -When the admission controller sets a compute resource request, it does this by *annotating* +When the admission controller sets a compute resource request, it does this by *annotating* the the pod spec rather than mutating the `container.resources` fields. The annotations added contain the information on what compute resources were auto-populated. -See the [InitialResources proposal](https://git.k8s.io/community/contributors/design-proposals/autoscaling/initial-resources.md) for more details. +See the [InitialResouces proposal](https://git.k8s.io/community/contributors/design-proposals/autoscaling/initial-resources.md) for more details. -### LimitPodHardAntiAffinityTopology +### LimitPodHardAntiAffinity This admission controller denies any pod that defines `AntiAffinity` topology key other than `kubernetes.io/hostname` in `requiredDuringSchedulingRequiredDuringExecution`. @@ -345,7 +344,7 @@ your Kubernetes deployment, you MUST use this admission controller to enforce th be used to apply default resource requests to Pods that don't specify any; currently, the default LimitRanger applies a 0.1 CPU requirement to all Pods in the `default` namespace. -See the [limitRange design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) and the [example of Limit Range](/docs/tasks/administer-cluster/memory-default-namespace/) for more details. +See the [limitRange design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_limit_range.md) and the [example of Limit Range](/docs/tasks/configure-pod-container/limit-range/) for more details. ### MutatingAdmissionWebhook (beta in 1.9) @@ -355,7 +354,7 @@ webhooks are called in serial; each one may modify the object if it desires. This admission controller (as implied by the name) only runs in the mutating phase. If a webhook called by this has side effects (for example, decrementing quota) it -*must* have a reconciliation system, as it is not guaranteed that subsequent +*must* have a reconcilation system, as it is not guaranteed that subsequent webhooks or validating admission controllers will permit the request to finish. If you disable the MutatingAdmissionWebhook, you must also disable the @@ -415,7 +414,11 @@ This admission controller also protects the access to `metadata.ownerReferences[ of an object, so that only users with "update" permission to the `finalizers` subresource of the referenced *owner* can change it. -### PersistentVolumeLabel (DEPRECATED) +### Persistent Volume Claim Protection (alpha) +{% assign for_k8s_version="v1.9" %}{% include feature-state-alpha.md %} +The `PVCProtection` plugin adds the `kubernetes.io/pvc-protection` finalizer to newly created Persistent Volume Claims (PVCs). In case a user deletes a PVC the PVC is not removed until the finalizer is removed from the PVC by PVC Protection Controller. Refer to the [PVC Protection](/docs/concepts/storage/persistent-volumes/#persistent-volume-claim-protection) for more detailed information. + +### PersistentVolumeLabel This admission controller automatically attaches region or zone labels to PersistentVolumes as defined by the cloud provider (for example, GCE or AWS). @@ -423,7 +426,7 @@ It helps ensure the Pods and the PersistentVolumes mounted are in the same region and/or zone. If the admission controller doesn't support automatic labelling your PersistentVolumes, you may need to add the labels manually to prevent pods from mounting volumes from -a different zone. PersistentVolumeLabel is DEPRECATED and labeling persistent volumes has been taken over by [cloud controller manager](/docs/tasks/administer-cluster/running-cloud-controller/). +a different zone. ### PodNodeSelector @@ -431,7 +434,7 @@ This admission controller defaults and limits what node selectors may be used wi #### Configuration File Format -`PodNodeSelector` uses a configuration file to set options for the behavior of the backend. +PodNodeSelector uses a configuration file to set options for the behavior of the backend. Note that the configuration file format will move to a versioned file in a future release. This file may be json or yaml and has the following format: @@ -442,7 +445,7 @@ podNodeSelectorPluginConfig: namespace2: ``` -Reference the `PodNodeSelector` configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: +Reference the PodNodeSelector configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: ```yaml kind: AdmissionConfiguration @@ -454,7 +457,7 @@ plugins: ``` #### Configuration Annotation Format -`PodNodeSelector` uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign node selectors to namespaces. +PodNodeSelector uses the annotation key `scheduler.alpha.kubernetes.io/node-selector` to assign node selectors to namespaces. ```yaml apiVersion: v1 @@ -465,30 +468,12 @@ metadata: name: namespace3 ``` -#### Internal Behavior -This admission controller has the following behavior: - - 1. If the `Namespace` has an annotation with a key `scheduler.alpha.kubernetes.io/node-selector`, use its value as the - node selector. - 1. If the namespace lacks such an annotation, use the `clusterDefaultNodeSelector` defined in the `PodNodeSelector` - plugin configuration file as the node selector. - 1. Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts result in rejection. - 1. Evaluate the pod's node selector against the namespace-specific whitelist defined the plugin configuration file. - Conflicts result in rejection. - -{{< note >}} -**Note:** PodNodeSelector allows forcing pods to run on specifically labeled nodes. Also see the PodTolerationRestriction -admission plugin, which allows preventing pods from running on specifically tainted nodes. -{{< /note >}} - ### PersistentVolumeClaimResize This admission controller implements additional validations for checking incoming `PersistentVolumeClaim` resize requests. - -{{< note >}} **Note:** Support for volume resizing is available as an alpha feature. Admins must set the feature gate `ExpandPersistentVolumes` to `true` to enable resizing. -{{< /note >}} +{: .note} After enabling the `ExpandPersistentVolumes` feature gate, enabling the `PersistentVolumeClaimResize` admission controller is recommended, too. This admission controller prevents resizing of all claims by default unless a claim's `StorageClass` @@ -532,21 +517,78 @@ for more information. ### PodTolerationRestriction -This admission controller first verifies any conflict between a pod's tolerations and its -namespace's tolerations, and rejects the pod request if there is a conflict. -It then merges the namespace's tolerations into the pod's tolerations. -The resulting tolerations are checked against the namespace's whitelist of -tolerations. If the check succeeds, the pod request is admitted otherwise -rejected. +This admission controller defaults and limits what tolerations may be used on Pods in a given namespace. +The plugin will build a list of defaults, and a whitelist of allows tolerations based on annotations on the namespace. +When the namespace does not specify defaults, the cluster defaults from the plugin configuration are used. +When the namespace does not specify a whitelist, the namespace-specifc whitelist from the plugin configuration are used. + +**Note:** If the annotations are specified, but empty, they are considered to override the cluster-level defaults. +{: .note} + +#### Configuration File Format + +PodTolerationRestriction uses a configuration file to define: + +1. A cluster-wide default node selector (this is used if the `` annotation is not present on the target Namespace) -If the pod's namespace does not have any associated default or whitelist of -tolerations, then the cluster-level default or whitelist of tolerations are used -instead if specified. +2. Namespace-specific default whitelists (these are used if the `` annotation is not present on the target Namespace) -Tolerations to a namespace are assigned via the -`scheduler.alpha.kubernetes.io/defaultTolerations` and -`scheduler.alpha.kubernetes.io/tolerationsWhitelist` -annotation keys. +Note that the configuration file format will move to a versioned file in a future release. +This file may be json or yaml and has the following format: + +```yaml +apiVersion: podtolerationrestriction.admission.k8s.io/v1alpha1 + kind: Configuration + default: + - key: key1 + operator: Equal + value: value1 + whitelist: + - key: key1 + operator: Equal + value: value1 + - key: key2 + value: value2 +``` + +Reference the PodNodeSelector configuration file from the file provided to the API server's command line flag `--admission-control-config-file`: + +```yaml +kind: AdmissionConfiguration +apiVersion: apiserver.k8s.io/v1alpha1 +plugins: +- name: PodTolerationRestriction + path: podtolerationrestriction.yaml +... +``` + +#### Internal Logic + +1. Determine the `Pod`'s effective tolerations: + + i. If the `Namespace` lacks an annotation with the key `scheduler.alpha.kubernetes.io/default-tolerations`, then + the cluster defaults from the `PodTolerationRestriction` plugin configuration will be used as the + default tolerations for the `Namespace`. + + i. If the `Namespace` contains an annotation with the key `scheduler.alpha.kubernetes.io/default-tolerations`, + the value of the annotation will be considered as the default tolerations for the namespace. An empty value is + respected, and can be used as a way of overriding the cluster-wide default tolerations. + + i. The Pods effective tolerations are the result of merging the default annotations with the pod's specified tolerations. + Any conflicts result in admission failure. + +1. Determine the Namespace's effective whitelist: + + i. If the namespace lacks an annotation with the key `scheduler.alpha.kubernetes.io/tolerations-whitelist`, then + the `Namespace`-specific defaults from the `PodTolerationRestriction` plugin configuration will be used as the tolerations + whitelist for the namespace. + + i. If the namespace contains an annotation with the key `scheduler.alpha.kubernetes.io/tolerations-whitelist`, + the value of the annotation will be considered as the tolerations whitelist for the namespace. An empty value is + respected, and overrides the `Namespace`-specific whitelist. + +1. Evaluate the `Pod`'s effective tolerations against the `Namespace`'s effective whitelist. Any conflicts result in + admission failure. ### Priority @@ -560,20 +602,18 @@ objects in your Kubernetes deployment, you MUST use this admission controller to See the [resourceQuota design doc](https://git.k8s.io/community/contributors/design-proposals/resource-management/admission_control_resource_quota.md) and the [example of Resource Quota](/docs/concepts/policy/resource-quotas/) for more details. +It is strongly encouraged that this admission controller is configured last in the sequence of admission controllers. This is +so that quota is not prematurely incremented only for the request to be rejected later in admission control. ### SecurityContextDeny -This admission controller will deny any pod that attempts to set certain escalating [SecurityContext](/docs/tasks/configure-pod-container/security-context/) fields. This should be enabled if a cluster doesn't utilize [pod security policies](/docs/concepts/policy/pod-security-policy/) to restrict the set of values a security context can take. +This admission controller will deny any pod that attempts to set certain escalating [SecurityContext](/docs/user-guide/security-context) fields. This should be enabled if a cluster doesn't utilize [pod security policies](/docs/user-guide/pod-security-policy) to restrict the set of values a security context can take. ### ServiceAccount -This admission controller implements automation for [serviceAccounts](/docs/tasks/configure-pod-container/configure-service-account/). +This admission controller implements automation for [serviceAccounts](/docs/user-guide/service-accounts). We strongly recommend using this admission controller if you intend to make use of Kubernetes `ServiceAccount` objects. -### Storage Object in Use Protection (beta) -{{< feature-state for_k8s_version="v1.10" state="beta" >}} -The `StorageObjectInUseProtection` plugin adds the `kubernetes.io/pvc-protection` or `kubernetes.io/pv-protection` finalizers to newly created Persistent Volume Claims (PVCs) or Persistent Volumes (PV). In case a user deletes a PVC or PV the PVC or PV is not removed until the finalizer is removed from the PVC or PV by PVC or PV Protection Controller. Refer to the [Storage Object in Use Protection](/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection) for more detailed information. - ### ValidatingAdmissionWebhook (alpha in 1.8; beta in 1.9) This admission controller calls any validating webhooks which match the request. Matching @@ -582,7 +622,7 @@ fails. This admission controller only runs in the validation phase; the webhooks mutate the object, as opposed to the webhooks called by the `MutatingAdmissionWebhook` admission controller. If a webhook called by this has side effects (for example, decrementing quota) it -*must* have a reconciliation system, as it is not guaranteed that subsequent +*must* have a reconcilation system, as it is not guaranteed that subsequent webhooks or other validating admission controllers will permit the request to finish. If you disable the ValidatingAdmissionWebhook, you must also disable the @@ -594,10 +634,10 @@ versions >= 1.9). ## Is there a recommended set of admission controllers to use? Yes. -For Kubernetes >= 1.9.0, we strongly recommend running the following set of admission controllers (order matters for 1.9 but not >1.10): +For Kubernetes >= 1.9.0, we strongly recommend running the following set of admission controllers (order matters): ```shell ---admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota +--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota ``` It's worth reiterating that in 1.9 and up, these happen in a mutating phase