From 72090c98af46398f2b3498a79ab5ff7b2b50254d Mon Sep 17 00:00:00 2001 From: Aldo Culquicondor Date: Tue, 2 Jun 2020 16:22:46 -0400 Subject: [PATCH 1/3] Add Scheduling Configuration reference doc Built from the existing Scheduling Profiles doc. Signed-off-by: Aldo Culquicondor --- .../scheduling-eviction/kube-scheduler.md | 4 +- .../scheduling-framework.md | 4 +- .../pods/pod-topology-spread-constraints.md | 4 +- content/en/docs/reference/_index.md | 2 +- .../scheduling/{profiles.md => config.md} | 112 ++++++++++++++---- .../en/docs/reference/scheduling/policies.md | 2 +- static/_redirects | 1 + 7 files changed, 96 insertions(+), 33 deletions(-) rename content/en/docs/reference/scheduling/{profiles.md => config.md} (70%) diff --git a/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md b/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md index 406c3f974baab..b6a68d4fbe5e4 100644 --- a/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md +++ b/content/en/docs/concepts/scheduling-eviction/kube-scheduler.md @@ -81,8 +81,8 @@ of the scheduler: 1. [Scheduling Policies](/docs/reference/scheduling/policies) allow you to configure _Predicates_ for filtering and _Priorities_ for scoring. -1. [Scheduling Profiles](/docs/reference/scheduling/profiles) allow you to - configure Plugins that implement different scheduling stages, including: +1. [Scheduling Profiles](/docs/reference/scheduling/config/#profiles) allow you + to configure Plugins that implement different scheduling stages, including: `QueueSort`, `Filter`, `Score`, `Bind`, `Reserve`, `Permit`, and others. You can also configure the kube-scheduler to run different profiles. diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index 5798b0579f092..b93a952c6c7d4 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -228,7 +228,7 @@ type PreFilterPlugin interface { You can enable or disable plugins in the scheduler configuration. If you are using Kubernetes v1.18 or later, most scheduling -[plugins](/docs/reference/scheduling/profiles/#scheduling-plugins) are in use and +[plugins](/docs/reference/scheduling/config/#scheduling-plugins) are in use and enabled by default. In addition to default plugins, you can also implement your own scheduling @@ -237,5 +237,5 @@ plugins and get them configured along with default plugins. You can visit If you are using Kubernetes v1.18 or later, you can configure a set of plugins as a scheduler profile and then define multiple profiles to fit various kinds of workload. -Learn more at [multiple profiles](/docs/reference/scheduling/profiles/#multiple-profiles). +Learn more at [multiple profiles](/docs/reference/scheduling/config/#multiple-profiles). diff --git a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md index 2b16894e6ba40..3210af95f6bf5 100644 --- a/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md +++ b/content/en/docs/concepts/workloads/pods/pod-topology-spread-constraints.md @@ -195,7 +195,7 @@ topology spread constraints are applied to a Pod if, and only if: - It belongs to a service, replication controller, replica set or stateful set. Default constraints can be set as part of the `PodTopologySpread` plugin args -in a [scheduling profile](/docs/reference/scheduling/profiles). +in a [scheduling profile](/docs/reference/scheduling/config/#profiles). The constraints are specified with the same [API above](#api), except that `labelSelector` must be empty. The selectors are calculated from the services, replication controllers, replica sets or stateful sets that the Pod belongs to. @@ -219,7 +219,7 @@ profiles: {{< note >}} The score produced by default scheduling constraints might conflict with the score produced by the -[`DefaultPodTopologySpread` plugin](/docs/reference/scheduling/profiles/#scheduling-plugins). +[`DefaultPodTopologySpread` plugin](/docs/reference/scheduling/config/#scheduling-plugins). It is recommended that you disable this plugin in the scheduling profile when using default constraints for `PodTopologySpread`. {{< /note >}} diff --git a/content/en/docs/reference/_index.md b/content/en/docs/reference/_index.md index 619430875ec1a..af25359434ee6 100644 --- a/content/en/docs/reference/_index.md +++ b/content/en/docs/reference/_index.md @@ -46,7 +46,7 @@ client libraries: * [kube-proxy](/docs/reference/command-line-tools-reference/kube-proxy/) - Can do simple TCP/UDP stream forwarding or round-robin TCP/UDP forwarding across a set of back-ends. * [kube-scheduler](/docs/reference/command-line-tools-reference/kube-scheduler/) - Scheduler that manages availability, performance, and capacity. * [kube-scheduler Policies](/docs/reference/scheduling/policies) - * [kube-scheduler Profiles](/docs/reference/scheduling/profiles) + * [kube-scheduler Profiles](/docs/reference/scheduling/config#profiles) ## Design Docs diff --git a/content/en/docs/reference/scheduling/profiles.md b/content/en/docs/reference/scheduling/config.md similarity index 70% rename from content/en/docs/reference/scheduling/profiles.md rename to content/en/docs/reference/scheduling/config.md index fe28d10bd1dab..19cd6d63aca1b 100644 --- a/content/en/docs/reference/scheduling/profiles.md +++ b/content/en/docs/reference/scheduling/config.md @@ -1,30 +1,57 @@ --- -title: Scheduling Profiles +title: Scheduler Configuration content_type: concept weight: 20 --- +{{< feature-state for_k8s_version="v1.19" state="beta" >}} + +The `KubeSchedulerConfiguration` is a configuration API for `kube-scheduler` +that can be provided as a file via `--config` command line flag. + -{{< feature-state for_k8s_version="v1.18" state="alpha" >}} + -A scheduling Profile allows you to configure the different stages of scheduling -in the {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}}. -Each stage is exposed in a extension point. Plugins provide scheduling behaviors -by implementing one or more of these extension points. +## Minimal Configuration -You can specify scheduling profiles by running `kube-scheduler --config `, -using the component config APIs -([`v1alpha1`](https://pkg.go.dev/k8s.io/kube-scheduler@{{< param "fullversion" >}}/config/v1alpha1?tab=doc#KubeSchedulerConfiguration) -or [`v1alpha2`](https://pkg.go.dev/k8s.io/kube-scheduler@{{< param "fullversion" >}}/config/v1alpha2?tab=doc#KubeSchedulerConfiguration)). -The `v1alpha2` API allows you to configure kube-scheduler to run -[multiple profiles](#multiple-profiles). +A minimal configuration looks as follows: +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta1 +kind: KubeSchedulerConfiguration +clientConnection: + kubeconfig: /etc/srv/kubernetes/kube-scheduler/kubeconfig +``` +## Upgrading from `v1alpha2` to `v1beta1` {#beta-changes} - +When migrating from `kubescheduler.config.k8s.io/v1alpha2` to `kubescheduler.config.k8s.io/v1beta1`, +beware of the following changes, if applicable: + +- `.bindTimeoutSeconds` was moved as part of plugin args for `VolumeBinding`, + which can be configured separately per [profile](#profiles). +- `.extenders` are updated to satisfy API standards. In particular: + - `.extenders` decoding is case sensitive. All fields are affected. + - `.extenders[*].httpTimeout` is of type `metav1.Duration`. + - `.extenders[*].enableHttps` is renamed to `.extenders[*].enableHTTPS`. +- `RequestedToCapacityRatio` args decoding is case sensitive. All fields are affected. +- `DefaultPodTopologySpread` [plugin](#scheduling-plugins) is renamed to `SelectorSpread`. +- `Unreserve` extension point is removed from Profile definition. All `Reserve` + plugins implement an `Unreserve` call. + +## Profiles -## Extension points +A scheduling Profile allows you to configure the different stages of scheduling +in the {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}}. +Each stage is exposed in a [extension point](#extension-points). +[Plugins](#scheduling-plugins) provide scheduling behaviors by implementing one +or more of these extension points. + +A single instance of `kube-scheduler` can be configured to run +[multiple profiles](#multiple-profiles) + +### Extension points Scheduling happens in a series of stages that are exposed through the following extension points: @@ -43,7 +70,9 @@ extension points: filtering phase. The scheduler will then select the node with the highest weighted scores sum. 1. `Reserve`: This is an informational extension point that notifies plugins - when resources have being reserved for a given Pod. + when resources have been reserved for a given Pod. Plugins also implement an + `Unreserve` call that gets called in the case of failure during or after + `Reserve`. 1. `Permit`: These plugins can prevent or delay the binding of a Pod. 1. `PreBind`: These plugins perform any work required before a Pod is bound. 1. `Bind`: The plugins bind a Pod to a Node. Bind plugins are called in order @@ -51,15 +80,35 @@ extension points: least one bind plugin is required. 1. `PostBind`: This is an informational extension point that is called after a Pod has been bound. -1. `UnReserve`: This is an informational extension point that is called if - a Pod is rejected after being reserved and put on hold by a `Permit` plugin. + +For each extension point, you could disable specific [default plugins](#scheduling-plugins) +or enable your own. For example: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta1 +kind: KubeSchedulerConfiguration +profiles: + - plugins: + score: + disabled: + - name: NodeResourcesLeastAllocated + enabled: + - name: MyCustomPluginA + weight: 2 + - name: MyCustomPluginB + weight: 1 +``` + +You can use `*` as name in the disabled array to disable all default plugins +for that extension point. This can also be used to rearrange plugins order, if +desired. -## Scheduling plugins +### Scheduling plugins The following plugins, enabled by default, implement one or more of these extension points: -- `DefaultTopologySpread`: Favors spreading across nodes for Pods that belong to +- `SelectorSpread`: Favors spreading across nodes for Pods that belong to {{< glossary_tooltip text="Services" term_id="service" >}}, {{< glossary_tooltip text="ReplicaSets" term_id="replica-set" >}} and {{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} @@ -99,7 +148,7 @@ extension points: Extension points: `Score`. - `VolumeBinding`: Checks if the node has or if it can bind the requested {{< glossary_tooltip text="volumes" term_id="volume" >}}. - Extension points: `Filter`. + Extension points: `PreFilter`, `Filter`, `Reserve`, `PreBind`. - `VolumeRestrictions`: Checks that volumes mounted in the node satisfy restrictions that are specific to the volume provider. Extension points: `Filter`. @@ -147,10 +196,25 @@ that are not enabled by default: Service across nodes. Extension points: `PreFilter`, `Filter`, `Score`. -## Multiple profiles +### Multiple profiles + +`kube-scheduler` can be configured to +run more than one profile. Each profile has an associated scheduler name. Each +profile can have a different set of plugins configured in its +[extension points](#extension-points). + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1beta1 +kind: KubeSchedulerConfiguration +profiles: + - schedulerName: default-scheduler + - schedulerName: no-scoring-scheduler + plugins: + score: + disabled: + - name: '*' +``` -When using the component config API v1alpha2, a scheduler can be configured to -run more than one profile. Each profile has an associated scheduler name. Pods that want to be scheduled according to a specific profile can include the corresponding scheduler name in its `.spec.schedulerName`. @@ -174,8 +238,6 @@ the same configuration parameters (if applicable). This is because the scheduler only has one pending pods queue. {{< /note >}} - - ## {{% heading "whatsnext" %}} * Learn about [scheduling](/docs/concepts/scheduling-eviction/kube-scheduler/) diff --git a/content/en/docs/reference/scheduling/policies.md b/content/en/docs/reference/scheduling/policies.md index 67d34e59f78e0..9601b22ec4e3e 100644 --- a/content/en/docs/reference/scheduling/policies.md +++ b/content/en/docs/reference/scheduling/policies.md @@ -122,5 +122,5 @@ The following *priorities* implement scoring: ## {{% heading "whatsnext" %}} * Learn about [scheduling](/docs/concepts/scheduling-eviction/kube-scheduler/) -* Learn about [kube-scheduler profiles](/docs/reference/scheduling/profiles/) +* Learn about [kube-scheduler Configuration](/docs/reference/scheduling/config/) diff --git a/static/_redirects b/static/_redirects index bb2c0aeba1ddb..8294ecd511dec 100644 --- a/static/_redirects +++ b/static/_redirects @@ -184,6 +184,7 @@ /docs/reference/glossary/maintainer/ /docs/reference/glossary/approver/ 301 /docs/reference/kubectl/kubectl/kubectl_*.md /docs/reference/generated/kubectl/kubectl-commands#:splat 301 +/docs/reference/scheduling/profiles/ /docs/reference/scheduling/config/#profiles 301 /docs/reporting-security-issues/ /security/ 301 From 34c3537a728a710855664d74bc9dcee90537dcfa Mon Sep 17 00:00:00 2001 From: Aldo Culquicondor Date: Mon, 6 Jul 2020 17:15:43 -0400 Subject: [PATCH 2/3] Review comments Signed-off-by: Aldo Culquicondor --- .../en/docs/reference/scheduling/config.md | 52 ++++++++++--------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/content/en/docs/reference/scheduling/config.md b/content/en/docs/reference/scheduling/config.md index 19cd6d63aca1b..c264a99253c97 100644 --- a/content/en/docs/reference/scheduling/config.md +++ b/content/en/docs/reference/scheduling/config.md @@ -6,8 +6,8 @@ weight: 20 {{< feature-state for_k8s_version="v1.19" state="beta" >}} -The `KubeSchedulerConfiguration` is a configuration API for `kube-scheduler` -that can be provided as a file via `--config` command line flag. +You can customize the behavior of the `kube-scheduler` by writing a configuration +file and passing its path as a command line argument. @@ -24,22 +24,6 @@ clientConnection: kubeconfig: /etc/srv/kubernetes/kube-scheduler/kubeconfig ``` -## Upgrading from `v1alpha2` to `v1beta1` {#beta-changes} - -When migrating from `kubescheduler.config.k8s.io/v1alpha2` to `kubescheduler.config.k8s.io/v1beta1`, -beware of the following changes, if applicable: - -- `.bindTimeoutSeconds` was moved as part of plugin args for `VolumeBinding`, - which can be configured separately per [profile](#profiles). -- `.extenders` are updated to satisfy API standards. In particular: - - `.extenders` decoding is case sensitive. All fields are affected. - - `.extenders[*].httpTimeout` is of type `metav1.Duration`. - - `.extenders[*].enableHttps` is renamed to `.extenders[*].enableHTTPS`. -- `RequestedToCapacityRatio` args decoding is case sensitive. All fields are affected. -- `DefaultPodTopologySpread` [plugin](#scheduling-plugins) is renamed to `SelectorSpread`. -- `Unreserve` extension point is removed from Profile definition. All `Reserve` - plugins implement an `Unreserve` call. - ## Profiles A scheduling Profile allows you to configure the different stages of scheduling @@ -48,7 +32,7 @@ Each stage is exposed in a [extension point](#extension-points). [Plugins](#scheduling-plugins) provide scheduling behaviors by implementing one or more of these extension points. -A single instance of `kube-scheduler` can be configured to run +You can configure a single instance of `kube-scheduler` to run [multiple profiles](#multiple-profiles) ### Extension points @@ -60,10 +44,12 @@ extension points: sort pending Pods in the scheduling queue. Exactly one queue sort plugin may be enabled at a time. 1. `PreFilter`: These plugins are used to pre-process or check information - about a Pod or the cluster before filtering. + about a Pod or the cluster before filtering. They can mark a pod as + unschedulable. 1. `Filter`: These plugins are the equivalent of Predicates in a scheduling Policy and are used to filter out nodes that can not run the Pod. Filters - are called in the configured order. + are called in the configured order. A pod is marked as unschedulable if no + nodes pass all the filters. 1. `PreScore`: This is an informational extension point that can be used for doing pre-scoring work. 1. `Score`: These plugins provide a score to each node that has passed the @@ -198,10 +184,9 @@ that are not enabled by default: ### Multiple profiles -`kube-scheduler` can be configured to -run more than one profile. Each profile has an associated scheduler name. Each -profile can have a different set of plugins configured in its -[extension points](#extension-points). +You can configure `kube-scheduler` to run more than one profile. +Each profile has an associated scheduler name and can have a different set of +plugins configured in its [extension points](#extension-points). For example: ```yaml apiVersion: kubescheduler.config.k8s.io/v1beta1 @@ -238,7 +223,24 @@ the same configuration parameters (if applicable). This is because the scheduler only has one pending pods queue. {{< /note >}} +## Upgrading from `v1alpha2` to `v1beta1` {#beta-changes} + +When migrating from `kubescheduler.config.k8s.io/v1alpha2` to `kubescheduler.config.k8s.io/v1beta1`, +beware of the following changes, if applicable: + +- `.bindTimeoutSeconds` was moved as part of plugin args for `VolumeBinding`, + which can be configured separately per [profile](#profiles). +- `.extenders` are updated to satisfy API standards. In particular: + - `.extenders` decoding is case sensitive. All fields are affected. + - `.extenders[*].httpTimeout` is of type `metav1.Duration`. + - `.extenders[*].enableHttps` is renamed to `.extenders[*].enableHTTPS`. +- `RequestedToCapacityRatio` args decoding is case sensitive. All fields are affected. +- `DefaultPodTopologySpread` [plugin](#scheduling-plugins) is renamed to `SelectorSpread`. +- `Unreserve` extension point is removed from Profile definition. All `Reserve` + plugins implement an `Unreserve` call. + ## {{% heading "whatsnext" %}} +* Read the [kube-scheduler reference](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) * Learn about [scheduling](/docs/concepts/scheduling-eviction/kube-scheduler/) From 9ed3f427a2e28dffe6635c26007ec29b4bd68769 Mon Sep 17 00:00:00 2001 From: Aldo Culquicondor Date: Tue, 7 Jul 2020 10:41:45 -0400 Subject: [PATCH 3/3] Review comments 2 Signed-off-by: Aldo Culquicondor --- content/en/docs/reference/scheduling/config.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/content/en/docs/reference/scheduling/config.md b/content/en/docs/reference/scheduling/config.md index c264a99253c97..1b6e90f15db5c 100644 --- a/content/en/docs/reference/scheduling/config.md +++ b/content/en/docs/reference/scheduling/config.md @@ -28,12 +28,12 @@ clientConnection: A scheduling Profile allows you to configure the different stages of scheduling in the {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}}. -Each stage is exposed in a [extension point](#extension-points). +Each stage is exposed in an [extension point](#extension-points). [Plugins](#scheduling-plugins) provide scheduling behaviors by implementing one or more of these extension points. You can configure a single instance of `kube-scheduler` to run -[multiple profiles](#multiple-profiles) +[multiple profiles](#multiple-profiles). ### Extension points @@ -85,7 +85,7 @@ profiles: weight: 1 ``` -You can use `*` as name in the disabled array to disable all default plugins +You can use `*` as name in the disabled array to disable all default plugins for that extension point. This can also be used to rearrange plugins order, if desired. @@ -186,7 +186,11 @@ that are not enabled by default: You can configure `kube-scheduler` to run more than one profile. Each profile has an associated scheduler name and can have a different set of -plugins configured in its [extension points](#extension-points). For example: +plugins configured in its [extension points](#extension-points). + +With the following sample configuration, the scheduler will run with two +profiles: one with the default plugins and one with all scoring plugins +disabled. ```yaml apiVersion: kubescheduler.config.k8s.io/v1beta1 @@ -195,6 +199,9 @@ profiles: - schedulerName: default-scheduler - schedulerName: no-scoring-scheduler plugins: + prescore: + disabled: + - name: '*' score: disabled: - name: '*'