-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Scheduling Configuration reference doc #21437
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,30 +1,41 @@ | ||
--- | ||
title: Scheduling Profiles | ||
title: Scheduler Configuration | ||
content_type: concept | ||
weight: 20 | ||
--- | ||
|
||
{{< feature-state for_k8s_version="v1.19" state="beta" >}} | ||
|
||
You can customize the behavior of the `kube-scheduler` by writing a configuration | ||
file and passing its path as a command line argument. | ||
|
||
<!-- overview --> | ||
|
||
{{< feature-state for_k8s_version="v1.18" state="alpha" >}} | ||
<!-- body --> | ||
|
||
A scheduling Profile allows you to configure the different stages of scheduling | ||
in the {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}}. | ||
Each stage is exposed in a extension point. Plugins provide scheduling behaviors | ||
by implementing one or more of these extension points. | ||
|
||
You can specify scheduling profiles by running `kube-scheduler --config <filename>`, | ||
using the component config APIs | ||
([`v1alpha1`](https://pkg.go.dev/k8s.io/kube-scheduler@{{< param "fullversion" >}}/config/v1alpha1?tab=doc#KubeSchedulerConfiguration) | ||
or [`v1alpha2`](https://pkg.go.dev/k8s.io/kube-scheduler@{{< param "fullversion" >}}/config/v1alpha2?tab=doc#KubeSchedulerConfiguration)). | ||
The `v1alpha2` API allows you to configure kube-scheduler to run | ||
[multiple profiles](#multiple-profiles). | ||
## Minimal Configuration | ||
|
||
A minimal configuration looks as follows: | ||
|
||
```yaml | ||
apiVersion: kubescheduler.config.k8s.io/v1beta1 | ||
kind: KubeSchedulerConfiguration | ||
clientConnection: | ||
kubeconfig: /etc/srv/kubernetes/kube-scheduler/kubeconfig | ||
``` | ||
|
||
<!-- body --> | ||
## Profiles | ||
|
||
## Extension points | ||
A scheduling Profile allows you to configure the different stages of scheduling | ||
in the {{< glossary_tooltip text="kube-scheduler" term_id="kube-scheduler" >}}. | ||
Each stage is exposed in an [extension point](#extension-points). | ||
[Plugins](#scheduling-plugins) provide scheduling behaviors by implementing one | ||
or more of these extension points. | ||
|
||
You can configure a single instance of `kube-scheduler` to run | ||
[multiple profiles](#multiple-profiles). | ||
|
||
### Extension points | ||
|
||
Scheduling happens in a series of stages that are exposed through the following | ||
extension points: | ||
|
@@ -33,33 +44,57 @@ extension points: | |
sort pending Pods in the scheduling queue. Exactly one queue sort plugin | ||
may be enabled at a time. | ||
1. `PreFilter`: These plugins are used to pre-process or check information | ||
about a Pod or the cluster before filtering. | ||
about a Pod or the cluster before filtering. They can mark a pod as | ||
unschedulable. | ||
1. `Filter`: These plugins are the equivalent of Predicates in a scheduling | ||
Policy and are used to filter out nodes that can not run the Pod. Filters | ||
are called in the configured order. | ||
are called in the configured order. A pod is marked as unschedulable if no | ||
nodes pass all the filters. | ||
1. `PreScore`: This is an informational extension point that can be used | ||
for doing pre-scoring work. | ||
1. `Score`: These plugins provide a score to each node that has passed the | ||
filtering phase. The scheduler will then select the node with the highest | ||
weighted scores sum. | ||
1. `Reserve`: This is an informational extension point that notifies plugins | ||
when resources have being reserved for a given Pod. | ||
when resources have been reserved for a given Pod. Plugins also implement an | ||
`Unreserve` call that gets called in the case of failure during or after | ||
`Reserve`. | ||
1. `Permit`: These plugins can prevent or delay the binding of a Pod. | ||
1. `PreBind`: These plugins perform any work required before a Pod is bound. | ||
1. `Bind`: The plugins bind a Pod to a Node. Bind plugins are called in order | ||
and once one has done the binding, the remaining plugins are skipped. At | ||
least one bind plugin is required. | ||
1. `PostBind`: This is an informational extension point that is called after | ||
a Pod has been bound. | ||
1. `UnReserve`: This is an informational extension point that is called if | ||
a Pod is rejected after being reserved and put on hold by a `Permit` plugin. | ||
|
||
alculquicondor marked this conversation as resolved.
Show resolved
Hide resolved
|
||
For each extension point, you could disable specific [default plugins](#scheduling-plugins) | ||
or enable your own. For example: | ||
|
||
```yaml | ||
apiVersion: kubescheduler.config.k8s.io/v1beta1 | ||
kind: KubeSchedulerConfiguration | ||
profiles: | ||
- plugins: | ||
score: | ||
disabled: | ||
- name: NodeResourcesLeastAllocated | ||
enabled: | ||
- name: MyCustomPluginA | ||
weight: 2 | ||
- name: MyCustomPluginB | ||
weight: 1 | ||
``` | ||
|
||
You can use `*` as name in the disabled array to disable all default plugins | ||
for that extension point. This can also be used to rearrange plugins order, if | ||
desired. | ||
|
||
## Scheduling plugins | ||
### Scheduling plugins | ||
|
||
The following plugins, enabled by default, implement one or more of these | ||
extension points: | ||
|
||
- `DefaultTopologySpread`: Favors spreading across nodes for Pods that belong to | ||
- `SelectorSpread`: Favors spreading across nodes for Pods that belong to | ||
{{< glossary_tooltip text="Services" term_id="service" >}}, | ||
{{< glossary_tooltip text="ReplicaSets" term_id="replica-set" >}} and | ||
{{< glossary_tooltip text="StatefulSets" term_id="statefulset" >}} | ||
|
@@ -99,7 +134,7 @@ extension points: | |
Extension points: `Score`. | ||
- `VolumeBinding`: Checks if the node has or if it can bind the requested | ||
{{< glossary_tooltip text="volumes" term_id="volume" >}}. | ||
Extension points: `Filter`. | ||
Extension points: `PreFilter`, `Filter`, `Reserve`, `PreBind`. | ||
- `VolumeRestrictions`: Checks that volumes mounted in the node satisfy | ||
restrictions that are specific to the volume provider. | ||
Extension points: `Filter`. | ||
|
@@ -147,10 +182,31 @@ that are not enabled by default: | |
Service across nodes. | ||
Extension points: `PreFilter`, `Filter`, `Score`. | ||
|
||
## Multiple profiles | ||
### Multiple profiles | ||
|
||
You can configure `kube-scheduler` to run more than one profile. | ||
Each profile has an associated scheduler name and can have a different set of | ||
plugins configured in its [extension points](#extension-points). | ||
|
||
With the following sample configuration, the scheduler will run with two | ||
profiles: one with the default plugins and one with all scoring plugins | ||
disabled. | ||
|
||
```yaml | ||
apiVersion: kubescheduler.config.k8s.io/v1beta1 | ||
kind: KubeSchedulerConfiguration | ||
profiles: | ||
- schedulerName: default-scheduler | ||
- schedulerName: no-scoring-scheduler | ||
plugins: | ||
alculquicondor marked this conversation as resolved.
Show resolved
Hide resolved
|
||
prescore: | ||
disabled: | ||
- name: '*' | ||
score: | ||
disabled: | ||
- name: '*' | ||
``` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would it make sense to explain this example? For example by adding a paragraph:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good suggestion There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is interesting, so are we saying we will be able to use the same scheduler binary to act as two different schedulers based on the scheduler field in the Pod ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct. |
||
|
||
When using the component config API v1alpha2, a scheduler can be configured to | ||
run more than one profile. Each profile has an associated scheduler name. | ||
Pods that want to be scheduled according to a specific profile can include | ||
the corresponding scheduler name in its `.spec.schedulerName`. | ||
|
||
|
@@ -174,9 +230,24 @@ the same configuration parameters (if applicable). This is because the scheduler | |
only has one pending pods queue. | ||
{{< /note >}} | ||
|
||
## Upgrading from `v1alpha2` to `v1beta1` {#beta-changes} | ||
|
||
When migrating from `kubescheduler.config.k8s.io/v1alpha2` to `kubescheduler.config.k8s.io/v1beta1`, | ||
beware of the following changes, if applicable: | ||
|
||
- `.bindTimeoutSeconds` was moved as part of plugin args for `VolumeBinding`, | ||
which can be configured separately per [profile](#profiles). | ||
- `.extenders` are updated to satisfy API standards. In particular: | ||
- `.extenders` decoding is case sensitive. All fields are affected. | ||
- `.extenders[*].httpTimeout` is of type `metav1.Duration`. | ||
- `.extenders[*].enableHttps` is renamed to `.extenders[*].enableHTTPS`. | ||
- `RequestedToCapacityRatio` args decoding is case sensitive. All fields are affected. | ||
- `DefaultPodTopologySpread` [plugin](#scheduling-plugins) is renamed to `SelectorSpread`. | ||
- `Unreserve` extension point is removed from Profile definition. All `Reserve` | ||
plugins implement an `Unreserve` call. | ||
|
||
## {{% heading "whatsnext" %}} | ||
|
||
* Read the [kube-scheduler reference](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/) | ||
* Learn about [scheduling](/docs/concepts/scheduling-eviction/kube-scheduler/) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will submit another PR finishing the
PostFilter
part.