From 111e6d1e57e6c3920384271238fbb3bc6b76bcf2 Mon Sep 17 00:00:00 2001 From: Jim Angel Date: Tue, 26 Mar 2019 22:54:29 -0500 Subject: [PATCH 01/31] initial commit From 22441ecaee8b26bf6d4372405a7da11285be31f0 Mon Sep 17 00:00:00 2001 From: Sandeep Rajan Date: Thu, 25 Apr 2019 14:31:46 -0400 Subject: [PATCH 02/31] Update content on kube-dns to coredns configmap translation (#13826) Updates and corrects the part of configmap for CoreDNS --- .../dns-custom-nameservers.md | 38 ++++++++----------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/dns-custom-nameservers.md b/content/en/docs/tasks/administer-cluster/dns-custom-nameservers.md index a2d4ff41d586d..7b594239d7b15 100644 --- a/content/en/docs/tasks/administer-cluster/dns-custom-nameservers.md +++ b/content/en/docs/tasks/administer-cluster/dns-custom-nameservers.md @@ -78,11 +78,10 @@ data: health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure - upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 - proxy . /etc/resolv.conf + forward . /etc/resolv.conf cache 30 loop reload @@ -97,10 +96,8 @@ The Corefile configuration includes the following [plugins](https://coredns.io/p > The `pods insecure` option is provided for backward compatibility with kube-dns. You can use the `pods verified` option, which returns an A record only if there exists a pod in same namespace with matching IP. The `pods disabled` option can be used if you don't use pod records. -> `Upstream` is used for resolving services that point to external hosts (External Services). - * [prometheus](https://coredns.io/plugins/prometheus/): Metrics of CoreDNS are available at http://localhost:9153/metrics in [Prometheus](https://prometheus.io/) format. -* [proxy](https://coredns.io/plugins/proxy/): Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers (/etc/resolv.conf). +* [forward](https://coredns.io/plugins/forward/): Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers (/etc/resolv.conf). * [cache](https://coredns.io/plugins/cache/): This enables a frontend cache. * [loop](https://coredns.io/plugins/loop/): Detects simple forwarding loops and halts the CoreDNS process if a loop is found. * [reload](https://coredns.io/plugins/reload): Allows automatic reload of a changed Corefile. After you edit the ConfigMap configuration, allow two minutes for your changes to take effect. @@ -110,7 +107,7 @@ You can modify the default CoreDNS behavior by modifying the ConfigMap. ### Configuration of Stub-domain and upstream nameserver using CoreDNS -CoreDNS has the ability to configure stubdomains and upstream nameservers using the [proxy plugin](https://coredns.io/plugins/proxy/). +CoreDNS has the ability to configure stubdomains and upstream nameservers using the [forward plugin](https://coredns.io/plugins/forward/). #### Example If a cluster operator has a [Consul](https://www.consul.io/) domain server located at 10.150.0.1, and all Consul names have the suffix .consul.local. To configure it in CoreDNS, the cluster administrator creates the following stanza in the CoreDNS ConfigMap. @@ -119,18 +116,15 @@ If a cluster operator has a [Consul](https://www.consul.io/) domain server locat consul.local:53 { errors cache 30 - proxy . 10.150.0.1 + forward . 10.150.0.1 } ``` -To explicitly force all non-cluster DNS lookups to go through a specific nameserver at 172.16.0.1, point the `proxy` and `upstream` to the nameserver instead of `/etc/resolv.conf` +To explicitly force all non-cluster DNS lookups to go through a specific nameserver at 172.16.0.1, point the `forward` to the nameserver instead of `/etc/resolv.conf` ``` -proxy . 172.16.0.1 +forward . 172.16.0.1 ``` -``` -upstream 172.16.0.1 -``` The final ConfigMap along with the default `Corefile` configuration looks like: @@ -147,11 +141,10 @@ data: health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure - upstream 172.16.0.1 fallthrough in-addr.arpa ip6.arpa } prometheus :9153 - proxy . 172.16.0.1 + forward . 172.16.0.1 cache 30 loop reload @@ -160,10 +153,12 @@ data: consul.local:53 { errors cache 30 - proxy . 10.150.0.1 + forward . 10.150.0.1 } ``` In Kubernetes version 1.10 and later, kubeadm supports automatic translation of the CoreDNS ConfigMap from the kube-dns ConfigMap. +***Note: While kube-dns accepts an FQDN for stubdomain and nameserver (eg: ns.foo.com), CoreDNS does not support this feature. +During translation, all FQDN nameservers will be omitted from the CoreDNS config.*** ## Kube-dns @@ -308,7 +303,7 @@ data: ## CoreDNS configuration equivalent to kube-dns CoreDNS supports the features of kube-dns and more. -A ConfigMap created for kube-dns to support `StubDomains`and `upstreamNameservers` translates to the `proxy` plugin in CoreDNS. +A ConfigMap created for kube-dns to support `StubDomains`and `upstreamNameservers` translates to the `forward` plugin in CoreDNS. Similarly, the `Federations` plugin in kube-dns translates to the `federation` plugin in CoreDNS. ### Example @@ -341,12 +336,12 @@ federation cluster.local { abc.com:53 { errors cache 30 - proxy . 1.2.3.4 + forward . 1.2.3.4 } my.cluster.local:53 { errors cache 30 - proxy . 2.3.4.5 + forward . 2.3.4.5 } ``` @@ -357,7 +352,6 @@ The complete Corefile with the default plugins: errors health kubernetes cluster.local in-addr.arpa ip6.arpa { - upstream 8.8.8.8 8.8.4.4 pods insecure fallthrough in-addr.arpa ip6.arpa } @@ -365,18 +359,18 @@ The complete Corefile with the default plugins: foo foo.feddomain.com } prometheus :9153 - proxy . 8.8.8.8 8.8.4.4 + forward . 8.8.8.8 8.8.4.4 cache 30 } abc.com:53 { errors cache 30 - proxy . 1.2.3.4 + forward . 1.2.3.4 } my.cluster.local:53 { errors cache 30 - proxy . 2.3.4.5 + forward . 2.3.4.5 } ``` From 99ae92bdf9dbf87bc873604cc69e60cb543c2b4a Mon Sep 17 00:00:00 2001 From: noctarius aka Christoph Engelbert Date: Mon, 6 May 2019 20:31:40 +0200 Subject: [PATCH 03/31] Readded the link to Instana and Sysdig, which seem to have gone missing due to a merge error or anything like that during preparationg of the 1.14 release. (#14184) original pull request: https://github.com/kubernetes/website/pull/12977 Signed-off-by: noctarius --- content/en/docs/concepts/workloads/controllers/daemonset.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/content/en/docs/concepts/workloads/controllers/daemonset.md b/content/en/docs/concepts/workloads/controllers/daemonset.md index 3a1a875e2e2e3..e426493325d66 100644 --- a/content/en/docs/concepts/workloads/controllers/daemonset.md +++ b/content/en/docs/concepts/workloads/controllers/daemonset.md @@ -20,8 +20,7 @@ Some typical uses of a DaemonSet are: - running a cluster storage daemon, such as `glusterd`, `ceph`, on each node. - running a logs collection daemon on every node, such as `fluentd` or `logstash`. -- running a node monitoring daemon on every node, such as [Prometheus Node Exporter]( - https://github.com/prometheus/node_exporter), `collectd`, [Dynatrace OneAgent](https://www.dynatrace.com/technologies/kubernetes-monitoring/), [AppDynamics Agent](https://docs.appdynamics.com/display/CLOUD/Container+Visibility+with+Kubernetes), [Datadog agent](https://docs.datadoghq.com/agent/kubernetes/daemonset_setup/), [New Relic agent](https://docs.newrelic.com/docs/integrations/kubernetes-integration/installation/kubernetes-installation-configuration), Ganglia `gmond` or Instana agent. +- running a node monitoring daemon on every node, such as [Prometheus Node Exporter](https://github.com/prometheus/node_exporter), [Sysdig Agent](https://sysdigdocs.atlassian.net/wiki/spaces/Platform), `collectd`, [Dynatrace OneAgent](https://www.dynatrace.com/technologies/kubernetes-monitoring/), [AppDynamics Agent](https://docs.appdynamics.com/display/CLOUD/Container+Visibility+with+Kubernetes), [Datadog agent](https://docs.datadoghq.com/agent/kubernetes/daemonset_setup/), [New Relic agent](https://docs.newrelic.com/docs/integrations/kubernetes-integration/installation/kubernetes-installation-configuration), Ganglia `gmond` or [Instana Agent](https://www.instana.com/supported-integrations/kubernetes-monitoring/). In a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon. A more complex setup might use multiple DaemonSets for a single type of daemon, but with From a1f811d8524e6e9dd8682927244252ed0e1a064b Mon Sep 17 00:00:00 2001 From: Marek Counts Date: Tue, 7 May 2019 14:30:41 -0400 Subject: [PATCH 04/31] Updated self-hosting documentation (#13866) * updated self-hosting documentation * removed more referances * updated wording * updated self-hosting warning to be more clear about the effects of pivot has on your cluster * Update content/en/docs/setup/independent/self-hosting.md Co-Authored-By: Klaven --- .../kubeadm/implementation-details.md | 20 ------ .../setup-tools/kubeadm/kubeadm-alpha.md | 4 +- .../setup-tools/kubeadm/kubeadm-init.md | 51 --------------- .../independent/create-cluster-kubeadm.md | 1 - .../en/docs/setup/independent/self-hosting.md | 63 +++++++++++++++++++ 5 files changed, 66 insertions(+), 73 deletions(-) create mode 100644 content/en/docs/setup/independent/self-hosting.md diff --git a/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md b/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md index 8a5fe68a8c0bf..c2d73023ec16f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md +++ b/content/en/docs/reference/setup-tools/kubeadm/implementation-details.md @@ -466,26 +466,6 @@ Deploy the `kube-dns` Deployment and Service: - It's the upstream CoreDNS deployment relatively unmodified - The `kube-dns` ServiceAccount is bound to the privileges in the `system:kube-dns` ClusterRole -### Optional self-hosting - -To enable self hosting on an existing static Pod control-plane use `kubeadm alpha selfhosting pivot`. - -Self hosting basically replaces static Pods for control plane components with DaemonSets; this is achieved by executing -following procedure for API server, scheduler and controller manager static Pods: - -- Load the static Pod specification from disk -- Extract the PodSpec from static Pod manifest file -- Mutate the PodSpec to be compatible with self-hosting, and more in detail: - - Add node selector attribute targeting nodes with `node-role.kubernetes.io/master=""` label, - - Add a toleration for `node-role.kubernetes.io/master:NoSchedule` taint, - - Set `spec.DNSPolicy` to `ClusterFirstWithHostNet` -- Build a new DaemonSet object for the self-hosted component in question. Use the above mentioned PodSpec -- Create the DaemonSet resource in `kube-system` namespace. Wait until the Pods are running. -- Remove the static Pod manifest file. The kubelet will stop the original static Pod-hosted component that was running - -Please note that self hosting is not yet resilient to node restarts; this can be fixed with external checkpointing or with kubelet checkpointing - for the control plane Pods. See [self-hosting](/docs/reference/setup-tools/kubeadm/kubeadm-init/#self-hosting) for more details. - ## kubeadm join phases internal design Similarly to `kubeadm init`, also `kubeadm join` internal workflow consists of a sequence of atomic work tasks to perform. diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md index ad59320140614..b5114e0861c59 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md @@ -51,7 +51,9 @@ to enable the DynamicKubeletConfiguration feature. ## kubeadm alpha selfhosting pivot {#cmd-selfhosting} -The subcommand `pivot` can be used to conver a static Pod-hosted control plane into a self-hosted one. +The subcommand `pivot` can be used to convert a static Pod-hosted control plane into a self-hosted one. + +[Documentation](/docs/setup/independent/self-hosting) {{< tabs name="selfhosting" >}} {{< tab name="selfhosting" include="generated/kubeadm_alpha_selfhosting.md" />}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index 3c2300fab417c..57d456b907dd3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -275,57 +275,6 @@ to the kubelet. Be aware that overriding the hostname can [interfere with cloud providers](https://github.com/kubernetes/website/pull/8873). -### Self-hosting the Kubernetes control plane {#self-hosting} - -As of 1.8, you can experimentally create a _self-hosted_ Kubernetes control -plane. This means that key components such as the API server, controller -manager, and scheduler run as [DaemonSet pods](/docs/concepts/workloads/controllers/daemonset/) -configured via the Kubernetes API instead of [static pods](/docs/tasks/administer-cluster/static-pod/) -configured in the kubelet via static files. - -To create a self-hosted cluster see the `kubeadm alpha selfhosting` command. - -#### Caveats - -1. Self-hosting in 1.8 and later has some important limitations. In particular, a - self-hosted cluster _cannot recover from a reboot of the control-plane node_ - without manual intervention. - -1. A self-hosted cluster is not upgradeable using `kubeadm upgrade`. - -1. By default, self-hosted control plane Pods rely on credentials loaded from - [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) - volumes. Except for initial creation, these credentials are not managed by - kubeadm. - -1. The self-hosted portion of the control plane does not include etcd, - which still runs as a static Pod. - -#### Process - -The self-hosting bootstrap process is documented in the [kubeadm design -document](https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.9.md#optional-self-hosting). - -In summary, `kubeadm alpha selfhosting` works as follows: - - 1. Waits for this bootstrap static control plane to be running and - healthy. This is identical to the `kubeadm init` process without self-hosting. - - 1. Uses the static control plane Pod manifests to construct a set of - DaemonSet manifests that will run the self-hosted control plane. - It also modifies these manifests where necessary, for example adding new volumes - for secrets. - - 1. Creates DaemonSets in the `kube-system` namespace and waits for the - resulting Pods to be running. - - 1. Once self-hosted Pods are operational, their associated static Pods are deleted - and kubeadm moves on to install the next component. This triggers kubelet to - stop those static Pods. - - 1. When the original static control plane stops, the new self-hosted control - plane is able to bind to listening ports and become active. - ### Running kubeadm without an internet connection For running kubeadm without an internet connection you have to pre-pull the required control-plane images. diff --git a/content/en/docs/setup/independent/create-cluster-kubeadm.md b/content/en/docs/setup/independent/create-cluster-kubeadm.md index a91c7e25b1141..cd73f6482be33 100644 --- a/content/en/docs/setup/independent/create-cluster-kubeadm.md +++ b/content/en/docs/setup/independent/create-cluster-kubeadm.md @@ -44,7 +44,6 @@ but you may also build them from source for other OSes. | kubeadm alpha subcommands | alpha | | High availability | alpha | | DynamicKubeletConfig | alpha | -| Self-hosting | alpha | kubeadm's overall feature state is **GA**. Some sub-features, like the configuration diff --git a/content/en/docs/setup/independent/self-hosting.md b/content/en/docs/setup/independent/self-hosting.md new file mode 100644 index 0000000000000..090b5efd5b590 --- /dev/null +++ b/content/en/docs/setup/independent/self-hosting.md @@ -0,0 +1,63 @@ +--- +reviewers: +- sig-cluster-lifecycle +title: Configuring your kubernetes cluster to self-host the control plane +content_template: templates/concept +weight: 100 +--- + +{{% capture overview %}} + +### Self-hosting the Kubernetes control plane {#self-hosting} + +As of 1.8, you can experimentally create a _self-hosted_ Kubernetes control +plane. This means that key components such as the API server, controller +manager, and scheduler run as [DaemonSet pods](/docs/concepts/workloads/controllers/daemonset/) +configured via the Kubernetes API instead of [static pods](/docs/tasks/administer-cluster/static-pod/) +configured in the kubelet via static files. + +To create a self-hosted cluster see the `kubeadm alpha selfhosting pivot` command. + +#### Caveats + +{{< caution >}} +This feature pivots your cluster into an unsupported state, rendering kubeadm unable +to manage you cluster any longer. This includes `kubeadm upagrade`. +{{< /caution >}} + +1. Self-hosting in 1.8 and later has some important limitations. In particular, a + self-hosted cluster _cannot recover from a reboot of the control-plane node_ + without manual intervention. + +1. By default, self-hosted control plane Pods rely on credentials loaded from + [`hostPath`](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) + volumes. Except for initial creation, these credentials are not managed by + kubeadm. + +1. The self-hosted portion of the control plane does not include etcd, + which still runs as a static Pod. + +#### Process + +The self-hosting bootstrap process is documented in the [kubeadm design +document](https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.9.md#optional-self-hosting). + +In summary, `kubeadm alpha selfhosting` works as follows: + + 1. Waits for this bootstrap static control plane to be running and + healthy. This is identical to the `kubeadm init` process without self-hosting. + + 1. Uses the static control plane Pod manifests to construct a set of + DaemonSet manifests that will run the self-hosted control plane. + It also modifies these manifests where necessary, for example adding new volumes + for secrets. + + 1. Creates DaemonSets in the `kube-system` namespace and waits for the + resulting Pods to be running. + + 1. Once self-hosted Pods are operational, their associated static Pods are deleted + and kubeadm moves on to install the next component. This triggers kubelet to + stop those static Pods. + + 1. When the original static control plane stops, the new self-hosted control + plane is able to bind to listening ports and become active. From c7c55c58834e13beb78a225d8ce5e467fc06cffc Mon Sep 17 00:00:00 2001 From: Wojciech Tyczynski Date: Mon, 20 May 2019 14:13:14 +0200 Subject: [PATCH 05/31] Watch bookmarks documentation (#14379) --- .../docs/reference/using-api/api-concepts.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/content/en/docs/reference/using-api/api-concepts.md b/content/en/docs/reference/using-api/api-concepts.md index b9aaada90cca7..6d2c4b694eb62 100644 --- a/content/en/docs/reference/using-api/api-concepts.md +++ b/content/en/docs/reference/using-api/api-concepts.md @@ -87,6 +87,24 @@ For example: ... A given Kubernetes server will only preserve a historical list of changes for a limited time. Clusters using etcd3 preserve changes in the last 5 minutes by default. When the requested watch operations fail because the historical version of that resource is not available, clients must handle the case by recognizing the status code `410 Gone`, clearing their local cache, performing a list operation, and starting the watch from the `resourceVersion` returned by that new list operation. Most client libraries offer some form of standard tool for this logic. (In Go this is called a `Reflector` and is located in the `k8s.io/client-go/cache` package.) +To mitigate the impact of short history window, we introduced a concept of `bookmark` watch event. It is a special kind of event to pass an information that all changes up to a given `resourceVersion` client is requesting has already been send. Object returned in that event is of the type requested by the request, but only `resourceVersion` field is set, e.g.: + + GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245&allowWatchBookmarks=true + --- + 200 OK + Transfer-Encoding: chunked + Content-Type: application/json + { + "type": "ADDED", + "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...} + } + ... + { + "type": "BOOKMARK", + "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "12746"} } + } + +`Bookmark` events can be requested by `allowWatchBookmarks=true` option in watch requests, but clients shouldn't assume bookmarks are returned at any specific interval, nor may they assume the server will send any `bookmark` event. As of 1.15 release, it is an Alpha feature. ## Retrieving large results sets in chunks From b8759a7dbe0f90b113c4c158e65b3270a7ddb44e Mon Sep 17 00:00:00 2001 From: John Griffith Date: Thu, 30 May 2019 18:38:24 -0600 Subject: [PATCH 06/31] Add docs for volume cloning (#14591) --- .../concepts/storage/persistent-volumes.md | 27 ++++++++ .../concepts/storage/volume-pvc-datasource.md | 68 +++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 content/en/docs/concepts/storage/volume-pvc-datasource.md diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index cb2cbef8a31af..52d3b32e8a558 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -680,6 +680,33 @@ spec: storage: 10Gi ``` +## Volume Cloning + +{{< feature-state for_k8s_version="v1.15" state="alpha" >}} + +Volume clone feature was added to support CSI Volume Plugins only. For details, see [volume cloning](/docs/concepts/storage/volume-pvc-datasource/). + +To enable support for cloning a volume from a pvc data source, enable the +`VolumePVCDataSource` feature gate on the apiserver and controller-manager. + +### Create Persistent Volume Claim from an existing pvc +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloned-pvc +spec: + storageClassName: my-csi-plugin + dataSource: + name: existing-src-pvc-name + kind: PersistentVolumeClaim + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi +``` + ## Writing Portable Configuration If you're writing configuration templates or examples that run on a wide range of clusters diff --git a/content/en/docs/concepts/storage/volume-pvc-datasource.md b/content/en/docs/concepts/storage/volume-pvc-datasource.md new file mode 100644 index 0000000000000..7819b83a2bd7e --- /dev/null +++ b/content/en/docs/concepts/storage/volume-pvc-datasource.md @@ -0,0 +1,68 @@ +--- +reviewers: +- jsafrane +- saad-ali +- thockin +- msau42 +title: CSI Volume Cloning +content_template: templates/concept +weight: 30 +--- + +{{% capture overview %}} + +{{< feature-state for_k8s_version="v1.15" state="alpha" >}} +This document describes the concept of cloning existing CSI Volumes in Kubernetes. Familiarity with [Volumes](/docs/concepts/storage/volumes) is suggested. + +This feature requires VolumePVCDataSource feature gate to be enabled: + +``` +--feature-gates=VolumePVCDataSource=true +``` + + +{{% /capture %}} + + +{{% capture body %}} + +## Introduction + +The {{< glossary_tooltip text="CSI" term_id="csi" >}} Volume Cloning feature adds support for specifying existing {{< glossary_tooltip text="PVC" term_id="persistent-volume-claim" >}}s in the `dataSource` field to indicate a user would like to clone a {{< glossary_tooltip term_id="volume" >}}. + +A Clone is defined as a duplicate of an existing Kubernetes Volume that can be consumed as any standard Volume would be. The only difference is that upon provisioning, rather than creating a "new" empty Volume, the back end device creates an exact duplicate of the specified Volume. + +The implementation of cloning, from the perspective of the Kubernetes API simply adds the ability to specify an existing unbound PVC as a dataSource during new pvc creation. + +Users need to be aware of the following when using this feature: + +* Cloning support (`VolumePVCDataSource`) is only available for CSI drivers. +* Cloning support is only available for dynamic provisioners. +* CSI drivers may or may not have implemented the volume cloning functionality. +* You can only clone a PVC when it exists in the same namespace as the destination PVC (source and destination must be in the same namespace). + +## Provisioning + +Clones are provisioned just like any other PVC with the exception of adding a dataSource that references an existing PVC in the same namespace. + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: clone-of-pvc-1 + namespace: myns +spec: + capacity: + storage: 10Gi + dataSource: + kind: PersistentVolumeClaim + name: pvc-1 +``` + +The result is a new PVC with the name `clone-of-pvc-1` that has the exact same content as the specified source `pvc-1`. + +## Usage + +Upon availability of the new PVC, the cloned PVC is consumed the same as other PVC. It's also expected at this point that the newly created PVC is an independent object. It can be consumed, cloned, snapshotted, or deleted independently and without consideration for it's original dataSource PVC. This also implies that the source is not linked in any way to the newly created clone, it may also be modified or deleted without affecting the newly created clone. + +{{% /capture %}} From 817ee7ed4517bc150232102745da689efef40016 Mon Sep 17 00:00:00 2001 From: Robert Krawitz Date: Fri, 31 May 2019 11:28:28 -0400 Subject: [PATCH 07/31] Add support for quotas for ephemeral storage monitoring. (#14268) --- .../manage-compute-resources-container.md | 64 +++++++++++++++++++ .../feature-gates.md | 2 + .../federation-apiserver.md | 1 + .../kube-apiserver.md | 2 +- .../kube-controller-manager.md | 2 +- .../kube-proxy.md | 2 +- .../kube-scheduler.md | 2 +- .../command-line-tools-reference/kubelet.md | 2 +- 8 files changed, 72 insertions(+), 5 deletions(-) diff --git a/content/en/docs/concepts/configuration/manage-compute-resources-container.md b/content/en/docs/concepts/configuration/manage-compute-resources-container.md index 34f332089317e..2aebf982044d7 100644 --- a/content/en/docs/concepts/configuration/manage-compute-resources-container.md +++ b/content/en/docs/concepts/configuration/manage-compute-resources-container.md @@ -384,6 +384,70 @@ The scheduler ensures that the sum of the resource requests of the scheduled Con For container-level isolation, if a Container's writable layer and logs usage exceeds its storage limit, the Pod will be evicted. For pod-level isolation, if the sum of the local ephemeral storage usage from all containers and also the Pod's emptyDir volumes exceeds the limit, the Pod will be evicted. +### Monitoring ephemeral-storage consumption + +When local ephemeral storage is used, it is monitored on an ongoing +basis by the kubelet. The monitoring is performed by scanning each +emptyDir volume, log directories, and writable layers on a periodic +basis. Starting with Kubernetes 1.15, emptyDir volumes (but not log +directories or writable layers) may, at the cluster operator's option, +be managed by use of [project +quotas](http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/xfs-quotas.html). +Project quotas were originally implemented in XFS, and have more +recently been ported to ext4fs. Project quotas can be used for both +monitoring and enforcement; as of Kubernetes 1.15, they are available +as alpha functionality for monitoring only. + +Quotas are faster and more accurate than directory scanning. When a +directory is assigned to a project, all files created under a +directory are created in that project, and the kernel merely has to +keep track of how many blocks are in use by files in that project. If +a file is created and deleted, but with an open file descriptor, it +continues to consume space. This space will be tracked by the quota, +but will not be seen by a directory scan. + +Kubernetes uses project IDs starting from 1048576. The IDs in use are +registered in `/etc/projects` and `/etc/projid`. If project IDs in +this range are used for other purposes on the system, those project +IDs must be registered in `/etc/projects` and `/etc/projid` to prevent +Kubernetes from using them. + +To enable use of project quotas, the cluster operator must do the +following: + +* Enable the `LocalStorageCapacityIsolationFSQuotaMonitoring=true` + feature gate in the kubelet configuration. This defaults to `false` + in Kubernetes 1.15, so must be explicitly set to `true`. + +* Ensure that the root partition (or optional runtime partition) is + built with project quotas enabled. All XFS filesystems support + project quotas, but ext4 filesystems must be built specially. + +* Ensure that the root partition (or optional runtime partition) is + mounted with project quotas enabled. + +#### Building and mounting filesystems with project quotas enabled + +XFS filesystems require no special action when building; they are +automatically built with project quotas enabled. + +Ext4fs filesystems must be built with quotas enabled, then they must +be enabled in the filesystem: + +``` +% sudo mkfs.ext4 other_ext4fs_args... -E quotatype=prjquota /dev/block_device +% sudo tune2fs -O project -Q prjquota /dev/block_device + +``` + +To mount the filesystem, both ext4fs and XFS require the `prjquota` +option set in `/etc/fstab`: + +``` +/dev/block_device /var/kubernetes_data defaults,prjquota 0 0 +``` + + ## Extended resources Extended resources are fully-qualified resource names outside the diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 8807d19b2d916..574b996a6ddd2 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -106,6 +106,7 @@ different Kubernetes components. | `KubeletPodResources` | `false` | Alpha | 1.13 | | | `LocalStorageCapacityIsolation` | `false` | Alpha | 1.7 | 1.9 | | `LocalStorageCapacityIsolation` | `true` | Beta| 1.10 | | +| `LocalStorageCapacityIsolationFSQuotaMonitoring` | `false` | Alpha| 1.15 | | | `MountContainers` | `false` | Alpha | 1.9 | | | `MountPropagation` | `false` | Alpha | 1.8 | 1.9 | | `MountPropagation` | `true` | Beta | 1.10 | 1.11 | @@ -276,6 +277,7 @@ Each feature gate is designed for enabling/disabling a specific feature: - `KubeletPodResources`: Enable the kubelet's pod resources grpc endpoint. See [Support Device Monitoring](https://git.k8s.io/community/keps/sig-node/compute-device-assignment.md) for more details. - `LocalStorageCapacityIsolation`: Enable the consumption of [local ephemeral storage](/docs/concepts/configuration/manage-compute-resources-container/) and also the `sizeLimit` property of an [emptyDir volume](/docs/concepts/storage/volumes/#emptydir). +- `LocalStorageCapacityIsolationFSQuotaMonitoring`: When `LocalStorageCapacityIsolation` is enabled for [local ephemeral storage](/docs/concepts/configuration/manage-compute-resources-container/) and the backing filesystem for [emptyDir volumes](/docs/concepts/storage/volumes/#emptydir) supports project quotas and they are enabled, use project quotas to monitor [emptyDir volume](/docs/concepts/storage/volumes/#emptydir) storage consumption rather than filesystem walk for better performance and accuracy. - `MountContainers`: Enable using utility containers on host as the volume mounter. - `MountPropagation`: Enable sharing volume mounted by one container to other containers or pods. For more details, please see [mount propagation](/docs/concepts/storage/volumes/#mount-propagation). diff --git a/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md b/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md index c219528332a90..46222dedee6c5 100644 --- a/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/federation-apiserver.md @@ -95,6 +95,7 @@ HugePages=true|false (BETA - default=true) HyperVContainer=true|false (ALPHA - default=false) Initializers=true|false (ALPHA - default=false) LocalStorageCapacityIsolation=true|false (ALPHA - default=false) +LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false) MountContainers=true|false (ALPHA - default=false) MountPropagation=true|false (ALPHA - default=false) PVCProtection=true|false (ALPHA - default=false) diff --git a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md index 433fba1bd30f5..01c433858d839 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-apiserver.md @@ -577,7 +577,7 @@ kube-apiserver [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) diff --git a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md index 2f80707248ec5..a33e7db33715d 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md @@ -343,7 +343,7 @@ kube-controller-manager [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) diff --git a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md index 209e694c293d6..3e56658ecf901 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-proxy.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-proxy.md @@ -111,7 +111,7 @@ kube-proxy [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) diff --git a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md index 511ee18e51359..c0a2639863832 100644 --- a/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md +++ b/content/en/docs/reference/command-line-tools-reference/kube-scheduler.md @@ -152,7 +152,7 @@ kube-scheduler [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) + A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
AllAlpha=true|false (ALPHA - default=false)
AppArmor=true|false (BETA - default=true)
AttachVolumeLimit=true|false (BETA - default=true)
BalanceAttachedNodeVolumes=true|false (ALPHA - default=false)
BlockVolume=true|false (BETA - default=true)
BoundServiceAccountTokenVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CRIContainerLogRotation=true|false (BETA - default=true)
CSIBlockVolume=true|false (ALPHA - default=false)
CSIDriverRegistry=true|false (ALPHA - default=false)
CSINodeInfo=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomPodDNS=true|false (BETA - default=true)
CustomResourceSubresources=true|false (BETA - default=true)
CustomResourceValidation=true|false (BETA - default=true)
CustomResourceWebhookConversion=true|false (ALPHA - default=false)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (BETA - default=true)
DryRun=true|false (BETA - default=true)
DynamicAuditing=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (BETA - default=true)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandInUsePersistentVolumes=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (BETA - default=true)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (BETA - default=true)
HyperVContainer=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletPodResources=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
NodeLease=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (BETA - default=true)
PodPriority=true|false (BETA - default=true)
PodReadinessGates=true|false (BETA - default=true)
PodShareProcessNamespace=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
QOSReserved=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
ResourceQuotaScopeSelectors=true|false (BETA - default=true)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
RunAsGroup=true|false (ALPHA - default=false)
RuntimeClass=true|false (ALPHA - default=false)
SCTPSupport=true|false (ALPHA - default=false)
ScheduleDaemonSetPods=true|false (BETA - default=true)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportPodPidsLimit=true|false (ALPHA - default=false)
Sysctls=true|false (BETA - default=true)
TTLAfterFinished=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
TokenRequest=true|false (BETA - default=true)
TokenRequestProjection=true|false (BETA - default=true)
ValidateProxyRedirects=true|false (ALPHA - default=false)
VolumeSnapshotDataSource=true|false (ALPHA - default=false)
VolumeSubpathEnvExpansion=true|false (ALPHA - default=false) diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index 02a2893eec41b..f19da3d42e8a2 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -545,7 +545,7 @@ kubelet [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ReadOnlyAPIDataVolumes=true|false
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
ServiceProxyAllowExternalIPs=true|false
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
VolumeScheduling=true|false (ALPHA - default=false)
VolumeSubpath=true|false
+ A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ReadOnlyAPIDataVolumes=true|false
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
ServiceProxyAllowExternalIPs=true|false
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
VolumeScheduling=true|false (ALPHA - default=false)
VolumeSubpath=true|false
From 7acab64de6764db7ccf14f62ad4a15144f462d15 Mon Sep 17 00:00:00 2001 From: Xing Yang Date: Mon, 3 Jun 2019 20:39:03 -0400 Subject: [PATCH 08/31] Add documentation for PVC in use protection (#14700) This PR adds doc to describe PVC in use protection for PVC actively in use as a snapshot source. --- content/en/docs/concepts/storage/volume-snapshots.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/content/en/docs/concepts/storage/volume-snapshots.md b/content/en/docs/concepts/storage/volume-snapshots.md index 6b57db96aabdb..cf105c16d815f 100644 --- a/content/en/docs/concepts/storage/volume-snapshots.md +++ b/content/en/docs/concepts/storage/volume-snapshots.md @@ -64,6 +64,12 @@ A user creates, or has already created in the case of dynamic provisioning, a `V VolumeSnapshots will remain unbound indefinitely if a matching VolumeSnapshotContent does not exist. VolumeSnapshots will be bound as matching VolumeSnapshotContents become available. +### Persistent Volume Claim in Use Protection + +The purpose of the Persistent Volume Claim Object in Use Protection feature is to ensure that in-use PVC API objects are not removed from the system (as this may result in data loss). + +If a PVC is in active use by a snapshot as a source to create the snapshot, the PVC is in-use. If a user deletes a PVC API object in active use as a snapshot source, the PVC object is not removed immediately. Instead, removal of the PVC object is postponed until the PVC is no longer actively used by any snapshots. A PVC is no longer used as a snapshot source when `ReadyToUse` of the snapshot `Status` becomes `true`. + ### Delete Deletion removes both the `VolumeSnapshotContent` object from the Kubernetes API, as well as the associated storage asset in the external infrastructure. From 40ed4669b66ef5d507900d1a6e4951048d8b618f Mon Sep 17 00:00:00 2001 From: Hemant Kumar Date: Mon, 3 Jun 2019 21:23:01 -0400 Subject: [PATCH 09/31] Add a placeholder doc (#14643) * Add a placeholder doc * Update persistent-volumes.md --- .../concepts/storage/persistent-volumes.md | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/content/en/docs/concepts/storage/persistent-volumes.md b/content/en/docs/concepts/storage/persistent-volumes.md index 52d3b32e8a558..fa8cb89625d10 100644 --- a/content/en/docs/concepts/storage/persistent-volumes.md +++ b/content/en/docs/concepts/storage/persistent-volumes.md @@ -226,15 +226,9 @@ CSI volume expansion requires enabling `ExpandCSIVolumes` feature gate and also You can only resize volumes containing a file system if the file system is XFS, Ext3, or Ext4. -When a volume contains a file system, the file system is only resized when a new Pod is started using -the `PersistentVolumeClaim` in ReadWrite mode. Therefore, if a pod or deployment is using a volume and -you want to expand it, you need to delete or recreate the pod after the volume has been expanded by the cloud provider in the controller-manager. You can check the status of resize operation by running the `kubectl describe pvc` command: - -``` -kubectl describe pvc -``` - -If the `PersistentVolumeClaim` has the status `FileSystemResizePending`, it is safe to recreate the pod using the PersistentVolumeClaim. +When a volume contains a file system, the file system is only resized when a new Pod is using +the `PersistentVolumeClaim` in ReadWrite mode. File system expansion is either done when Pod is starting up +or is done when Pod is running and underlying file system supports online expansion. FlexVolumes allow resize if the driver is set with the `RequiresFSResize` capability to true. The FlexVolume can be resized on pod restart. @@ -243,14 +237,15 @@ The FlexVolume can be resized on pod restart. #### Resizing an in-use PersistentVolumeClaim -Expanding in-use PVCs is an alpha feature. To use it, enable the `ExpandInUsePersistentVolumes` feature gate. +Expanding in-use PVCs is a beta feature and is enabled by default via `ExpandInUsePersistentVolumes` feature gate. In this case, you don't need to delete and recreate a Pod or deployment that is using an existing PVC. Any in-use PVC automatically becomes available to its Pod as soon as its file system has been expanded. This feature has no effect on PVCs that are not in use by a Pod or deployment. You must create a Pod which uses the PVC before the expansion can complete. -Expanding in-use PVCs for FlexVolumes is added in release 1.13. To enable this feature use `ExpandInUsePersistentVolumes` and `ExpandPersistentVolumes` feature gates. The `ExpandPersistentVolumes` feature gate is already enabled by default. If the `ExpandInUsePersistentVolumes` is set, FlexVolume can be resized online without pod restart. - + +Similar to other volume types - FlexVolume volumes can also be expanded when in-use by a pod. + {{< note >}} **Note:** FlexVolume resize is possible only when the underlying driver supports resize. {{< /note >}} From 6e75d5c4de9a47ac3d9fd9e5209c83cc89e7c752 Mon Sep 17 00:00:00 2001 From: Deep Debroy Date: Thu, 6 Jun 2019 08:18:10 +0300 Subject: [PATCH 10/31] Update CSI migration docs with Azure Disk/File details (#14707) Signed-off-by: Deep Debroy --- content/en/docs/concepts/storage/volumes.md | 24 ++++++++++++++++++- .../feature-gates.md | 4 ++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index bcf6abe2d9c72..abcbd26e1afe4 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -166,6 +166,17 @@ A `azureDisk` is used to mount a Microsoft Azure [Data Disk](https://azure.micro More details can be found [here](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/azure_disk/README.md). +#### CSI Migration + +{{< feature-state for_k8s_version="v1.15" state="alpha" >}} + +The CSI Migration feature for azureDisk, when enabled, shims all plugin operations +from the existing in-tree plugin to the `disk.csi.azure.com` Container +Storage Interface (CSI) Driver. In order to use this feature, the [Azure Disk CSI +Driver](https://github.com/kubernetes-sigs/azuredisk-csi-driver) +must be installed on the cluster and the `CSIMigration` and `CSIMigrationAzureDisk` +Alpha features must be enabled. + ### azureFile {#azurefile} A `azureFile` is used to mount a Microsoft Azure File Volume (SMB 2.1 and 3.0) @@ -173,6 +184,17 @@ into a Pod. More details can be found [here](https://github.com/kubernetes/examples/tree/{{< param "githubbranch" >}}/staging/volumes/azure_file/README.md). +#### CSI Migration + +{{< feature-state for_k8s_version="v1.15" state="alpha" >}} + +The CSI Migration feature for azureFile, when enabled, shims all plugin operations +from the existing in-tree plugin to the `file.csi.azure.com` Container +Storage Interface (CSI) Driver. In order to use this feature, the [Azure File CSI +Driver](https://github.com/kubernetes-sigs/azurefile-csi-driver) +must be installed on the cluster and the `CSIMigration` and `CSIMigrationAzureFile` +Alpha features must be enabled. + ### cephfs {#cephfs} A `cephfs` volume allows an existing CephFS volume to be @@ -1344,7 +1366,7 @@ configuration changes to existing Storage Classes, PVs or PVCs (referring to in-tree plugins) when transitioning to a CSI driver that supersedes an in-tree plugin. In the alpha state, the operations and features that are supported include -provisioning/delete, attach/detach and mount/unmount of volumes with `volumeMode` set to `filesystem` +provisioning/delete, attach/detach, mount/unmount and resizing of volumes. In-tree plugins that support CSI Migration and have a corresponding CSI driver implemented are listed in the "Types of Volumes" section above. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 574b996a6ddd2..90436cfc733cc 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -62,6 +62,8 @@ different Kubernetes components. | `CSIInlineVolume` | `false` | Alpha | 1.14 | - | | `CSIMigration` | `false` | Alpha | 1.14 | | | `CSIMigrationAWS` | `false` | Alpha | 1.14 | | +| `CSIMigrationAzureDisk` | `false` | Alpha | 1.15 | | +| `CSIMigrationAzureFile` | `false` | Alpha | 1.15 | | | `CSIMigrationGCE` | `false` | Alpha | 1.14 | | | `CSIMigrationOpenStack` | `false` | Alpha | 1.14 | | | `CSINodeInfo` | `false` | Alpha | 1.12 | 1.13 | @@ -228,6 +230,8 @@ Each feature gate is designed for enabling/disabling a specific feature: - `CSIDriverRegistry`: Enable all logic related to the CSIDriver API object in csi.storage.k8s.io. - `CSIMigration`: Enables shims and translation logic to route volume operations from in-tree plugins to corresponding pre-installed CSI plugins - `CSIMigrationAWS`: Enables shims and translation logic to route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin +- `CSIMigrationAzureDisk`: Enables shims and translation logic to route volume operations from the Azure-Disk in-tree plugin to Azure Disk CSI plugin +- `CSIMigrationAzureFile`: Enables shims and translation logic to route volume operations from the Azure-File in-tree plugin to Azure File CSI plugin - `CSIMigrationGCE`: Enables shims and translation logic to route volume operations from the GCE-PD in-tree plugin to PD CSI plugin - `CSIMigrationOpenStack`: Enables shims and translation logic to route volume operations from the Cinder in-tree plugin to Cinder CSI plugin - `CSINodeInfo`: Enable all logic related to the CSINodeInfo API object in csi.storage.k8s.io. From 19e9d312ea609491661c4618cc65256ded7b6e9b Mon Sep 17 00:00:00 2001 From: Fabrizio Pandini Date: Thu, 6 Jun 2019 07:20:10 +0200 Subject: [PATCH 11/31] kubeadm-1.15-certs-renewal (#14716) --- .../kubeadm/kubeadm-certs.md | 168 +++++++++++++----- 1 file changed, 119 insertions(+), 49 deletions(-) diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index 5afdcffa150c0..55aa0902d0d87 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -7,46 +7,112 @@ content_template: templates/task {{% capture overview %}} -This page explains how to manage certificates manually with kubeadm. +{{< feature-state for_k8s_version="v1.15" state="stable" >}} + +Client certificates generated by [kubeadm](/docs/reference/setup-tools/kubeadm/kubeadm/) expire after 1 year. This page explains how to manage certificate renewals with kubeadm. {{% /capture %}} {{% capture prerequisites %}} -These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If kubeadm with the default configuration satisfies your needs, you should let kubeadm manage certificates instead. +Be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/certificates/). -You should be familiar with [PKI certificates and requirements in Kubernetes](/docs/setup/certificates/). +Have a working Kubernetes cluster installed using kubeadm with your certificates stored in `/etc/kubernetes/pki` folder; in case you are using a different location, the following can be adapted accordingly. {{% /capture %}} {{% capture steps %}} -## Renew certificates with the certificates API +## Check certificate expiration + +`check-expiration` can be used to check certificate expiration. + +``` +kubeadm alpha certs check-expiration +``` + +The output is similar to this: + +``` +CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED +admin.conf May 15, 2020 13:03 UTC 364d false +apiserver May 15, 2020 13:00 UTC 364d false +apiserver-etcd-client May 15, 2020 13:00 UTC 364d false +apiserver-kubelet-client May 15, 2020 13:00 UTC 364d false +controller-manager.conf May 15, 2020 13:03 UTC 364d false +etcd-healthcheck-client May 15, 2020 13:00 UTC 364d false +etcd-peer May 15, 2020 13:00 UTC 364d false +etcd-server May 15, 2020 13:00 UTC 364d false +front-proxy-client May 15, 2020 13:00 UTC 364d false +scheduler.conf May 15, 2020 13:03 UTC 364d false +``` + +The command shows expiration/residual time for the client certificates in the `/etc/kubernetes/pki` folder and for the client certificate embedded in the KUBECONFIG files used by kubeadm (`admin.conf`, `controller-manager.conf` and `scheduler.conf`). + +Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the user should take care of managing certificate renewal manually/using other tools. + +{{< warning >}} +`kubeadm` cannot manage certificates signed by an external CA. +{{< /warning >}} + +{{< note >}} +`kubelet.conf` is not included in the list above because kubeadm configures kubelet for automatic certificate renewal. +{{< /note >}} + +## Automatic certificate renewal + +`kubeadm` renews all the certificates during control plane [upgrade](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/). + +This feature is designed for addressing the simplest use cases; +if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure. + +{{< note >}} +It is a best practice to upgrade your cluster frequently in order to stay secure. +{{< /note >}} + +If you have more complex requirements for certificate renewal, you can opt out from the default behavior by passing `--certificate-renewal=false` to `kubeadm upgrade apply` or to `kubeadm upgrade node`. + + +## Manual certificate renewal + +You can renew your certificates manually at any time with the `kubeadm alpha certs renew` command. + +This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in `/etc/kubernetes/pki`. + +{{< warning >}} +If you are running an HA cluster, this command needs to be executed on all the control-plane nodes. +{{< /warning >}} + +{{< note >}} +`alpha certs renew` uses the existing certificates as the authoritative source for attributes (Common Name, Organization, SAN, etc.) instead of the kubeadm-config ConfigMap. It is strongly recommended to keep them both in sync. +{{< /note >}} + +`kubeadm alpha certs renew` provides the following options: -The Kubernetes certificates normally reach their expiration date after one year. +- `--use-api` allows you to use the Kubernetes certificates API for signing certificates (instead of the local CA/front-proxy-CA); see next paragraphs for more information. -Kubeadm can renew certificates with the `kubeadm alpha certs renew` commands; you should run these commands on control-plane nodes only. +- `--csr-only` can be used to renew certificats with an external CA by generating certificate signing requests (without actually renewing certificates in place); see next paragraph for more information. -Typically this is done by loading on-disk CA certificates and keys and using them to issue new certificates. -This approach works well if your certificate tree is self-contained. However, if your certificates are externally -managed, you might need a different approach. +- It's also possible to renew a single certificate instead of all. -As an alternative, Kubernetes provides its own [API for managing certificates][manage-tls]. -With kubeadm, you can use this API by running `kubeadm alpha certs renew --use-api`. +## Renew certificates with the Kubernetes certificates API -## Set up a signer +This section provide more details about how to execute manual certificate renewal using the Kubernetes certificates API. + +{{< caution >}} +These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead. +{{< /caution >}} + +### Set up a signer The Kubernetes Certificate Authority does not work out of the box. You can configure an external signer such as [cert-manager][cert-manager-issuer], or you can use the build-in signer. -The built-in signer is part of [`kube-controller-manager`][kcm]. -To activate the build-in signer, you pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` arguments. -You pass these arguments in any of the following ways: +The built-in signer is part of [`kube-controller-manager`][kcm]. -* Edit `/etc/kubernetes/manifests/kube-controller-manager.yaml` to add the arguments to the command. - Remember that your changes could be overwritten when you upgrade. +To activate the build-in signer, you must pass the `--cluster-signing-cert-file` and `--cluster-signing-key-file` flags. -* If you're creating a new cluster, you can use a kubeadm [configuration file][config]: +If you're creating a new cluster, you can use a kubeadm [configuration file][config]: ```yaml apiVersion: kubeadm.k8s.io/v1beta1 @@ -57,73 +123,77 @@ You pass these arguments in any of the following ways: cluster-signing-key-file: /etc/kubernetes/pki/ca.key ``` -* You can also upload a config file using [`kubeadm config upload from-files`][config-upload] - [cert-manager-issuer]: https://cert-manager.readthedocs.io/en/latest/tutorials/ca/creating-ca-issuer.html [kcm]: /docs/reference/command-line-tools-reference/kube-controller-manager/ -[config]: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1 -[config-upload]: /docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-from-file +[config]: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2 + +### Create certificate signing requests (CSR) -### Approve requests +You can create the certificate signing requests for the Kubernetes certificates API with `kubeadm alpha certs renew --use-api`. -If you set up an external signer such as [cert-manager][cert-manager], certificate signing requests (CSRs) are automatically approved. -Otherwise, you must manually approve certificates with the [`kubectl certificate`][certs] command. -The following kubeadm command outputs the name of the certificate to approve, then blocks and waits for approval to occur: +The command outputs the name of the certificate to approve, then blocks and waits for approval to occur. e.g.: ```shell sudo kubeadm alpha certs renew apiserver --use-api & ``` +The output is similar to this: ``` [1] 2890 [certs] certificate request "kubeadm-cert-kube-apiserver-ld526" created ``` + +### Approve certificate signing requests (CSR) + +If you set up an external signer, certificate signing requests (CSRs) are automatically approved. + +Otherwise, you must manually approve certificates with the [`kubectl certificate`][certs] command. e.g. + ```shell kubectl certificate approve kubeadm-cert-kube-apiserver-ld526 +``` +The output is similar to this: +```shell certificatesigningrequest.certificates.k8s.io/kubeadm-cert-kube-apiserver-ld526 approved -[1]+ Done sudo kubeadm alpha certs renew apiserver --use-api ``` You can view a list of pending certificates with `kubectl get csr`. -[manage-tls]: /docs/tasks/tls/managing-tls-in-a-cluster/ -[cert-manager]: https://github.com/jetstack/cert-manager -[certs]: /docs/reference/generated/kubectl/kubectl-commands#certificate +## Renew certificates with external CA -## Certificate requests with kubeadm +This section provide more details about how to execute manual certificate renewal using an external CA. -To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs). -A CSR represents a request to a CA for a signed certificate for a client. -In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR. +{{< caution >}} +These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead. +{{< /caution >}} -You can create an individual CSR with `kubeadm init phase certs apiserver --csr-only`. -The `--csr-only` flag can be applied only to individual phases. After [all certificates are in place][certs], you can run `kubeadm init --external-ca`. +### Create certificate signing requests (CSR) -You can pass in a directory with `--csr-dir` to output the CSRs to the specified location. -If `--csr-dir` is not specified, the default certificate directory (`/etc/kubernetes/pki`) is used. -Both the CSR and the accompanying private key are given in the output. After a certificate is signed, the certificate and the private key must be copied to the PKI directory (by default `/etc/kubernetes/pki`). +To better integrate with external CAs, kubeadm can produce certificate signing requests (CSRs). -### Renew certificates +A CSR represents a request to a CA for a signed certificate for a client. + +You can create certificate signing requests with `kubeadm alpha certs renew --csr-only`. + +Both the CSR and the accompanying private key are given in the output; you can pass in a directory with `--csr-dir` to output the CSRs to the specified location. -Certificates can be renewed with `kubeadm alpha certs renew --csr-only`. -As with `kubeadm init`, an output directory can be specified with the `--csr-dir` flag. -To use the new certificates, copy the signed certificate and private key into the PKI directory (by default `/etc/kubernetes/pki`) +### Renew certificates -## Cert usage +A CSR contains a certificate's name, domain(s), and IPs, but it does not specify usages. -A CSR contains a certificate's name, domains, and IPs, but it does not specify usages. It is the responsibility of the CA to specify [the correct cert usages][cert-table] when issuing a certificate. - + * In `openssl` this is done with the [`openssl ca` command][openssl-ca]. * In `cfssl` you specify [usages in the config file][cfssl-usages] -## CA selection - -Kubeadm sets up [three CAs][cert-cas] by default. Make sure to sign the CSRs with a corresponding CA. +After a certificate is signed using your preferred method, the certificate and the private key must be copied to the PKI directory (by default `/etc/kubernetes/pki`). [openssl-ca]: https://superuser.com/questions/738612/openssl-ca-keyusage-extension [cfssl-usages]: https://github.com/cloudflare/cfssl/blob/master/doc/cmd/cfssl.txt#L170 [certs]: /docs/setup/certificates [cert-cas]: /docs/setup/certificates/#single-root-ca [cert-table]: /docs/setup/certificates/#all-certificates +[manage-tls]: /docs/tasks/tls/managing-tls-in-a-cluster/ +[cert-manager]: https://github.com/jetstack/cert-manager +[certs]: /docs/reference/generated/kubectl/kubectl-commands#certificate {{% /capture %}} From e7b5f0e9f0c95c1a29cb233ddea2121ed489c6fb Mon Sep 17 00:00:00 2001 From: David Ashpole Date: Wed, 5 Jun 2019 22:28:10 -0700 Subject: [PATCH 12/31] move podresources endpoint to beta (#14622) --- .../extend-kubernetes/compute-storage-net/device-plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md index cf9c8fdf57f36..9c7e97934f31b 100644 --- a/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md +++ b/content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md @@ -164,7 +164,7 @@ DaemonSet, `/var/lib/kubelet/pod-resources` must be mounted as a in the plugin's [PodSpec](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#podspec-v1-core). -Support for the "PodResources service" is still in alpha. +Support for the "PodResources service" is in beta, and is enabled by default. ## Examples From 3ad640e8bc3f7b553ad55a2d48a0b52e7f90963e Mon Sep 17 00:00:00 2001 From: Jordan Liggitt Date: Mon, 10 Jun 2019 00:15:05 -0400 Subject: [PATCH 13/31] Add webhook admission outline (#14671) --- .../extensible-admission-controllers.md | 645 +++++++++++++++++- content/en/docs/setup/version-skew-policy.md | 2 +- 2 files changed, 616 insertions(+), 31 deletions(-) diff --git a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md index d2fb12a3e84b3..f9555caa514d5 100644 --- a/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md +++ b/content/en/docs/reference/access-authn-authz/extensible-admission-controllers.md @@ -2,45 +2,40 @@ reviewers: - smarterclayton - lavalamp -- whitlockjc - caesarxuchao - deads2k - liggitt - mbohlool +- jpbetz title: Dynamic Admission Control content_template: templates/concept weight: 40 --- {{% capture overview %}} -The [admission controllers documentation](/docs/reference/access-authn-authz/admission-controllers/) -introduces how to use standard, plugin-style admission controllers. However, -plugin admission controllers are not flexible enough for all use cases, due to -the following: - -* They need to be compiled into kube-apiserver. -* They are only configurable when the apiserver starts up. - -*Admission Webhooks* (beta in 1.9) addresses these limitations. It allows -admission controllers to be developed out-of-tree and configured at runtime. - -This page describes how to use Admission Webhooks. - +In addition to [compiled-in admission plugins](/docs/reference/access-authn-authz/admission-controllers/), +admission plugins can be developed as extensions and run as webhooks configured at runtime. +This page describes how to build, configure, and use admission webhooks. {{% /capture %}} {{% capture body %}} -### What are admission webhooks? +## What are admission webhooks? Admission webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, [validating admission Webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook) and [mutating admission webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook). -With validating admission Webhooks, you may reject requests to enforce custom -admission policies. With mutating admission Webhooks, you may change requests to -enforce custom defaults. +Mutating admission Webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults. +After all object modifications are complete, and after the incoming object is validated by the API server, +validating admission webhooks are invoked and can reject requests to enforce custom policies. + +{{< note >}} +Admission webhooks that need to guarantee they see the final state of the object in order to enforce policy +should use a validating admission webhook, since objects can be modified after being seen by mutating webhooks. +{{< /note >}} -### Experimenting with admission webhooks +## Experimenting with admission webhooks Admission webhooks are essentially part of the cluster control-plane. You should write and deploy them with great caution. Please read the [user @@ -64,15 +59,12 @@ In the following, we describe how to quickly experiment with admission webhooks. Please refer to the implementation of the [admission webhook server](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/main.go) that is validated in a Kubernetes e2e test. The webhook handles the -`admissionReview` requests sent by the apiservers, and sends back its decision -wrapped in `admissionResponse`. +`AdmissionReview` request sent by the apiservers, and sends back its decision +as an `AdmissionReview` object in the same version it received. + +See the [webhook request](#request) section for details on the data sent to webhooks. -the `admissionReview` request can have different versions (e.g. v1beta1 or `v1` in a future version). -The webhook can define what version they accept using `admissionReviewVersions` field. API server -will try to use first version in the list which it supports. If none of the versions specified -in this list supported by API server, validation will fail for this object. If the webhook -configuration has already been persisted, calls to the webhook will fail and be -subject to the failure policy. +See the [webhook response](#response) section for the data expected from webhooks. The example admission webhook server leaves the `ClientAuth` field [empty](https://github.com/kubernetes/kubernetes/blob/v1.13.0/test/images/webhook/config.go#L47-L48), @@ -96,12 +88,12 @@ your [webhook client configurations](https://github.com/kubernetes/kubernetes/bl You can dynamically configure what resources are subject to what admission webhooks via -[ValidatingWebhookConfiguration](https://github.com/kubernetes/kubernetes/blob/v1.13.0/staging/src/k8s.io/api/admissionregistration/v1beta1/types.go#L84) +[ValidatingWebhookConfiguration](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#validatingwebhookconfiguration-v1beta1-admissionregistration-k8s-io) or -[MutatingWebhookConfiguration](https://github.com/kubernetes/kubernetes/blob/v1.13.0/staging/src/k8s.io/api/admissionregistration/v1beta1/types.go#L114). +[MutatingWebhookConfiguration](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#mutatingwebhookconfiguration-v1beta1-admissionregistration-k8s-io). The following is an example `validatingWebhookConfiguration`, a mutating webhook -configuration is similar. +configuration is similar. See the [webhook configuration](#webhook-configuration) section for details about each config field. ```yaml apiVersion: admissionregistration.k8s.io/v1beta1 @@ -185,6 +177,7 @@ plugins: The schema of `admissionConfiguration` is defined [here](https://github.com/kubernetes/kubernetes/blob/v1.13.0/staging/src/k8s.io/apiserver/pkg/apis/apiserver/v1alpha1/types.go#L27). +See the [webhook configuration](#webhook-configuration) section for details about each config field. * In the kubeConfig file, provide the credentials: @@ -210,4 +203,596 @@ users: ``` Of course you need to set up the webhook server to handle these authentications. + +## Webhook request and response + +### Request + +Webhooks are sent a POST request, with `Content-Type: application/json`, +with an `AdmissionReview` API object in the `admission.k8s.io` API group +serialized to JSON as the body. + +Webhooks can specify what versions of `AdmissionReview` objects they accept +with the `admissionReviewVersions` field in their configuration: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + admissionReviewVersions: ["v1beta1"] + ... +``` + +If no `admissionReviewVersions` are specified, the default when creating +`admissionregistration.k8s.io/v1beta1` webhook configurations is `v1beta1`. + +API servers send the first `AdmissionReview` version in the `admissionReviewVersions` list they support. +If none of the versions in the list are supported by the API server, the configuration will not be allowed to be created. +If an API server encounters a webhook configuration that was previously created and does not support any of the `AdmissionReview` +versions the API server knows how to send, attempts to call to the webhook will fail and be subject to the [failure policy](#failure-policy). + +This example shows the data contained in an `AdmissionReview` object +for a request to update the `scale` subresource of an `apps/v1` `Deployment`: + +```json +{ + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "request": { + // Random uid uniquely identifying this admission call + "uid": "705ab4f5-6393-11e8-b7cc-42010a800002", + + // Fully-qualified group/version/kind of the incoming object + "kind": {"group":"autoscaling","version":"v1","kind":"Scale"}, + // Fully-qualified group/version/kind of the resource being modified + "resource": {"group":"apps","version":"v1","resource":"deployments"}, + // subresource, if the request is to a subresource + "subResource": "scale", + + // Fully-qualified group/version/kind of the incoming object in the original request to the API server. + // This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the + // original request to the API server was converted to a version the webhook registered for. + // Only sent by v1.15+ API servers. + "requestKind": {"group":"autoscaling","version":"v1","kind":"Scale"}, + // Fully-qualified group/version/kind of the resource being modified in the original request to the API server. + // This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the + // original request to the API server was converted to a version the webhook registered for. + // Only sent by v1.15+ API servers. + "requestResource": {"group":"apps","version":"v1","resource":"deployments"}, + // subresource, if the request is to a subresource + // This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the + // original request to the API server was converted to a version the webhook registered for. + // Only sent by v1.15+ API servers. + "requestSubResource": "scale", + + // Name of the resource being modified + "name": "my-deployment", + // Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object) + "namespace": "my-namespace", + + // operation can be CREATE, UPDATE, DELETE, or CONNECT + "operation": "UPDATE", + + "userInfo": { + // Username of the authenticated user making the request to the API server + "username": "admin", + // UID of the authenticated user making the request to the API server + "uid": "014fbff9a07c", + // Group memberships of the authenticated user making the request to the API server + "groups": ["system:authenticated","my-admin-group"], + // Arbitrary extra info associated with the user making the request to the API server. + // This is populated by the API server authentication layer and should be included + // if any SubjectAccessReview checks are performed by the webhook. + "extra": { + "some-key":["some-value1", "some-value2"] + } + }, + + // object is the new object being admitted. + // It is null for DELETE operations. + "object": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, + // oldObject is the existing object. + // It is null for CREATE and CONNECT operations (and for DELETE operations in API servers prior to v1.15.0) + "oldObject": {"apiVersion":"autoscaling/v1","kind":"Scale",...}, + // options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions. + // It is null for CONNECT operations. + // Only sent by v1.15+ API servers. + "options": {"apiVersion":"meta.k8s.io/v1","kind":"UpdateOptions",...}, + + // dryRun indicates the API request is running in dry run mode and will not be persisted. + // Webhooks with side effects should avoid actuating those side effects when dryRun is true. + // See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details. + "dryRun": false + } +} +``` + +### Response + +Webhooks respond with a 200 HTTP status code, `Content-Type: application/json`, +and a body containing an `AdmissionReview` object (in the same version they were sent), +with the `response` stanza populated, serialized to JSON. + +At a minimum, the `response` stanza must contain the following fields: +* `uid`, copied from the `request.uid` sent to the webhook +* `allowed`, either set to `true` or `false` + +Example of a minimal response from a webhook to allow a request: +```json +{ + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "response": { + "uid": "", + "allowed": true + } +} +``` + +Example of a minimal response from a webhook to forbid a request: +```json +{ + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "response": { + "uid": "", + "allowed": false + } +} +``` + +When rejecting a request, the webhook can customize the http code and message returned to the user using the `status` field. +The specified status object is returned to the user. +See [API documentation](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#status-v1-meta) for details about the status type. +Example of a response to forbid a request, customizing the HTTP status code and message presented to the user: +```json +{ + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "response": { + "uid": "", + "allowed": false, + "status": { + "code": 403, + "message": "You cannot do this because it is Tuesday and your name starts with A" + } + } +} +``` + +When allowing a request, a mutating admission webhook may optionally modify the incoming object as well. +This is done using the `patch` and `patchType` fields in the response. +The only currently supported `patchType` is `JSONPatch`. +See [JSON patch](http://jsonpatch.com/) documentation for more details. +For `patchType: JSONPatch`, the `patch` field contains a base64-encoded array of JSON patch operations. + +As an example, a single patch operation that would set `spec.replicas` would be `[{"op": "add", "path": "/spec/replicas", "value": 3}]` + +Base64-encoded, this would be `W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=` + +So a webhook response to add that label would be: +```json +{ + "apiVersion": "admission.k8s.io/v1beta1", + "kind": "AdmissionReview", + "response": { + "uid": "", + "allowed": true, + "patchType": "JSONPatch", + "patch": "W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=" + } +} +``` + +## Webhook configuration + +To register admission webhooks, create `MutatingWebhookConfiguration` or `ValidatingWebhookConfiguration` API objects. +Each configuration can contain one or more webhooks. Each webhook defines the following things. + +### Matching requests: rules + +Each webhook must specify a list of rules used to determine if a request to the API server should be sent to the webhook. +Each rule specifies one or more operations, apiGroups, apiVersions, and resources, and a resource scope: + +* `operations` lists one or more operations to match. Can be `"CREATE"`, `"UPDATE"`, `"DELETE"`, `"CONNECT"`, or `"*"` to match all. +* `apiGroups` lists one or more API groups to match. `""` is the core API group. `"*"` matches all API groups. +* `apiVersions` lists one or more API versions to match. `"*"` matches all API versions. +* `resources` lists one or more resources to match. + * `"*"` matches all resources, but not subresources. + * `"*/*"` matches all resources and subresources. + * `"pods/*"` matches all subresources of pods. + * `"*/status"` matches all status subresources. +* `scope` specifies a scope to match. Valid values are `"Cluster"`, `"Namespaced"`, and `"*"`. Subresources match the scope of their parent resource. Supported in v1.14+. Default is `"*"`, matching pre-1.14 behavior. + * `"Cluster"` means that only cluster-scoped resources will match this rule (Namespace API objects are cluster-scoped). + * `"Namespaced"` means that only namespaced resources will match this rule. + * `"*"` means that there are no scope restrictions. + +If an incoming request matches one of the specified operations, groups, versions, resources, and scope for any of a webhook's rules, the request is sent to the webhook. + +Here are other examples of rules that could be used to specify which resources should be intercepted. + +Match `CREATE` or `UPDATE` requests to `apps/v1` and `apps/v1beta1` `deployments` and `replicasets`: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + rules: + - operations: ["CREATE", "UPDATE"] + apiGroups: ["apps"] + apiVersions: ["v1", "v1beta1"] + resources: ["deployments", "replicasets"] + scope: "Namespaced" + ... +``` + +Match create requests for all resources (but not subresources) in all API groups and versions: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "*" + ... +``` + +Match update requests for all `status` subresources in all API groups and versions: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + rules: + - operations: ["UPDATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*/status"] + scope: "*" + ... +``` + +### Matching requests: objectSelector + +In v1.15+, webhooks may optionally limit which requests are intercepted based on the labels of the +objects they would be sent, by specifying an `objectSelector`. If specified, the objectSelector +is evaluated against both the object and oldObject that would be sent to the webhook, +and is considered to match if either object matches the selector. + +A null object (oldObject in the case of create, or newObject in the case of delete), +or an object that cannot have labels (like a `DeploymentRollback` or a `PodProxyOptions` object) +is not considered to match. + +Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. + +This example shows a mutating webhook that would match a `CREATE` of any resource with the label `foo: bar`: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + objectSelector: + matchLabels: + foo: bar + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "*" + ... +``` + +See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels for more examples of label selectors. + +### Matching requests: namespaceSelector + +Webhooks may optionally limit which requests for namespaced resources are intercepted, +based on the labels of the containing namespace, by specifying a `namespaceSelector`. + +The `namespaceSelector` decides whether to run the webhook on a request for a namespaced resource +(or a Namespace object), based on whether the namespace's labels match the selector. +If the object itself is a namespace, the matching is performed on object.metadata.labels. +If the object is a cluster scoped resource other than a Namespace, `namespaceSelector` has no effect. + +This example shows a mutating webhook that matches a `CREATE` of any namespaced resource inside a namespace +that does not have a "runlevel" label of "0" or "1": + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + namespaceSelector: + matchExpressions: + - key: runlevel + operator: NotIn + values: ["0","1"] + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "Namespaced" + ... +``` + +This example shows a validating webhook that matches a `CREATE` of any namespaced resource inside a namespace +that is associated with the "environment" of "prod" or "staging": + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + namespaceSelector: + matchExpressions: + - key: environment + operator: In + values: ["prod","staging"] + rules: + - operations: ["CREATE"] + apiGroups: ["*"] + apiVersions: ["*"] + resources: ["*"] + scope: "Namespaced" + ... +``` + +See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels for more examples of label selectors. + +### Matching requests: matchPolicy + +API servers can make objects available via multiple API groups or versions. +For example, the Kubernetes API server allows creating and modifying `Deployment` objects +via `extensions/v1beta1`, `apps/v1beta1`, `apps/v1beta2`, and `apps/v1` APIs. + +For example, if a webhook only specified a rule for some API groups/versions (like `apiGroups:["apps"], apiVersions:["v1","v1beta1"]`), +and a request was made to modify the resource via another API group/version (like `extensions/v1beta1`), +the request would not be sent to the webhook. + +In v1.15+, `matchPolicy` lets a webhook define how its `rules` are used to match incoming requests. +Allowed values are `Exact` or `Equivalent`. The default in `v1beta1` is `Exact`. + +* `Exact` means a request should be intercepted only if it exactly matches a specified rule. +* `Equivalent` means a request should be intercepted if modifies a resource listed in `rules`, even via another API group or version. + +In the example given above, the webhook that only registered for `apps/v1` could use `matchPolicy`: +* `matchPolicy: Exact` would mean the `extensions/v1beta1` request would not be sent to the webhook +* `matchPolicy: Equivalent` means the `extensions/v1beta1` request would be sent to the webhook (with the objects converted to a version the webhook had specified: `apps/v1`) + +Specifying `Equivalent` is recommended, and ensures that webhooks continue to intercept the +resources they expect when upgrades enable new versions of the resource in the API server. + +When a resource stops being served by the API server, it is no longer considered equivalent to other versions of that resource that are still served. +For example, deprecated `extensions/v1beta1` deployments are scheduled to stop being served by default in v1.16. +Once that occurs, a webhook with a `apiGroups:["extensions"], apiVersions:["v1beta1"], resources:["deployments"]` rule +would no longer intercept deployments created via `apps/v1` APIs. For that reason, webhooks should prefer registering +for stable versions of resources. + +This example shows a validating webhook that intercepts modifications to deployments (no matter the API group or version), +and is always sent an `apps/v1` `Deployment` object: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + matchPolicy: Equivalent + rules: + - operations: ["CREATE","UPDATE","DELETE"] + apiGroups: ["apps"] + apiVersions: ["v1"] + resources: ["deployments"] + scope: "Namespaced" + ... +``` + +### Contacting the webhook + +Once the API server has determined a request should be sent to a webhook, +it needs to know how to contact the webhook. This is specified in the `clientConfig` +stanza of the webhook configuration. + +Webhooks can either be called via a URL or a service reference, +and can optionally include a custom CA bundle to use to verify the TLS connection. + +#### URL + +`url` gives the location of the webhook, in standard URL form +(`scheme://host:port/path`). + +The `host` should not refer to a service running in the cluster; use +a service reference by specifying the `service` field instead. +The host might be resolved via external DNS in some apiservers +(e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would +be a layering violation). `host` may also be an IP address. + +Please note that using `localhost` or `127.0.0.1` as a `host` is +risky unless you take great care to run this webhook on all hosts +which run an apiserver which might need to make calls to this +webhook. Such installs are likely to be non-portable, i.e., not easy +to turn up in a new cluster. + +The scheme must be "https"; the URL must begin with "https://". + +Attempting to use a user or basic auth e.g. "user:password@" is not allowed. +Fragments ("#...") and query parameters ("?...") are also not allowed. + +Here is an example of a mutating webhook configured to call a URL +(and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle): +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + clientConfig: + url: "https://my-webhook.example.com:9443/my-webhook-path" + ... +``` + +#### Service reference + +The `service` stanza inside `clientConfig` is a reference to the service for this webhook. +If the webhook is running within the cluster, then you should use `service` instead of `url`. +The service namespace and name are required. The port is optional and defaults to 443. +The path is optional and defaults to "/". + +Here is an example of a mutating webhook configured to call a service on port "1234" +at the subpath "/my-path", and to verify the TLS connection against the ServerName +`my-service-name.my-service-namespace.svc` using a custom CA bundle: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + clientConfig: + caBundle: "Ci0tLS0tQk......tLS0K" + service: + namespace: my-service-namespace + name: my-service-name + path: /my-path + port: 1234 + ... +``` + +### Side effects + +Webhooks typically operate only on the content of the `AdmissionReview` sent to them. +Some webhooks, however, make out-of-band changes as part of processing admission requests. + +Webhooks that make out-of-band changes ("side effects") must also have a reconcilation mechanism +(like a controller) that periodically determines the actual state of the world, and adjusts +the out-of-band data modified by the admission webhook to reflect reality. +This is because a call to an admission webhook does not guarantee the admitted object will be persisted as is, or at all. +Later webhooks can modify the content of the object, a conflict could be encountered while writing to storage, +or the server could power off before persisting the object. + +Additionally, webhooks with side effects should skip those side-effects when `dryRun: true` admission requests are handled. +A webhook must explicitly indicate that it will not have side-effects when run with `dryRun`, +or the dry-run request will not be sent to the webhook and the API request fill fail instead. + +Webhooks indicate whether they have side effects using the `sideEffects` field in the webhook configuration. +`sideEffects` may be set to `Unknown`, `None`, `Some`, `NoneOnDryRun`. The default is `Unknown`. + +* `Unknown`: no information is known about the side effects of calling the webhook. +If a request with `dryRun: true` would trigger a call to this webhook, the request will instead fail, and the webhook will not be called. +* `None`: calling the webhook will have no side effects. +* `Some`: calling the webhook will possibly have side effects. +If a request with the dry-run attribute would trigger a call to this webhook, the request will instead fail, and the webhook will not be called. +* `NoneOnDryRun`: calling the webhook will possibly have side effects, +but if a request with `dryRun: true` is sent to the webhook, the webhook will suppress the side effects (the webhook is `dryRun`-aware). + +Here is an example of a validating webhook indicating it has no side effects on `dryRun: true` requests: +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + sideEffects: NoneOnDryRun + ... +``` + +### Timeouts + +Because webhooks add to API request latency, they should evaluate as quickly as possible. +`timeoutSeconds` allows configuring how long the API server should wait for a webhook to respond +before treating the call as a failure. + +If the timeout expires before the webhook responds, the webhook call will be ignored or +the API call will be rejected based on the [failure policy](#failure-policy). + +The timeout value must be between 1 and 30 seconds, and defaults to 30 seconds. + +Here is an example of a validating webhook with a custom timeout of 2 seconds: +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: ValidatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + timeoutSeconds: 2 + ... +``` + +### Reinvocation policy + +A single ordering of mutating admissions plugins (including webhooks) does not work for all cases +(see https://issue.k8s.io/64333 as an example). A mutating webhook can add a new sub-structure +to the object (like adding a `container` to a `pod`), and other mutating plugins which have already +run may have opinions on those new structures (like setting an `imagePullPolicy` on all containers). + +In v1.15+, to allow mutating admission plugins to observe changes made by other plugins, +built-in mutating admission plugins are re-run if a mutating webhook modifies an object, +and mutating webhooks can specify a `reinvocationPolicy` to control whether they are reinvoked as well. + +`reinvocationPolicy` may be set to `Never` or `IfNeeded`. It defaults to `Never`. + +* `Never`: the webhook must not be called more than once in a single admission evaluation +* `IfNeeded`: the webhook may be called again as part of the admission evaluation if the object +being admitted is modified by other admission plugins after the initial webhook call. + {{< note >}} + * the number of additional invocations is not guaranteed to be exactly one. + * if additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again. + * webhooks that use this option may be reordered to minimize the number of additional invocations. + * to validate an object after all mutations are guaranteed complete, use a validating admission webhook instead (recommended for webhooks with side-effects). + {{< /note >}} + +Here is an example of a mutating webhook opting into being re-invoked if later admission plugins modify the object: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + reinvocationPolicy: IfNeeded + ... +``` + +Mutating webhooks must be idempotent, able to successfully process an object they have already admitted +and potentially modified. This is true for all mutating admission webhooks, since any change they can make +in an object could already exist in the user-provided object, but it is essential for webhooks that opt into reinvocation. + +### Failure policy + +`failurePolicy` defines how unrecognized errors and timeout errors from the admission webhook +are handled. Allowed values are `Ignore` or `Fail`. Defaults to `Ignore` in v1beta1. + +* `Ignore` means that an error calling the webhook is ignored and the API request is allowed to continue. +* `Fail` means that an error calling the webhook causes the admission to fail and the API request to be rejected. + +Here is a mutating webhook configured to reject an API request if errors are encountered calling the admission webhook: + +```yaml +apiVersion: admissionregistration.k8s.io/v1beta1 +kind: MutatingWebhookConfiguration +... +webhooks: +- name: my-webhook.example.com + failurePolicy: Fail + ... +``` + {{% /capture %}} diff --git a/content/en/docs/setup/version-skew-policy.md b/content/en/docs/setup/version-skew-policy.md index e2078028e1758..3a1a8c36d738b 100644 --- a/content/en/docs/setup/version-skew-policy.md +++ b/content/en/docs/setup/version-skew-policy.md @@ -113,7 +113,7 @@ Pre-requisites: * The `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` instances that communicate with this server are at version **1.n** (this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version) * `kubelet` instances on all nodes are at version **1.n** or **1.(n-1)** (this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version) * Registered admission webhooks are able to handle the data the new `kube-apiserver` instance will send them: - * `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` objects are updated to include any new versions of REST resources added in **1.(n+1)** + * `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` objects are updated to include any new versions of REST resources added in **1.(n+1)** (or use the [`matchPolicy: Equivalent` option](/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchpolicy) available in v1.15+) * The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in **1.(n+1)** Upgrade `kube-apiserver` to **1.(n+1)** From fc86f8fef9166ee8f826a75226a54e7d00f2900d Mon Sep 17 00:00:00 2001 From: Jordan Liggitt Date: Mon, 10 Jun 2019 15:25:10 -0400 Subject: [PATCH 14/31] Add custom resource quota example (#14492) --- content/en/docs/concepts/policy/resource-quotas.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/content/en/docs/concepts/policy/resource-quotas.md b/content/en/docs/concepts/policy/resource-quotas.md index 40c813e00c35b..a196805143029 100644 --- a/content/en/docs/concepts/policy/resource-quotas.md +++ b/content/en/docs/concepts/policy/resource-quotas.md @@ -136,6 +136,9 @@ Here is an example set of resources users may want to put under object count quo * `count/cronjobs.batch` * `count/deployments.extensions` +The 1.15 release added support for custom resources using the same syntax. +For example, to create a quota on a `widgets` custom resource in the `example.com` API group, use `count/widgets.example.com`. + When using `count/*` resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources. For example, you may want to quota the number of secrets in a server given their large size. Too many secrets in a cluster can From b1a0711f119549a179d25095b8624023d23aa6fe Mon Sep 17 00:00:00 2001 From: "Rostislav M. Georgiev" Date: Mon, 10 Jun 2019 22:27:17 +0300 Subject: [PATCH 15/31] kubeadm: Document new v1beta2 config format (#14607) Signed-off-by: Rostislav M. Georgiev --- .../concepts/cluster-administration/cloud-providers.md | 4 ++-- .../kubeadm/generated/kubeadm_config_migrate.md | 4 ++-- .../reference/setup-tools/kubeadm/kubeadm-init-phase.md | 8 ++++---- .../docs/reference/setup-tools/kubeadm/kubeadm-init.md | 9 ++++----- content/en/docs/setup/independent/control-plane-flags.md | 8 ++++---- content/en/docs/setup/independent/high-availability.md | 4 ++-- .../docs/setup/independent/setup-ha-etcd-with-kubeadm.md | 2 +- .../tasks/administer-cluster/kubeadm/kubeadm-certs.md | 2 +- 8 files changed, 20 insertions(+), 21 deletions(-) diff --git a/content/en/docs/concepts/cluster-administration/cloud-providers.md b/content/en/docs/concepts/cluster-administration/cloud-providers.md index e468092d0130a..067c362c14a5a 100644 --- a/content/en/docs/concepts/cluster-administration/cloud-providers.md +++ b/content/en/docs/concepts/cluster-administration/cloud-providers.md @@ -17,14 +17,14 @@ kubeadm has configuration options to specify configuration information for cloud in-tree cloud provider can be configured using kubeadm as shown below: ```yaml -apiVersion: kubeadm.k8s.io/v1beta1 +apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration nodeRegistration: kubeletExtraArgs: cloud-provider: "openstack" cloud-config: "/etc/kubernetes/cloud.conf" --- -apiVersion: kubeadm.k8s.io/v1beta1 +apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.13.0 apiServer: diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md index 6d84dbd52620a..11bf770de8cf0 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md @@ -8,10 +8,10 @@ Read an older version of the kubeadm configuration API types from a file, and ou This command lets you convert configuration objects of older versions to the latest supported version, locally in the CLI tool without ever touching anything in the cluster. In this version of kubeadm, the following API versions are supported: -- kubeadm.k8s.io/v1alpha3 - kubeadm.k8s.io/v1beta1 +- kubeadm.k8s.io/v1beta2 -Further, kubeadm can only write out config of version "kubeadm.k8s.io/v1beta1", but read both types. +Further, kubeadm can only write out config of version "kubeadm.k8s.io/v1beta2", but read both types. So regardless of what version you pass to the --old-config parameter here, the API object will be read, deserialized, defaulted, converted, validated, and re-serialized when written to stdout or --new-config if specified. diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md index b5644d854c50d..c65da6fddb1d9 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md @@ -145,18 +145,18 @@ kubeadm config images list/pull --config=someconfig.yaml kubeadm upgrade apply --config=someconfig.yaml ``` -The file has to contain a [`DNS`](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1#DNS) field in[`ClusterConfiguration`](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1#ClusterConfiguration) +The file has to contain a [`DNS`](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#DNS) field in[`ClusterConfiguration`](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#ClusterConfiguration) and also a type for the addon - `kube-dns` (default value is `CoreDNS`). ```yaml -apiVersion: kubeadm.k8s.io/v1beta1 +apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration dns: type: "kube-dns" ``` -For more details on each field in the `v1beta1` configuration you can navigate to our -[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1) +For more details on each field in the `v1beta2` configuration you can navigate to our +[API reference pages.] (https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) ## What's next * [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index 57d456b907dd3..07d797b708e01 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -118,12 +118,11 @@ configuration file options. This file is passed in the `--config` option. In Kubernetes 1.11 and later, the default configuration can be printed out using the [kubeadm config print](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command. -It is **recommended** that you migrate your old `v1alpha3` configuration to `v1beta1` using -the [kubeadm config migrate](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command, -because `v1alpha3` will be removed in Kubernetes 1.15. +It is **recommended** that you migrate your old `v1beta1` configuration to `v1beta2` using +the [kubeadm config migrate](/docs/reference/setup-tools/kubeadm/kubeadm-config/) command. -For more details on each field in the `v1beta1` configuration you can navigate to our -[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1). +For more details on each field in the `v1beta2` configuration you can navigate to our +[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2). ### Adding kube-proxy parameters {#kube-proxy} diff --git a/content/en/docs/setup/independent/control-plane-flags.md b/content/en/docs/setup/independent/control-plane-flags.md index e3b5edfdd516a..7a03137669907 100644 --- a/content/en/docs/setup/independent/control-plane-flags.md +++ b/content/en/docs/setup/independent/control-plane-flags.md @@ -23,7 +23,7 @@ The `extraArgs` field consist of `key: value` pairs. To override a flag for a co 2. Add the flags to override to the field. For more details on each field in the configuration you can navigate to our -[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#ClusterConfiguration). +[API reference pages](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#ClusterConfiguration). {{% /capture %}} @@ -35,7 +35,7 @@ For details, see the [reference documentation for kube-apiserver](/docs/referenc Example usage: ```yaml -apiVersion: kubeadm.k8s.io/v1beta1 +apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.13.0 metadata: @@ -54,7 +54,7 @@ For details, see the [reference documentation for kube-controller-manager](/docs Example usage: ```yaml -apiVersion: kubeadm.k8s.io/v1beta1 +apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.13.0 metadata: @@ -72,7 +72,7 @@ For details, see the [reference documentation for kube-scheduler](/docs/referenc Example usage: ```yaml -apiVersion: kubeadm.k8s.io/v1beta1 +apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.13.0 metadata: diff --git a/content/en/docs/setup/independent/high-availability.md b/content/en/docs/setup/independent/high-availability.md index 38f0425dfc530..c07b4da478eaa 100644 --- a/content/en/docs/setup/independent/high-availability.md +++ b/content/en/docs/setup/independent/high-availability.md @@ -106,7 +106,7 @@ option. Your cluster requirements may need a different configuration. 1. On the first control plane node, create a configuration file called `kubeadm-config.yaml`: - apiVersion: kubeadm.k8s.io/v1beta1 + apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: stable controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" @@ -231,7 +231,7 @@ in the kubeadm config file. 1. Create a file called `kubeadm-config.yaml` with the following contents: - apiVersion: kubeadm.k8s.io/v1beta1 + apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: stable controlPlaneEndpoint: "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" diff --git a/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md b/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md index d0a85cdb3ca60..0d6cdeef1423b 100644 --- a/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md +++ b/content/en/docs/setup/independent/setup-ha-etcd-with-kubeadm.md @@ -82,7 +82,7 @@ this example. HOST=${ETCDHOSTS[$i]} NAME=${NAMES[$i]} cat << EOF > /tmp/${HOST}/kubeadmcfg.yaml - apiVersion: "kubeadm.k8s.io/v1beta1" + apiVersion: "kubeadm.k8s.io/v1beta2" kind: ClusterConfiguration etcd: local: diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md index 55aa0902d0d87..ed35866836742 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-certs.md @@ -115,7 +115,7 @@ To activate the build-in signer, you must pass the `--cluster-signing-cert-file` If you're creating a new cluster, you can use a kubeadm [configuration file][config]: ```yaml - apiVersion: kubeadm.k8s.io/v1beta1 + apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration controllerManager: extraArgs: From 1ab395770409855f6daaf031e15b4c195be65fc1 Mon Sep 17 00:00:00 2001 From: Vladimir Vivien Date: Mon, 10 Jun 2019 15:29:11 -0400 Subject: [PATCH 16/31] CSI Inline Ephemeral Documentation Update (#14704) * CSI inline ephemeral ver update in doc * CSI inline ephemeral feature-gate update --- content/en/docs/concepts/storage/volumes.md | 2 +- .../reference/command-line-tools-reference/feature-gates.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index abcbd26e1afe4..15ab6e4859d6f 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -1315,7 +1315,7 @@ Learn how to #### CSI ephemeral volumes -{{< feature-state for_k8s_version="v1.14" state="alpha" >}} +{{< feature-state for_k8s_version="v1.15" state="alpha" >}} This feature allows CSI volumes to be directly embedded in the Pod specification instead of a PersistentVolume. Volumes specified in this way are ephemeral and do not persist across Pod restarts. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 90436cfc733cc..3a48a4b5dd26c 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -59,7 +59,7 @@ different Kubernetes components. | `CSIBlockVolume` | `true` | Beta | 1.14 | | | `CSIDriverRegistry` | `false` | Alpha | 1.12 | 1.13 | | `CSIDriverRegistry` | `true` | Beta | 1.14 | | -| `CSIInlineVolume` | `false` | Alpha | 1.14 | - | +| `CSIInlineVolume` | `false` | Alpha | 1.15 | - | | `CSIMigration` | `false` | Alpha | 1.14 | | | `CSIMigrationAWS` | `false` | Alpha | 1.14 | | | `CSIMigrationAzureDisk` | `false` | Alpha | 1.15 | | From 12238578ae46e10e84b32094af22cbb51224c574 Mon Sep 17 00:00:00 2001 From: Morten Torkildsen Date: Mon, 10 Jun 2019 12:31:09 -0700 Subject: [PATCH 17/31] Docs for feature: PDB support for custom resource with scale subresource (#14519) --- .../custom-resources/custom-resource-definitions.md | 2 ++ content/en/docs/tasks/run-application/configure-pdb.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md index 29af0df9d81d3..807aaf15b1270 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md @@ -660,6 +660,8 @@ kubectl get crontabs my-new-cron-object -o jsonpath='{.spec.replicas}' 5 ``` +You can use a [PodDisruptionBudget](docs/tasks/run-application/configure-pdb/) to protect custom resources that have the scale subresource enabled. + ### Categories Categories is a list of grouped resources the custom resource belongs to (eg. `all`). diff --git a/content/en/docs/tasks/run-application/configure-pdb.md b/content/en/docs/tasks/run-application/configure-pdb.md index c11ee1eb5e7f9..91c668d4d1964 100644 --- a/content/en/docs/tasks/run-application/configure-pdb.md +++ b/content/en/docs/tasks/run-application/configure-pdb.md @@ -49,6 +49,8 @@ specified by one of the built-in Kubernetes controllers: In this case, make a note of the controller's `.spec.selector`; the same selector goes into the PDBs `.spec.selector`. +From version 1.15 PDBs support custom controllers where the [scale subresource](docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#scale-subresource) is enabled. + You can also use PDBs with pods which are not controlled by one of the above controllers, or arbitrary groups of pods, but there are some restrictions, described in [Arbitrary Controllers and Selectors](#arbitrary-controllers-and-selectors). From d1bdefd251b6dc028235eb12c6373b448c7f6585 Mon Sep 17 00:00:00 2001 From: Kevin Taylor Date: Mon, 10 Jun 2019 21:07:07 +0100 Subject: [PATCH 18/31] VolumeSubpathEnvExpansion Beta Documentation (#13846) --- content/en/docs/concepts/storage/volumes.md | 2 +- .../reference/command-line-tools-reference/feature-gates.md | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/docs/concepts/storage/volumes.md b/content/en/docs/concepts/storage/volumes.md index 15ab6e4859d6f..4d6a9e60a3765 100644 --- a/content/en/docs/concepts/storage/volumes.md +++ b/content/en/docs/concepts/storage/volumes.md @@ -1154,7 +1154,7 @@ spec: ### Using subPath with expanded environment variables -{{< feature-state for_k8s_version="v1.14" state="alpha" >}} +{{< feature-state for_k8s_version="v1.15" state="beta" >}} Use the `subPathExpr` field to construct `subPath` directory names from Downward API environment variables. diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 3a48a4b5dd26c..21783da5e2130 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -157,7 +157,8 @@ different Kubernetes components. | `VolumeScheduling` | `false` | Alpha | 1.9 | 1.9 | | `VolumeScheduling` | `true` | Beta | 1.10 | 1.12 | | `VolumeScheduling` | `true` | GA | 1.13 | | -| `VolumeSubpathEnvExpansion` | `false` | Alpha | 1.14 | | +| `VolumeSubpathEnvExpansion` | `false` | Alpha | 1.14 | 1.14 | +| `VolumeSubpathEnvExpansion` | `true` | Beta | 1.15 | | | `VolumeSnapshotDataSource` | `false` | Alpha | 1.12 | - | | `ScheduleDaemonSetPods` | `false` | Alpha | 1.11 | 1.11 | | `ScheduleDaemonSetPods` | `true` | Beta | 1.12 | | From b51345a6812e6c2a796ae60ad1ae70c4dcbfa956 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 11 Jun 2019 02:18:19 +0300 Subject: [PATCH 19/31] kubeadm-setup: update all setup related documents for 1.15 (#14594) --- .../independent/create-cluster-kubeadm.md | 101 +++++++++--------- .../en/docs/setup/independent/ha-topology.md | 4 +- .../setup/independent/high-availability.md | 50 +++++---- .../docs/setup/independent/install-kubeadm.md | 13 ++- .../independent/troubleshooting-kubeadm.md | 4 +- 5 files changed, 94 insertions(+), 78 deletions(-) diff --git a/content/en/docs/setup/independent/create-cluster-kubeadm.md b/content/en/docs/setup/independent/create-cluster-kubeadm.md index cd73f6482be33..4a389d00c0457 100644 --- a/content/en/docs/setup/independent/create-cluster-kubeadm.md +++ b/content/en/docs/setup/independent/create-cluster-kubeadm.md @@ -1,18 +1,18 @@ --- reviewers: - sig-cluster-lifecycle -title: Creating a single master cluster with kubeadm +title: Creating a single control-plane cluster with kubeadm content_template: templates/task weight: 30 --- {{% capture overview %}} -**kubeadm** helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices. With kubeadm, your cluster should pass [Kubernetes Conformance tests](https://kubernetes.io/blog/2017/10/software-conformance-certification). Kubeadm also supports other cluster -lifecycle functions, such as upgrades, downgrade, and managing [bootstrap tokens](/docs/reference/access-authn-authz/bootstrap-tokens/). +**kubeadm** helps you bootstrap a minimum viable Kubernetes cluster that conforms to best practices. With kubeadm, your cluster should pass [Kubernetes Conformance tests](https://kubernetes.io/blog/2017/10/software-conformance-certification). Kubeadm also supports other cluster +lifecycle functions, such as upgrades, downgrade, and managing [bootstrap tokens](/docs/reference/access-authn-authz/bootstrap-tokens/). -Because you can install kubeadm on various types of machine (e.g. laptop, server, -Raspberry Pi, etc.), it's well suited for integration with provisioning systems +Because you can install kubeadm on various types of machine (e.g. laptop, server, +Raspberry Pi, etc.), it's well suited for integration with provisioning systems such as Terraform or Ansible. kubeadm's simplicity means it can serve a wide range of use cases: @@ -33,17 +33,17 @@ installing deb or rpm packages. The responsible SIG for kubeadm, but you may also build them from source for other OSes. -### kubeadm Maturity +### kubeadm maturity | Area | Maturity Level | |---------------------------|--------------- | | Command line UX | GA | | Implementation | GA | -| Config file API | beta | +| Config file API | Beta | | CoreDNS | GA | -| kubeadm alpha subcommands | alpha | -| High availability | alpha | -| DynamicKubeletConfig | alpha | +| kubeadm alpha subcommands | Alpha | +| High availability | Beta | +| DynamicKubeletConfig | Alpha | kubeadm's overall feature state is **GA**. Some sub-features, like the configuration @@ -69,6 +69,8 @@ timeframe; which also applies to `kubeadm`. | v1.11.x | June 2018 | March 2019   | | v1.12.x | September 2018 | June 2019   | | v1.13.x | December 2018 | September 2019   | +| v1.14.x | March 2019 | December 2019   | +| v1.15.x | June 2019 | March 2020   | {{% /capture %}} @@ -77,17 +79,17 @@ timeframe; which also applies to `kubeadm`. - One or more machines running a deb/rpm-compatible OS, for example Ubuntu or CentOS - 2 GB or more of RAM per machine. Any less leaves little room for your apps. -- 2 CPUs or more on the master +- 2 CPUs or more on the control-plane node - Full network connectivity among all machines in the cluster. A public or private network is fine. - + {{% /capture %}} {{% capture steps %}} ## Objectives -* Install a single master Kubernetes cluster or [high availability cluster](/docs/setup/independent/high-availability/) +* Install a single control-plane Kubernetes cluster or [high-availability cluster](/docs/setup/independent/high-availability/) * Install a Pod network on the cluster so that your Pods can talk to each other @@ -102,17 +104,17 @@ If you have already installed kubeadm, run `apt-get update && apt-get upgrade` or `yum update` to get the latest version of kubeadm. When you upgrade, the kubelet restarts every few seconds as it waits in a crashloop for -kubeadm to tell it what to do. This crashloop is expected and normal. -After you initialize your master, the kubelet runs normally. +kubeadm to tell it what to do. This crashloop is expected and normal. +After you initialize your control-plane, the kubelet runs normally. {{< /note >}} -### Initializing your master +### Initializing your control-plane node -The master is the machine where the control plane components run, including +The control-plane node is the machine where the control plane components run, including etcd (the cluster database) and the API server (which the kubectl CLI communicates with). -1. Choose a pod network add-on, and verify whether it requires any arguments to +1. Choose a pod network add-on, and verify whether it requires any arguments to be passed to kubeadm initialization. Depending on which third-party provider you choose, you might need to set the `--pod-network-cidr` to a provider-specific value. See [Installing a pod network add-on](#pod-network). @@ -120,18 +122,18 @@ a provider-specific value. See [Installing a pod network add-on](#pod-network). by using a list of well known domain socket paths. To use different container runtime or if there are more than one installed on the provisioned node, specify the `--cri-socket` argument to `kubeadm init`. See [Installing runtime](/docs/setup/independent/install-kubeadm/#installing-runtime). -1. (Optional) Unless otherwise specified, kubeadm uses the network interface associated -with the default gateway to advertise the master's IP. To use a different -network interface, specify the `--apiserver-advertise-address=` argument -to `kubeadm init`. To deploy an IPv6 Kubernetes cluster using IPv6 addressing, you +1. (Optional) Unless otherwise specified, kubeadm uses the network interface associated +with the default gateway to advertise the control-plane's IP. To use a different +network interface, specify the `--apiserver-advertise-address=` argument +to `kubeadm init`. To deploy an IPv6 Kubernetes cluster using IPv6 addressing, you must specify an IPv6 address, for example `--apiserver-advertise-address=fd00::101` -1. (Optional) Run `kubeadm config images pull` prior to `kubeadm init` to verify -connectivity to gcr.io registries. +1. (Optional) Run `kubeadm config images pull` prior to `kubeadm init` to verify +connectivity to gcr.io registries. Now run: ```bash -kubeadm init +kubeadm init ``` ### More information @@ -150,7 +152,7 @@ components do not currently support multi-architecture. `kubeadm init` first runs a series of prechecks to ensure that the machine is ready to run Kubernetes. These prechecks expose warnings and exit on errors. `kubeadm init` -then downloads and installs the cluster control plane components. This may take several minutes. +then downloads and installs the cluster control plane components. This may take several minutes. The output should look like: ```none @@ -239,8 +241,8 @@ export KUBECONFIG=/etc/kubernetes/admin.conf Make a record of the `kubeadm join` command that `kubeadm init` outputs. You need this command to [join nodes to your cluster](#join-nodes). -The token is used for mutual authentication between the master and the joining -nodes. The token included here is secret. Keep it safe, because anyone with this +The token is used for mutual authentication between the control-plane node and the joining +nodes. The token included here is secret. Keep it safe, because anyone with this token can add authenticated nodes to your cluster. These tokens can be listed, created, and deleted with the `kubeadm token` command. See the [kubeadm reference guide](/docs/reference/setup-tools/kubeadm/kubeadm-token/). @@ -258,8 +260,8 @@ each other. kubeadm only supports Container Network Interface (CNI) based networks (and does not support kubenet).** Several projects provide Kubernetes pod networks using CNI, some of which also -support [Network Policy](/docs/concepts/services-networking/networkpolicies/). See the [add-ons page](/docs/concepts/cluster-administration/addons/) for a complete list of available network add-ons. -- IPv6 support was added in [CNI v0.6.0](https://github.com/containernetworking/cni/releases/tag/v0.6.0). +support [Network Policy](/docs/concepts/services-networking/networkpolicies/). See the [add-ons page](/docs/concepts/cluster-administration/addons/) for a complete list of available network add-ons. +- IPv6 support was added in [CNI v0.6.0](https://github.com/containernetworking/cni/releases/tag/v0.6.0). - [CNI bridge](https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md) and [local-ipam](https://github.com/containernetworking/plugins/blob/master/plugins/ipam/host-local/README.md) are the only supported IPv6 network plugins in Kubernetes version 1.9. Note that kubeadm sets up a more secure cluster by default and enforces use of [RBAC](/docs/reference/access-authn-authz/rbac/). @@ -423,8 +425,8 @@ out our [troubleshooting docs](/docs/setup/independent/troubleshooting-kubeadm/) ### Control plane node isolation -By default, your cluster will not schedule pods on the master for security -reasons. If you want to be able to schedule pods on the master, e.g. for a +By default, your cluster will not schedule pods on the control-plane node for security +reasons. If you want to be able to schedule pods on the control-plane node, e.g. for a single-machine Kubernetes cluster for development, run: ```bash @@ -440,7 +442,7 @@ taint "node-role.kubernetes.io/master:" not found ``` This will remove the `node-role.kubernetes.io/master` taint from any nodes that -have it, including the master node, meaning that the scheduler will then be able +have it, including the control-plane node, meaning that the scheduler will then be able to schedule pods everywhere. ### Joining your nodes {#join-nodes} @@ -455,7 +457,7 @@ The nodes are where your workloads (containers and pods, etc) run. To add new no kubeadm join --token : --discovery-token-ca-cert-hash sha256: ``` -If you do not have the token, you can get it by running the following command on the master node: +If you do not have the token, you can get it by running the following command on the control-plane node: ``` bash kubeadm token list @@ -472,7 +474,7 @@ TOKEN TTL EXPIRES USAGES DESCRIPTION ``` By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, -you can create a new token by running the following command on the master node: +you can create a new token by running the following command on the control-plane node: ``` bash kubeadm token create @@ -484,7 +486,7 @@ The output is similar to this: 5didvk.d09sbcov8ph2amjw ``` -If you don't have the value of `--discovery-token-ca-cert-hash`, you can get it by running the following command chain on the master node: +If you don't have the value of `--discovery-token-ca-cert-hash`, you can get it by running the following command chain on the control-plane node: ``` bash openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ @@ -517,12 +519,12 @@ Run 'kubectl get nodes' on the master to see this machine join. ``` A few seconds later, you should notice this node in the output from `kubectl get -nodes` when run on the master. +nodes` when run on the control-plane node. -### (Optional) Controlling your cluster from machines other than the master +### (Optional) Controlling your cluster from machines other than the control-plane node In order to get a kubectl on some other computer (e.g. laptop) to talk to your -cluster, you need to copy the administrator kubeconfig file from your master +cluster, you need to copy the administrator kubeconfig file from your control-plane node to your workstation like this: ``` bash @@ -562,7 +564,7 @@ To undo what kubeadm did, you should first [drain the node](/docs/reference/generated/kubectl/kubectl-commands#drain) and make sure that the node is empty before shutting it down. -Talking to the master with the appropriate credentials, run: +Talking to the control-plane node with the appropriate credentials, run: ```bash kubectl drain --delete-local-data --force --ignore-daemonsets @@ -650,18 +652,17 @@ supports your chosen platform. ## Limitations {#limitations} -Please note: kubeadm is a work in progress and these limitations will be -addressed in due course. +The cluster created here has a single control-plane node, with a single etcd database +running on it. This means that if the control-plane node fails, your cluster may lose +data and may need to be recreated from scratch. + +Workarounds: -1. The cluster created here has a single master, with a single etcd database - running on it. This means that if the master fails, your cluster may lose - data and may need to be recreated from scratch. Adding HA support - (multiple etcd servers, multiple API servers, etc) to kubeadm is - still a work-in-progress. +* Regularly [back up etcd](https://coreos.com/etcd/docs/latest/admin_guide.html). The + etcd data directory configured by kubeadm is at `/var/lib/etcd` on the control-plane node. - Workaround: regularly - [back up etcd](https://coreos.com/etcd/docs/latest/admin_guide.html). The - etcd data directory configured by kubeadm is at `/var/lib/etcd` on the master. +* Use multiple control-plane nodes by completing the + [HA setup](/docs/setup/independent/ha-topology) instead. ## Troubleshooting {#troubleshooting} diff --git a/content/en/docs/setup/independent/ha-topology.md b/content/en/docs/setup/independent/ha-topology.md index 2cd28f32ec8e7..692e5179c55dc 100644 --- a/content/en/docs/setup/independent/ha-topology.md +++ b/content/en/docs/setup/independent/ha-topology.md @@ -34,7 +34,7 @@ Each control plane node creates a local etcd member and this etcd member communi the `kube-apiserver` of this node. The same applies to the local `kube-controller-manager` and `kube-scheduler` instances. -This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster +This topology couples the control planes and etcd members on the same nodes. It is simpler to set up than a cluster with external etcd nodes, and simpler to manage for replication. However, a stacked cluster runs the risk of failed coupling. If one node goes down, both an etcd member and a control @@ -43,7 +43,7 @@ plane instance are lost, and redundancy is compromised. You can mitigate this ri You should therefore run a minimum of three stacked control plane nodes for an HA cluster. This is the default topology in kubeadm. A local etcd member is created automatically -on control plane nodes when using `kubeadm init` and `kubeadm join --experimental-control-plane`. +on control plane nodes when using `kubeadm init` and `kubeadm join --control-plane`. ![Stacked etcd topology](/images/kubeadm/kubeadm-ha-topology-stacked-etcd.svg) diff --git a/content/en/docs/setup/independent/high-availability.md b/content/en/docs/setup/independent/high-availability.md index c07b4da478eaa..153fb608050aa 100644 --- a/content/en/docs/setup/independent/high-availability.md +++ b/content/en/docs/setup/independent/high-availability.md @@ -19,12 +19,10 @@ control plane nodes and etcd members are separated. Before proceeding, you should carefully consider which approach best meets the needs of your applications and environment. [This comparison topic](/docs/setup/independent/ha-topology/) outlines the advantages and disadvantages of each. -You should also be aware that setting up HA clusters with kubeadm is still experimental and will be further -simplified in future versions. You might encounter issues with upgrading your clusters, for example. -We encourage you to try either approach, and provide us with feedback in the kubeadm -[issue tracker](https://github.com/kubernetes/kubeadm/issues/new). +If you encounter issues with setting up the HA cluster, please provide us with feedback +in the kubeadm [issue tracker](https://github.com/kubernetes/kubeadm/issues/new). -See also [The upgrade documentation](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-14). +See also [The upgrade documentation](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15). {{< caution >}} This page does not address running your cluster on a cloud provider. In a cloud @@ -127,20 +125,20 @@ the `networking` object of `ClusterConfiguration`. 1. Initialize the control plane: ```sh - sudo kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs + sudo kubeadm init --config=kubeadm-config.yaml --upload-certs ``` - - The `--experimental-upload-certs` flags is used to upload the certificates that should be shared + - The `--upload-certs` flags is used to upload the certificates that should be shared across all the control-plane instances to the cluster. If instead, you prefer to copy certs across control-plane nodes manually or using automation tools, please remove this flag and refer to [Manual certificate distribution](#manual-certs) section bellow. After the command completes you should see something like so: - + ```sh ... You can now join any number of control-plane node by running the following command on each as a root: - kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --experimental-control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07 - + kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07 + Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use kubeadm init phase upload-certs to reload certs afterward. @@ -149,15 +147,27 @@ the `networking` object of `ClusterConfiguration`. ``` - Copy this output to a text file. You will need it later to join control plane and worker nodes to the cluster. - - When `--experimental-upload-certs` is used with `kubeadm init`, the certificates of the primary control plane + - When `--upload-certs` is used with `kubeadm init`, the certificates of the primary control plane are encrypted and uploaded in the `kubeadm-certs` Secret. - To re-upload the certificates and generate a new decryption key, use the following command on a control plane node that is already joined to the cluster: ```sh - sudo kubeadm init phase upload-certs --experimental-upload-certs + sudo kubeadm init phase upload-certs --upload-certs + ``` + + - You can also specify a custom `--certificate-key` during `init` that can later be used by `join`. + To generate such a key you can use the following command: + + ```sh + kubeadm alpha certs certificate-key ``` +{{< note >}} +The `kubeadm init` flags `--config` and `--certificate-key` cannot be mixed, therefore if you want +to use the [kubeadm configuration](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2) you must add the `certificateKey` field in the appropriate config locations (under `InitConfiguration` and `JoinConfiguration: controlPlane`). +{{< /note >}} + {{< note >}} The `kubeadm-certs` Secret and decryption key expire after two hours. {{< /note >}} @@ -186,9 +196,11 @@ As stated in the command output, the certificate-key gives access to cluster sen ### Steps for the rest of the control plane nodes -{{< caution >}} -You must join new control plane nodes sequentially, only after the first node has finished initializing. -{{< /caution >}} +{{< note >}} +Since kubeadm version 1.15 you can join multiple control-plane nodes in parallel. +Prior to this version, you must join new control plane nodes sequentially, only after +the first node has finished initializing. +{{< /note >}} For each additional control plane node you should: @@ -196,10 +208,10 @@ For each additional control plane node you should: It should look something like this: ```sh - sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --experimental-control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07 + sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery-token-ca-cert-hash sha256:7c2e69131a36ae2a042a339b33381c6d0d43887e2de83720eff5359e26aec866 --control-plane --certificate-key f8902e114ef118304e561c3ecd4d0b543adc226b7a07f675f56564185ffe0c07 ``` - - The `--experimental-control-plane` flag tells `kubeadm join` to create a new control plane. + - The `--control-plane` flag tells `kubeadm join` to create a new control plane. - The `--certificate-key ...` will cause the control plane certificates to be downloaded from the `kubeadm-certs` Secret in the cluster and be decrypted using the given key. @@ -261,7 +273,7 @@ etcd topology this is managed automatically. The following steps are exactly the same as described for stacked etcd setup: -1. Run `sudo kubeadm init --config kubeadm-config.yaml --experimental-upload-certs` on this node. +1. Run `sudo kubeadm init --config kubeadm-config.yaml --upload-certs` on this node. 1. Write the output join commands that are returned to a text file for later use. @@ -293,7 +305,7 @@ sudo kubeadm join 192.168.0.200:6443 --token 9vr73a.a8uxyaju799qwdjv --discovery ## Manual certificate distribution {#manual-certs} -If you choose to not use `kubeadm init` with the `--experimental-upload-certs` flag this means that +If you choose to not use `kubeadm init` with the `--upload-certs` flag this means that you are going to have to manually copy the certificates from the primary control plane node to the joining control plane nodes. diff --git a/content/en/docs/setup/independent/install-kubeadm.md b/content/en/docs/setup/independent/install-kubeadm.md index db25539b94eee..6b6d222808373 100644 --- a/content/en/docs/setup/independent/install-kubeadm.md +++ b/content/en/docs/setup/independent/install-kubeadm.md @@ -54,7 +54,7 @@ route, we recommend you add IP route(s) so Kubernetes cluster addresses go via t ## Check required ports -### Master node(s) +### Control-plane node(s) | Protocol | Direction | Port Range | Purpose | Used By | |----------|-----------|------------|-------------------------|---------------------------| @@ -76,7 +76,7 @@ route, we recommend you add IP route(s) so Kubernetes cluster addresses go via t Any port numbers marked with * are overridable, so you will need to ensure any custom ports you provide are also open. -Although etcd ports are included in master nodes, you can also host your own +Although etcd ports are included in control-plane nodes, you can also host your own etcd cluster externally or on custom ports. The pod network plugin you use (see below) may also require certain ports to be @@ -201,7 +201,7 @@ systemctl enable --now kubelet Install CNI plugins (required for most pod network): ```bash -CNI_VERSION="v0.6.0" +CNI_VERSION="v0.7.5" mkdir -p /opt/cni/bin curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" | tar -C /opt/cni/bin -xz ``` @@ -209,7 +209,7 @@ curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_ Install crictl (required for kubeadm / Kubelet Container Runtime Interface (CRI)) ```bash -CRICTL_VERSION="v1.11.1" +CRICTL_VERSION="v1.12.0" mkdir -p /opt/bin curl -L "https://github.com/kubernetes-incubator/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | tar -C /opt/bin -xz ``` @@ -241,7 +241,7 @@ systemctl enable --now kubelet The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. -## Configure cgroup driver used by kubelet on Master Node +## Configure cgroup driver used by kubelet on control-plane node When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet and set it in the `/var/lib/kubelet/kubeadm-flags.env` file during runtime. @@ -266,6 +266,9 @@ systemctl daemon-reload systemctl restart kubelet ``` +The automatic detection of cgroup driver for other container runtimes +like CRI-O and containerd is work in progress. + ## Troubleshooting If you are running into difficulties with kubeadm, please consult our [troubleshooting docs](/docs/setup/independent/troubleshooting-kubeadm/). diff --git a/content/en/docs/setup/independent/troubleshooting-kubeadm.md b/content/en/docs/setup/independent/troubleshooting-kubeadm.md index 87ca0a3cb5d27..441fd4ba6915e 100644 --- a/content/en/docs/setup/independent/troubleshooting-kubeadm.md +++ b/content/en/docs/setup/independent/troubleshooting-kubeadm.md @@ -60,7 +60,7 @@ This may be caused by a number of problems. The most common are: 1. Install Docker again following instructions [here](/docs/setup/independent/install-kubeadm/#installing-docker). 1. Change the kubelet config to match the Docker cgroup driver manually, you can refer to - [Configure cgroup driver used by kubelet on Master Node](/docs/setup/independent/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-master-node) + [Configure cgroup driver used by kubelet on control-plane node](/docs/setup/independent/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-master-node) for detailed instructions. - control plane Docker containers are crashlooping or hanging. You can check this by running `docker ps` and investigating each container by running `docker logs`. @@ -100,7 +100,7 @@ Right after `kubeadm init` there should not be any pods in these states. until you have deployed the network solution. - If you see Pods in the `RunContainerError`, `CrashLoopBackOff` or `Error` state after deploying the network solution and nothing happens to `coredns` (or `kube-dns`), - it's very likely that the Pod Network solution that you installed is somehow broken. + it's very likely that the Pod Network solution that you installed is somehow broken. You might have to grant it more RBAC privileges or use a newer version. Please file an issue in the Pod Network providers' issue tracker and get the issue triaged there. - If you install a version of Docker older than 1.12.1, remove the `MountFlags=slave` option From 9e102b5cbf7728489d88b5470a31f6d2e9b0c5fd Mon Sep 17 00:00:00 2001 From: Zihong Zheng Date: Thu, 23 May 2019 12:38:56 -0700 Subject: [PATCH 20/31] Add a section for service load balancer cleanup --- .../create-external-load-balancer.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/content/en/docs/tasks/access-application-cluster/create-external-load-balancer.md b/content/en/docs/tasks/access-application-cluster/create-external-load-balancer.md index b8a5b1c35280d..8df81ca9d48ae 100644 --- a/content/en/docs/tasks/access-application-cluster/create-external-load-balancer.md +++ b/content/en/docs/tasks/access-application-cluster/create-external-load-balancer.md @@ -194,6 +194,25 @@ Known issues: {{% capture discussion %}} +## Garbage Collecting Load Balancers + +In usual case, the correlating load balancer resources in cloud provider should +be cleaned up soon after a LoadBalancer type Service is deleted. But it is known +that there are various corner cases where cloud resources are orphaned after the +associated Service is deleted. Finalizer Protection for Service LoadBalancers was +introduced to prevent this from happening. By using finalizers, a Service resource +will never be deleted until the correlating load balancer resources are also deleted. + +Specifically, if a Service has Type=LoadBalancer, the service controller will attach +a finalizer named `service.kubernetes.io/load-balancer-cleanup`. +The finalizer will only be removed after the load balancer resource is cleaned up. +This prevents dangling load balancer resources even in corner cases such as the +service controller crashing. + +This feature was introduced as alpha in Kubernetes v1.15. You can start using it by +enabling the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +`ServiceLoadBalancerFinalizer`. + ## External Load Balancer Providers It is important to note that the datapath for this functionality is provided by a load balancer external to the Kubernetes cluster. From b495bd16319d096c45b97bc7d6c86d395a0889f3 Mon Sep 17 00:00:00 2001 From: M00nF1sh Date: Mon, 10 Jun 2019 17:52:16 -0700 Subject: [PATCH 21/31] promote AWS-NLB Support from alpha to beta (#14451) --- .../en/docs/concepts/services-networking/service.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/content/en/docs/concepts/services-networking/service.md b/content/en/docs/concepts/services-networking/service.md index a6bc859b79cab..ab58196c67a15 100644 --- a/content/en/docs/concepts/services-networking/service.md +++ b/content/en/docs/concepts/services-networking/service.md @@ -700,15 +700,11 @@ There are other annotations to manage Classic Elastic Load Balancers that are de # A list of additional security groups to be added to ELB ``` -#### Network Load Balancer support on AWS [alpha] +#### Network Load Balancer support on AWS -{{< warning >}} -This is an alpha feature and not recommended for production clusters yet. -{{< /warning >}} +{{< feature-state for_k8s_version="v1.15" state="beta" >}} -Starting in version 1.9.0, Kubernetes supports Network Load Balancer (NLB). To -use a Network Load Balancer on AWS, use the annotation `service.beta.kubernetes.io/aws-load-balancer-type` -with the value set to `nlb`. +To use a Network Load Balancer on AWS, use the annotation `service.beta.kubernetes.io/aws-load-balancer-type` with the value set to `nlb`. ```yaml metadata: From 71e69fd1b18c29f926e61eee88b934e98b208b9a Mon Sep 17 00:00:00 2001 From: Vallery Lancey Date: Mon, 10 Jun 2019 17:54:15 -0700 Subject: [PATCH 22/31] Added explanation of alpha non-preempting PriorityClasses to the "Pod Priority and Preemption" doc. (#14599) --- .../configuration/pod-priority-preemption.md | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/content/en/docs/concepts/configuration/pod-priority-preemption.md b/content/en/docs/concepts/configuration/pod-priority-preemption.md index ef5cf9b6c3a18..856bdf5d92713 100644 --- a/content/en/docs/concepts/configuration/pod-priority-preemption.md +++ b/content/en/docs/concepts/configuration/pod-priority-preemption.md @@ -77,6 +77,13 @@ when a cluster is under resource pressure. For this reason, it is not recommended to disable preemption. {{< /note >}} +{{< note >}} +In Kubernetes 1.15 and later, +if the feature `NonPreemptingPriority` is enabled, +PriorityClasses have the option to set `preemptionPolicy: Never`. +This will prevent pods of that PriorityClass from preempting other pods. +{{< /note >}} + In Kubernetes 1.11 and later, preemption is controlled by a kube-scheduler flag `disablePreemption`, which is set to `false` by default. If you want to disable preemption despite the above note, you can set @@ -145,6 +152,55 @@ globalDefault: false description: "This priority class should be used for XYZ service pods only." ``` +### Non-preempting PriorityClasses (alpha) {#non-preempting-priority-class} + +1.15 adds the `PreemptionPolicy` field as an alpha feature. +It is disabled by default in 1.15, +and requires the `NonPreemptingPriority`[feature gate](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ +) to be enabled. + +Pods with `PreemptionPolicy: Never` will be placed in the scheduling queue +ahead of lower-priority pods, +but they cannot preempt other pods. +A non-preempting pod waiting to be scheduled will stay in the scheduling queue, +until sufficient resources are free, +and it can be scheduled. +Non-preempting pods, +like other pods, +are subject to scheduler back-off. +This means that if the scheduler tries these pods and they cannot be scheduled, +they will be retried with lower frequency, +allowing other pods with lower priority to be scheduled before them. + +Non-preempting pods may still be preempted by other, +high-priority pods. + +`PreemptionPolicy` defaults to `PreemptLowerPriority`, +which will allow pods of that PriorityClass to preempt lower-priority pods +(as is existing default behavior). +If `PreemptionPolicy` is set to `Never`, +pods in that PriorityClass will be non-preempting. + +An example use case is for data science workloads. +A user may submit a job that they want to be prioritized above other workloads, +but do not wish to discard existing work by preempting running pods. +The high priority job with `PreemptionPolicy: Never` will be scheduled +ahead of other queued pods, +as soon as sufficient cluster resources "naturally" become free. + +#### Example Non-preempting PriorityClass + +```yaml +apiVersion: scheduling.k8s.io/v1 +kind: PriorityClass +metadata: + name: high-priority-nonpreempting +value: 1000000 +preemptionPolicy: Never +globalDefault: false +description: "This priority class will not cause other pods to be preempted." +``` + ## Pod priority After you have one or more PriorityClasses, you can create Pods that specify one From 5532ab32c1bb3e863e6e3d78d6b4d8450170af7e Mon Sep 17 00:00:00 2001 From: "Babak \"Bobby\" Salamat" Date: Mon, 10 Jun 2019 18:38:13 -0700 Subject: [PATCH 23/31] Add a user document for the scheduling framework (#14388) --- .../configuration/scheduling-framework.md | 296 ++++++++++++++++++ .../docs/scheduling-framework-extensions.png | Bin 0 -> 75195 bytes 2 files changed, 296 insertions(+) create mode 100644 content/en/docs/concepts/configuration/scheduling-framework.md create mode 100644 static/images/docs/scheduling-framework-extensions.png diff --git a/content/en/docs/concepts/configuration/scheduling-framework.md b/content/en/docs/concepts/configuration/scheduling-framework.md new file mode 100644 index 0000000000000..c83a026b4eac0 --- /dev/null +++ b/content/en/docs/concepts/configuration/scheduling-framework.md @@ -0,0 +1,296 @@ +--- +reviewers: +- ahg-g +title: Scheduling Framework +content_template: templates/concept +weight: 70 +--- + +{{% capture overview %}} + +{{< feature-state for_k8s_version="1.15" state="alpha" >}} + +The scheduling framework is a new plugable architecture for Kubernetes Scheduler +that makes scheduler customizations easy. It adds a new set of "plugin" APIs to +the existing scheduler. Plugins are compiled into the scheduler. The APIs +allow most scheduling features to be implemented as plugins, while keeping the +scheduling "core" simple and maintainable. Refer to the [design proposal of the +scheduling framework][kep] for more technical information on the design of the +framework. + +[kep]: (https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md) + +{{% /capture %}} + +{{% capture body %}} + +# Framework workflow + +The Scheduling Framework defines a few extension points. Scheduler plugins +register to be invoked at one or more extension points. Some of these plugins +can change the scheduling decisions and some are informational only. + +Each attempt to schedule one Pod is split into two phases, the **scheduling +cycle** and the **binding cycle**. + +## Scheduling Cycle & Binding Cycle + +The scheduling cycle selects a node for the Pod, and the binding cycle applies +that decision to the cluster. Together, a scheduling cycle and binding cycle are +referred to as a "scheduling context". + +Scheduling cycles are run serially, while binding cycles may run concurrently. + +A scheduling or binding cycle can be aborted if the Pod is determined to +be unschedulable or if there is an internal error. The Pod will be returned to +the queue and retried. + +## Extension points + +The following picture shows the scheduling context of a Pod and the extension +points that the scheduling framework exposes. In this picture "Filter" is +equivalent to "Predicate" and "Scoring" is equivalent to "Priority function". + +One plugin may register at multiple extension points to perform more complex or +stateful tasks. + +{{< figure src="/images/docs/scheduling-framework.png" title="scheduling framework extension points" >}} + +### Queue sort + +These plugins are used to sort Pods in the scheduling queue. A queue sort plugin +essentially will provide a "less(Pod1, Pod2)" function. Only one queue sort +plugin may be enabled at a time. + +### Pre-filter + +These plugins are used to pre-process info about the Pod, or to check certain +conditions that the cluster or the Pod must meet. If a pre-filter plugin returns +an error, the scheduling cycle is aborted. + +### Filter + +These plugins are used to filter out nodes that cannot run the Pod. For each +node, the scheduler will call filter plugins in their configured order. If any +filter plugin marks the node as infeasible, the remaining plugins will not be +called for that node. Nodes may be evaluated concurrently. + +### Post-filter + +This is an informational extension point. Plugins will be called with a list of +nodes that passed the filtering phase. A plugin may use this data to update +internal state or to generate logs/metrics. + +**Note:** Plugins wishing to perform "pre-scoring" work should use the +post-filter extension point. + +### Scoring + +These plugins are used to rank nodes that have passed the filtering phase. The +scheduler will call each scoring plugin for each node. There will be a well +defined range of integers representing the minimum and maximum scores. After the +[normalize scoring](#normalize-scoring) phase, the scheduler will combine node +scores from all plugins according to the configured plugin weights. + +### Normalize scoring + +These plugins are used to modify scores before the scheduler computes a final +ranking of Nodes. A plugin that registers for this extension point will be +called with the [scoring](#scoring) results from the same plugin. This is called +once per plugin per scheduling cycle. + +For example, suppose a plugin `BlinkingLightScorer` ranks Nodes based on how +many blinking lights they have. + +```go +func ScoreNode(_ *v1.pod, n *v1.Node) (int, error) { + return getBlinkingLightCount(n) +} +``` + +However, the maximum count of blinking lights may be small compared to +`NodeScoreMax`. To fix this, `BlinkingLightScorer` should also register for this +extension point. + +```go +func NormalizeScores(scores map[string]int) { + highest := 0 + for _, score := range scores { + highest = max(highest, score) + } + for node, score := range scores { + scores[node] = score*NodeScoreMax/highest + } +} +``` + +If any normalize-scoring plugin returns an error, the scheduling cycle is +aborted. + +**Note:** Plugins wishing to perform "pre-reserve" work should use the +normalize-scoring extension point. + +### Reserve + +This is an informational extension point. Plugins which maintain runtime state +(aka "stateful plugins") should use this extension point to be notified by the +scheduler when resources on a node are being reserved for a given Pod. This +happens before the scheduler actually binds the Pod to the Node, and it exists +to prevent race conditions while the scheduler waits for the bind to succeed. + +This is the last step in a scheduling cycle. Once a Pod is in the reserved +state, it will either trigger [Un-reserve](#un-reserve) plugins (on failure) or +[Post-bind](#post-bind) plugins (on success) at the end of the binding cycle. + +*Note: This concept used to be referred to as "assume".* + +### Permit + +These plugins are used to prevent or delay the binding of a Pod. A permit plugin +can do one of three things. + +1. **approve** \ + Once all permit plugins approve a Pod, it is sent for binding. + +1. **deny** \ + If any permit plugin denies a Pod, it is returned to the scheduling queue. + This will trigger [Un-reserve](#un-reserve) plugins. + +1. **wait** (with a timeout) \ + If a permit plugin returns "wait", then the Pod is kept in the permit phase + until a [plugin approves it](#frameworkhandle). If a timeout occurs, **wait** + becomes **deny** and the Pod is returned to the scheduling queue, triggering + [un-reserve](#un-reserve) plugins. + +**Approving a Pod binding** + +While any plugin can access the list of "waiting" Pods from the cache and +approve them (see [`FrameworkHandle`](#frameworkhandle)) we expect only the permit +plugins to approve binding of reserved Pods that are in "waiting" state. Once a +Pod is approved, it is sent to the pre-bind phase. + +### Pre-bind + +These plugins are used to perform any work required before a Pod is bound. For +example, a pre-bind plugin may provision a network volume and mount it on the +target node before allowing the Pod to run there. + +If any pre-bind plugin returns an error, the Pod is [rejected](#un-reserve) and +returned to the scheduling queue. + +### Bind + +These plugins are used to bind a Pod to a Node. Bind plugins will not be called +until all pre-bind plugins have completed. Each bind plugin is called in the +configured order. A bind plugin may choose whether or not to handle the given +Pod. If a bind plugin chooses to handle a Pod, **the remaining bind plugins are +skipped**. + +### Post-bind + +This is an informational extension point. Post-bind plugins are called after a +Pod is successfully bound. This is the end of a binding cycle, and can be used +to clean up associated resources. + +### Unreserve + +This is an informational extension point. If a Pod was reserved and then +rejected in a later phase, then unreserve plugins will be notified. Unreserve +plugins should clean up state associated with the reserved Pod. + +Plugins that use this extension point usually should also use +[Reserve](#reserve). + +## Plugin API + +There are two steps to the plugin API. First, plugins must register and get +configured, then they use the extension point interfaces. Extension point +interfaces have the following form. + +```go +type Plugin interface { + Name() string +} + +type QueueSortPlugin interface { + Plugin + Less(*v1.pod, *v1.pod) bool +} + +type PreFilterPlugin interface { + Plugin + PreFilter(PluginContext, *v1.pod) error +} + +// ... +``` + +# Plugin Configuration + +Plugins can be enabled in the scheduler configuration. Also, default plugins can +be disabled in the configuration. In 1.15, there are no default plugins for the +scheduling framework. + +The scheduler configuration can include configuration for plugins as well. Such +configurations are passed to the plugins at the time the scheduler initializes +them. The configuration is an arbitrary value. The receiving plugin should +decode and process the configuration. + +The following example shows a scheduler configuration that enables some +plugins at `reserve` and `preBind` extension points and disables a plugin. It +also provides a configuration to plugin `foo`. + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1alpha1 +kind: KubeSchedulerConfiguration + +... + +plugins: + reserve: + enabled: + - name: foo + - name: bar + disabled: + - name: baz + preBind: + enabled: + - name: foo + disabled: + - name: baz + +pluginConfig: +- name: foo + args: > + Arbitrary set of args to plugin foo +``` + +When an extension point is omitted from the configuration default plugins for +that extension points are used. When an extension point exists and `enabled` is +provided, the `enabled` plugins are called in addition to default plugins. +Default plugins are called first and then the additional enabled plugins are +called in the same order specified in the configuration. If a different order of +calling default plugins is desired, default plugins must be `disabled` and +`enabled` in the desired order. + +Assuming there is a default plugin called `foo` at `reserve` and we are adding +pluing `bar` that we want to be invoked before `foo`, we should disable `foo` +and enable `bar` and `foo` in order. The following example shows the +configuration that achieves this: + +```yaml +apiVersion: kubescheduler.config.k8s.io/v1alpha1 +kind: KubeSchedulerConfiguration + +... + +plugins: + reserve: + enabled: + - name: bar + - name: foo + disabled: + - name: foo +``` + +{{% /capture %}} diff --git a/static/images/docs/scheduling-framework-extensions.png b/static/images/docs/scheduling-framework-extensions.png new file mode 100644 index 0000000000000000000000000000000000000000..b38b3ad8d76da0e48f86c9c46fe563b300b17875 GIT binary patch literal 75195 zcmd4&1yfy3*9D5=79<3Bhu|LE-3buf3GNo$gS)%CLvV-S?(XjHeinJ&@BV;Wb*fHN zNUgnA_v|TSjGlWFEH5jD0EZ0+0s?{{Aug-{0s?Ue0s^`Y0|i`xYz{L70YOi5RZ?|O z&~qZRv9mTZwJ;=faIrBYG;}sK0s(PeDo@#uCTT(Qe+qbu zf0VPbA!EMN+?jXa*~Kw^`*7R$i%x(4{C2^}et2aiP8;t=-@VR6?CqPQw#Vw%hy6yk zSIed2W_&)G{o9Vr%H?*ex7Tx3m_ErBdgpB?o35*tBT66S-{_o`(d%?dl_VMWygxnr zqdX(hUA&%YogWQCbo7m%kHaolQTz7hVIMWZwEMN5e!ZLqgiP!sb3A9%(Gys_qxb0; zIE|I;pBVq*F(Qs1HCM1EYCUJS{XAqtKm3CFCf7=TUwEmQytVT^)B@47z;@In$3xR$ z*wJaFy>eP60Q2?EU*)_I>0T9~myOR_A;)v&(*K4Ah3YZsC#}swM`Z8ne*JUnA(fnV z1asTA(a;t>-(I2L`|$X4_}KSD&O+Ti3bU}`(gLgiro#0!5x5nXUY6B|HG=DmxkxWb z3qrrcfF`CctA6{?=oK+b|J40XmqF3%xnaXLEB+woU-}GmQJ?`BF$Q71&s(^*Wjlq; zb_6*DKRfex5}5;6en&|lu6EWm6*Rh4$h~pE*A}swL<^dk5XPVxA93s?DULP$)Kj9*?un74%b%H&tjz99F{{X*TQak5ha{w_TM03(tDECOI$deD zciazK3nb9%i5A5-oS78Oj2jPQ9_1b`s~K&7feXFUaY3s&GAp%ipv=c}T5?I$a$0_s zH>XH*o*FDnb6q+-S93GGHJ@8)7PSt-P~-Ny8QgBO4NeBd;(XO#%|{&@A!D zU=!H_MsSqex&v>XaL@RJFNp=}NYP7)f5$Rr1aT*GXbFC^_*9_7VY9R>A$_{tnrX$s_W0&( zGfU#>`scB6e_!M6;viG>;c##dDe|q9GIf3Q(YnyWGsE6AcVtIA2W5NT)g3=~8Q;i9 zEGL)OwDkL+hiz`Wr5P58W>A$DxD=w$aWc`4qJ4ECoz9X(!j@KP4~Lz$C3}lj;fiQF zg@&Xxr_kl2Vw)5)@gR43Bdckqw#JYnc?tdw{!DA3>it5XUSHj5I&TPANM48U_iBFy zkMGs_D%IbItj3i@2Wpk%L55wyR{IGQG1` z2SfHZi6}cJf%;;p+L*QA&vvw~8p_|YOJqiG`5|hk_(S+l0?DrjxyCqsbhj4lcQ9M` zX#o(0lIdhj%X7t`umiUg{Ts?q+SZxz1{m0-_e?z8qYiSRsUqTxmVTxX=Gp2iJnySb_V%yBeKf1)DLDhpnkc0ju&GULfOiV|6Xxo zyRckA)7*t_>-!G{N2^r5D*eB{DJlg7zm$b0I>dQ9iqh2N_e1+~2JQGc$c4H56`)Iu zGx&*_$gqU`GNh|a1|?+J|N0nmz)Rzrul@G6CT#_TcpKNkDJm(ew4XUjV!*AcX1)S)O4cg z;SS<{G&V7S&u6K*P7RJ4$t$RJJxuu!pBdHrG{NeUW%B>rv45ssqU2j!Ki@iu><{+U92*ifopH?+h7qU-vvEWUmZEZSYY8C z)}vNqVYhfZk2w)oI|=c3i%@r$*`sI|Z!24(dk;>eMdGBz#ey=O>XxeyTQ@DCO(9gWG4>E%lUoykOai1K5X-P9yAEA;$i zCw+@c4|*}aunjV0P?=;EM%&4#YnwyG!Hzf!astN3Fb~S#^k}hzeG$h5p|>

=3C5 z>Fd)3b4ZJ-ze3UwkiMzr4O)oc{1bY3g6d&f5v-Hz3o1V3M&df!b_3VTadiA>5`ehyffW0_?`@29unxc8jc$>0KQ7!|urG z7u?t$P&uU7W&%_wVFx#iuV7bCmU}->{0jU1^r47j4Q&K@5NVfU_~A4XEonuV#KV15 zViS09NH70RDzzbRi+vrSQ0>eY7&%A@aaR(`;vaUXLZAVm-K{Scat<=D`h(BGPX6E* z#qw14g`t56#SAhI3_XfUQ>PInf9&_{Crd(YHWIDNt9CqXZYlLFL?=T8nM^lT@D%0h zh*!wrvMV?p_D0CH_$MCd6|CStpi)#;AS1F*Kbbd6X$MS#@NkFqZ2E;pe^9>)7<13i zo|44wNTLLHOEgLgnsfV7Z7$RP(V{B)v0x?TVm;a;A*cDeXHcwVDm2-!k4~(p?vSNX zV*3d~z!{zjfvGx!gW$6Xu__pwn+>uHYx>rvvz3d#-xL*?jfEuw6gvM}Ln+f|79zS# z97WSl+IjNcC=!|L(lJo?$Xr9>F=<$ON8S6-pHgVMswt|JIaSd$GmHjj zFtpKRb&`rRbJZGQnST@ZI%DGNnC0qu&LbjIS)GvBZmvJMrXoGyN5a>{3;y0}*jnEp zW%@Bap{k`xh}D{`CF&zGU)EWg3zOgy!n5b`mxR%FjM*m$VsuT@7u<(eAVD!_4J8w0 zdl#mWPS7pHi*3LLVllzU(L_|gij-Mvvy&QmKp-ZGP??&v`bHLRmbS`FTv9Q`6enRYwDx4Ea zWf`7r2XWf&--4H+U~0a#-CaDHAhF#dBL3<-&rRI@YGq3@uC^jJaq<&ls0)HoP5cKO zqiD`F+c`#k!h;7Wstzd5*3G|B#+=kXJC5$L8Q$eMz*gN8t`bYOrRbq#YBkfbAzv_;*q-eCz#nKLF7W zpTYMx*E%GuATL``n2U{fb}lG*`u;L%1RoUbPz(`98)ofb9VTZjyr!DZ?+NN!q#NlJ z9`sx@fdQzWMBEW!f`T-++6xa%8G+LzSLp$HB1JLiCYK5d*w9V+INsEOd;Pxr6f=3N z-fE(n-;{EMS@L^ceY%I+UPZLAnj!5EH zSlxZII7E2rw5^Z{3|HThF6E&Ce599Q)>rUjbEq(Mq_1Q*#2KJ6LX~?(p{~R(u_*_v zQWX=SgOC#$mmGWBzttINU# zg&E8k=>*8X_2Pw264R6+NczKFFuzw*6x2j*?BXVa&RQFhn6XF9Ud4Q_=AAWf9oSKI zdNo12QTM&jB0v2^9THUS+o=|fnx&4dKj8m|BsfWO45N^!@%vqR^F~{CTr~w&RPAFP z$fE6!dG5=LI`rT>6{dXNLwkqq=pqAoQq(p!x7y*aL^us`Q&3mm$uTK=XjgE#l!M_u zv3%ylq5KrW5z~Xpe;dAU`e@KUiD1NINf7vt$x?KKuNs_58g2vYd9b~@;eAeT(>q#UVp@#-M7C8ZF3^sHrQ3=VRz0Ws>fZbM)A zMjDH;)b)hg{Txt3Bgo@wVs{XW=!(9;6eZhmgrNF_(DaO95g&by&l?pxy~qlnzQYwm z%lMV>(|?^GtP15Gh3JIk`{n`jSJaS_32Ejph$~kh4dQpupLU#0JLRMHxD*xjAb%Uv zGu?Hsk`>2$Gf-zD?Mh2@Ot&N?4G;=}KVTG#Z(!8y528#mBY_Z;4ke5E*C&jyi#HQ< z(rDHBXcY!@;$}g@a6kHwMq+xmu?!J)DAj!w2cYd_uFpErNNulkE~f~5lw~&8%JO+=!@_S*?XPzs zikR({s_@QIptFP;AmSh6^vh*lk=WD+(8~9}6pa5$T7Z`~i?-Djat?72(dAq5#t+7y zGIz?x$X#ngH5Z*)82Ygk-bRgTqAVT05XE*w=ZXA%+&*~9%EOnTs4-8rT)5u6M%5Q; z?`e!X{H(zw{_a8SmJ>di*jp6I$OQ>{V~fK3`~}Bo5GHK}LD0oyTPVG-JjL#3fS5Dk zMX>|vUgHo(oxhTNO+0Ff5YJh@f92;CP(wwkn2^HX5&8ypEM&Ky^SrhEVo0%R#@OmM zUDBTWcnvbfgz8s4xF*^4lcQ0b=-A=AEYy+I+3Cnfm~^le1+PkAEwL-BJy`m%pAPql z1ed_1dPt$8>z3^0L=MJtX;Y-nqJEn~AGB0b59Y5%1@@rzmx-YN{33MQ#d=yI|6F9j zPUf)>J3x^&kvA1WGiyw76mkgEd}-$ zA+6*JSLIxTfw0Y~Pv9W#S($kWrp0j4i-;~L$Gi3Xuh>^%4A<+v z&MRg~Lg+M(pE}jBFpWZd+e9s&$!9O#ZNb9f1|v^yLjDmSbTjmS42=i9M zs{Rmg$e_BQS(UQ9QfxMj2>ZkD8ze}4e3A!tE81~1t__0^g!0hs;^+|4bZsg3WF6ar z0c@lVL0c9CVmr6V>KAJMo2l&|6JF;djKX3t7r}{t8FQcSC|^7h5hr967?#-LEGfV@ z)naX;Lyn~J<~e0YHl7%$%){Whg^(nLa~CZJKZ_M&pD!KVl1Hoe3#|U+R=ACaS5M$X zfZ`t3K1y9GS4T_0f*bk$GwvAaYmi;TLdW5?wPDx z?U{90NHtj}hvR%zpJHGr3(xu~+fPIP>XaC^qLz8)$0zijm6J*FgR>((`bKRsx$E#p z5k*~V@d{%Qa-BD4G?pqTFXi;Xm~M7upTsJ|8Kfapdv39+bUN-jxE)ZJBeCa>em+Iv z5rJ)Q6q*lB8zL*_I-HrI`vj}5g4lGHobr?U4&8I4iu>CqG_oy`za-BRrcq6JVF4kw z^ZD-%#}BV4*O$m|pLw{ii6Y%a^BYFuxSI*@q1d({bLQ~s_SbHTcM{{MKtAEwChB#K z;LHTQS!^%v2)GbNk&AYL{kb+vSl!yWB<+f&I{Dq5rDrP0vNL#DRYpAP#6P5E^(nH7 zJb~hiooWV=(jWMFI!hIKQDV-(VMnH+p9r%MiavMJlLYjGZW?k;MfzVNQwowaez*Nf zK5+7&4|HRlfqv?>yH0|?S&+Ad7CPjTNQQNVN}8;XB4CxDx`HLOJGdkwlM`c3&5Cy% z%Ha)7H4)cj=8%q3LsK{Lsx>%*>jP_$Zeo@x$0>!3g4X3%j1n>oXt`;SPo6NCz0$h7 zfO$h&^KVL<)8IedG-(9wkjI(ufCOjw>Hy7qUsvVaFg5lqUajcqoOot+$?N%GnPTBB z@2F8pR!h9u6ywTh0il!633>ib%si!)YsrkfZ$pigWSsXv!@qSluw;9fVsBV%#!*}S zQKak+n#ET*Q<+4fWGI;kX=t;4 zPnDPvGqSZ?Sp7*b6lLr}`yKerY9DPUc224=;R=%wDNFV*SXC#ksx12_VG{P5_Ws&f zSP^`zevx2qjnDjeL!?L4rl5a?E##eN%|eZ0F-g_vMN(hPRkK8vqei)eIFtjcy@f%Y zJr5Mi`$R%SL+|xxDT(kSltH^l+>l8-+D#cI=hRJECOf`07q{)FBKp`~NG68w)b9wd zeWjqj%A%+}XlGC8jxq#kjrmrLAlIF6VjJ^KnjO*r?a-gFK^S9t@b}&s&H&~%`?uBx z!lQwf0dHIRV}}~w>7|o;*>yM4GwF~Ugj4y+HK`V1A+EhiEAFn83#uTc`o>5IwUgS~ z?5;H_MH6!Zm)Tk$&RTJT2@o_Z9Pl$*6GY3)i5ZRVD{h77vWOHg4G)j@K`@*d?=ZB{ z(f~3>?j5^>{lCVH`iDhNd&lHiMGK1qHr-VfaM4-Urs`F3*Dc*AT+rTX92GFjC z(w7nw26_MZ%4#c&1+Ku_h^yIyfY6bAe1kgY^Zf!YLOV#vh(K>cd_loxS+P7M00AKc zkq{P8a$Y)Fb2FWK%PV8(Jv=jm#h(XHUVLuMv!?y# z^W)~qRP#ONc`)Mtw+CB{Bo!DkGBV=oWdRwQNftFw%p;OS+NDr(1;A~X00 z2(?uS{)4VMOGgLI$MRw#!v98IQW;1E`o+G}hEN5hgrtUXche}HzB;%MGk!vsIsfjL zk$Kzh%OUf0s$$MRT#*JN-?-t%X{&1ts|pL} z$y+HP2L%O*BPV+k@q*pqWJMU$mp}-8v9A_}3KAZAPwBfE4=6>Z9IRD-}`#%@4V+E2& zTwJiGSHs`ew$T;DT~8$EAUS!gNA>v%#tCeH8AuFhX>NGt#LcBiwivJ`PvAhgp2i=J zQdvIki}!O+C_Faf8ecUN1zc0bvho9%HpC36YVN}69WJVHETEqrH3uH5L|q&f7BqwH zg(sxdt##?7cPe$vh-Wb|sd>y{v($Q*>M;W~qPt-* z+)B=bH6h1xuXhNGf(PYFcxLW9hoGGRusBdv@?`7Yx=nXJ1K4_8$_@8QRtS)j!yx9b=-6yWMS zwosYNjiasH-Vdwz@^=hu zoV(XP*Re#NB9#vH>jT-K%{0%tJ=&4p+sF$B9-urvMgtu43feKW!SZqY@HV&?*qKV^ z%NX{`Kq&0+=ec#8HrHL25aY&UqyUy?*9Yw7!0RFWxF2@-yH)u8y;K%q-CEQJYBG0D znw2ci1Ot)z4yKzx1teXa|*lH5yy8VMH!x0 zZLKK09=Bz?Ujt)%$EW~%x3|aLf3p1+$w^?^xbE5Th9|H_eN-VI`#*&7|FJ-rZW~b+ zsl`p;XTX2RWqvFBSFBW2h(iS}Paj2_uK6dT)yBwDr1$AGUp%e!F+GSTa8G}PDgUSzu^9>T ziEJGrKN2llcoUiKYBoifDzy8qvF)8RmxbwK+;ni6a;>y}%)%^*FvSgGh<<I^yj=md9cPsOSJ^Smy65|}}?Sg5m;B5269&EX%4Zm?EscpSBjB``H`P8O3jf@cZ4lBCik~sQOz2Xt4b( zwyzqMTWr^dFNB8VqRyz!R3t3J4XWmnt0@v%h>B*@ujG>mSG(gh83vpqo8XkNMu+#g z)7&bX^zE7{Trn=1eGMj|v94+uY}hd7DjsJA6N998}!UVb(MQ9@V&9RLK-Bg#Y=j?)Q zVr30*-JW-Pzgb*y;ku!^a$+P`_h($&a2e1yi#(=@!OXh~l5z1Erx}#_2b#v+5^4$Z z_VaD;Js_l=CAY%onE)9Ww<0gO%~Q{q=^-OK-|do(y4^Op(ESqjb;HrIBJgn(&$R5B zdF7;L^6*_cX>;wJCzG>;Fdbk8V4#+w?cjyk-CuK61V&@+mt2$T8o{HsS}s_f;LR~- zvWUvN3ve39=TIkPH@nb2MFk&cH}^db73|v6=x9PSN5;NS@Y$qMn2X>G1l{59EMo>C zu*CUWzO^vp=Y3xQQP+N?uQ0$o9%=>K>0Yf%Kj^BfLnx(@jv903E2rOcTVJrZ%YuF$ zA0}Dv^T*_FQz=BdXnw*z+EhatV+(unE1tDI;Yn5XekX&Q3JwylZrw!vaCq2o=Y6NF zF4{$Ts7Dx&MGUw2^?E?RBSK_&n=K+5VfdCl)Gsx!&M#GF&^LeH7RKtBs?ZO>Pizzs z*Inf+9wq31@`mlQN4>ZaJaV=HZ(CNj8}5}{44zqjQ&8UJt!!S1@aI`>7I1T zPBE*D=UGbKjR;%MW_$dT_t;I_q*Mpl-PpdzWBG6UKBsd8RU;a=R0(Um~1eH2`zhyXxDe8qq0vGSPw_kO|dcJuYhk@Gw=geaW-P{mnNK(0vqw}@m+b>WXF=w}tD>!oYi=bc+0?y{hL_b78AOzZAz z-8o8Kl#8s}b5YGtQ&{8crKpwta+=4Cjv7SM?B{!~Xt%BpO5U8>vwpAC7@MjcU^;e| zyj>NYDPRSRzUFOT`8ubhMjo48zr0+tanprs8}%7@g2B@jIP7Dcd1>(9z|yoC-5=;Q z1K!U|Re}KauHZX`5Q*bf+;RZII#9P3{3q8x58A;r2)U% zu*ZjX)S*>XBn?!>6s;!oPHE^T4T97vn!jI)ybgi&V^H%@01XmTJAcvG0fg%Qaq8j$ zrF(nrF>kUZXy1Z+4h7W$#(=(i`+c9Be)UQs7moow2??HR@3^}Eg1yi0b=wExLtid? zvTDx5Tp@BMrwx(PyTsdn8XAkzGhRlB40M)f3r&e+)jYmMhvnAvo*&y#uezqtO;2t9 zKH+w|o-cpKXG<%OA@Bw|1=We9s;uK!_r=jsH-Gaydd~d0c@xw6=)-3pLzAA`yL2uS zxi=~Umfb*{xiZ5vy7PSQ)w8>LU&-;fzO?h*W}S<==c)D7x#pAYn2)TfMQ;s03p0GB z#4p#ed3nT+SSMZc($p(XtA&h2m~KyM&A7`SNJGfSw+*Mye$)2b@Z;sg*i(;q<6spe zX+t()q1Vi7UXIt|=2=APu{#-p{dX&8U}@7Q0LkpTS+Tju?bfVSo%_0Hc#T%I{b7x) z-(QKZJAHI!CHYSwDBy;0iSNG%aHGugtg3Hgj->AE6BtzOa$yiGb(axDRVvgH;;}zuRIggz?3x|48ltzh?c(1i`a8lL=tl(#l#Zn#W8 zcj%&XY#uIvpvY_52vce{eb@&TYd_rpi zf2I(peDa+Z&vy{du>G7aQRSi#X?*e`wPw2B^bz}?0F9X=l}H_ zCrLlISskuu{D|O;N0OYUXZMsFL*+4YutDeeqcMJjPXY|v{LQxeP_B>Ss)I7RuY=2M z;(9ei7?mPGgc=?xLC3mjr5md~?iCcOV2+i$iem3#YxbLHDp3{9ifs+A?CS)y+>LT_ zF>Z}*u>Dy#<}PXGgc04^6kI&(%Oc4hh2+4o1TH5ggdZRZ_$BZL4TAG-Tgcw6&%XKn z-;FB}pkJ_t^KJ?BlHu1-qE!P#my{j3kt{lyklQP7F?H2iH&gZ5&&6qkR5do(mCjUX zO5LpbZKK0pgfYj~zj2?#B+P!}`)g~cn(Cu@3|dDOSxb#`<;9YDZDz5`Hf4j)2QD#Z z;B3buJ;(BdGqmTX+f;W#;?q*L%=afDxuqFnxDNj}ANX?F{LLT00b^cr-PP%ldfab_ zh1|2ff8+xxZdaYdemurW7#uUo%-KJtnNz^vi7E%4Ox?1WpcO6ulurJrC&*oS|J<+fJM&9jF*Dv!X zWCOqA5hvFk<~SH;h>VmAr5f-qiaGQ~#!CHPniCU~jVHEZY%r9_zFq$f4hkATI3h>$ z`2aHUY=2!WsN~>hjwOq*>uitQ8~}#@8w4E8kdTk5{|!_W9UA))mp^b9&pRD|OaAY2 z5UK*-2J;3RF1USirzlM+_P-Oil7tB1hs6pf#AO|$os?*S|G)eH|N4VEUI{`WKnY6F zn;{Eb9uG#T8kh#yd+TjK{0qzroV)catt$b51!!ffN>Q?s$1?INSr`+YPexyRnzcUm zd3A?!K^p~e0&xeL_=5)jvaTKJH1|H6nAnVM7#s3%#}yXb70l0jY>rJF@#W>U$X3eH zSI7^x7)!b?%S@^GOIFMWtKm%bh9>{iZVXyX>y|TS8fc9gob2we&~CEpedKvD)TO@w zy+A%P-7n*uz&Sv2=oaB0TR$IHF%M9xL{&aU|10QMjE^d=`=j?%Sa>M*8pCt<#b0`4 z4_7vQYygyt^QmvSBqZ7> zQ(&a~ITHgjI|%U3Oyv^s@eWtoNM@}n3KYHo=|u^JLI3mWuzq}@RS!q!T#AL)@Oh&8 z9*Q~!xl=le&O0*lewGeTT?u=pgPNzMo~Ue&wuFV4n62s>gXz)ay5m7t3)gMyU8~^V zp4pSDZqibN4Xu#Qtmb3-GjN4!T1YN0FTcZn>@+TmKrX{*R)gSTa-Hq%(tNLyO^wYO z93Og$+WF26?Ak}%d!XPpiT6;z_1>&9!;YAEOW`San#R+BmYNq3f`5B;sN;XN6ceVS zLZf^R-NYFl*po@-zJ>)BQ^@~dO1pF|Af!hPyt=zrvntHA1e|o>SF!^1a_1YrEc|)mtnWE#Dw8V6rO+{9y-o#?LZ~A5z&L!BjLI8Sr^{ zay{M(4HEZxjkAp?X2!VgYHn08Wkbu0%?X0|28KLqT|5?NoRM|j$?Kfls>JkBZHD54 zFDfG`-1|qtH?i?m(lI=6swxpRN?6V~m|MG!)3b#Xk!388mwFJ&2ot^N$kD{hb^GRh zyo`|~{jzpE%Q7O0c~znyINln>scgHiGUt=6KbWnW56R=h0Pl-qDL&6JK&|p+O%?And3|$tyl(c77v*CI(a{<8q9G>k&6O3V#dpo3?|;80 zSuenEvpOyCbx@Ee*JZpQ-r#ZQ(NViP6QqJxQXT}B2#VA5x_U$5IX{*S*D(9s;YP3W z-nfKr0%8I6_LXAM1mOhiXtrZ}uJnDWy-Wc}W(Qjl;k*SGxA$Oy*JcIgM$XKWm2rak z?9~|?70RHYPUWnchKBpurpl2L7Q%u|$&Wr}UfQFp4sm2pM9b51{yLSOTCdgDgiUV4 zMg9%&&IXl+diTaHNCay%#B8m z3flvsRA@k0qwS8Ctq~cG7Mwlb&d2MKaKXQ}PE2M=8rbfiP=bA4HZgK7FDv3SSpO+J#Vrj!8;G27iom%8}qB7 zfRcg`O=r4Cxr|55`=cO7D_p*AIyB$l8+@8Ub89KC%Mvm5U-ZWcxr{zHD8ocT9~`j2wB@1d$MMbQfZj>9WSU`GY&XbB>`@ zbwOrDpGbjXkqj;3l6WRcw^ocq*Z2U1#%uVA!o}6^Di|Htl4PT=L?2)hvR3} z>e|#Je6orWaQA1#qg;@WhQ$CvflQ$C4c-iNCx-JWC&3!T2y_zPzAfYdkkcc_OOI9o zy}}q$EyWoQTV;?@4;1Houhk@a%QAIlfhGU)z>47mdtB?<&dv4!CF^rU6An#Y~%%N_@aSQwGM@BdHfV zRGwVDh&xB;?t{q6?eZn#dBLbUuAi7$ENoKgegiXoil=^#PWQ;4%4mD1Q}{4VpXKv{ zjlFzbnwCVS^W)?5e;H~e53tBozFH)o=jh8VIU(Fb!EYoEaOZaWz`c{Tc1;XGFb>57 zlo4#=kXg>AmAxc0DnTLI4bms)p#=KTii*b3j4uI&Z}sv%z5crs!>bKgBMLT>_EZI@ z13a7FSpph{JJ+D_eM#mv4<&V|Q;ohd)%^}{NwIioboRfLW@`laWiMlo2hV!{v(jh9~cID0XYU1&-ZRbL6qG^^-(RzpfsPvYg=<^I}Ni^Zk=pP5YujNZ}(0v z)0O5{27qLYNbiV}s9Vy4o3$#Lj{~`Zpkgz}O8zf}f-rv+WAg);rTeJgpaqMYL%`oK zR{o!j^i^gcv^m)5B_B3+a?aeYdGfG=et>e&&Q!gD`Es`vS(H2dguDt+vh2-{O)c^t zP!9GPc4BWSmHwY?EqmoDi}>|l&uV4JE9XGcm)H~q01oq?XqWj3gbeZ!f)FC>iB^CL z_T%=!-3Ta0OC%f&rzNi!+go&R2AP%*U=hycI!66xhP%%Zzz5R=A8KRjGw?{Xrr9iF z@AC39RRX;eHHN{IkJrm(pp)(_h`UGIj|pJ=M8dxVmzL_l>Qc$kRuSkS2#cOZz2rv( zmbERRZC(bDFZ@!*EwhWS#(!xrlfXcG`pw0*b9 zDBbXb#_u!c_}xf+2VMzrX-~IS+s}EoR4|l|2uC6pGDDldLV!$y;D4~(?A6AIAm9+Z z3YPv@fh950l-RgYD8L3*``N zxOsyAZ5^4C6p-f9L2yVmO+1Z8w5=ZQMe9|R9`BrCi1#roZYlE zeIhwH$y4%vJJ+A{37e=o7w@hnMEdS^u#=JXFaxi&`xzu(5B+h!my?y&Jz9WfclrkI z9iin`?cD47<0o=7Q-#yAXd18M-(CA`y-nw2=^5*VxMu>-u%ZM@CEX7v&qo7t$_RJ` zBJlU#lINKP6xa5*YGx~x?Se7=nQg`RcZIvHcb=}M8m!vWmGpw|w82Fha%if;dAO!i zWn{d(?TE{EBSsN<`Af@b80Z~BH{O_N2|#QI9CHZM3^C6{6bbG1 ze#*lYSIYiKPE87+GMgrgOi@9SG3Y=N^s;gdxImJ5{_nE@IW8_^H=569Ax<9W<7*lgPg-JSiolpK=5z^8G-2WWLnYB zy|9Iw!FJ}C(FC4CBavS19Aw{Z-&-vQB^t?WzR>S$7S!mCy7k;03Ioq+5tBH(j5&Bd zpT>yoS8a`oghd2HZS2>)8#+kWzFKQ=q)GGj^+fvx!y)Mow4I8}aBRpccR#NSm2#Oa zR1P=)u#PtP*1|2@mrUTYUk7;uEK_s=^7N}U^nP6(taI?jfSbeh>**PM_ADMNCff^r z9sy4o4j_>+7tq0@KJ))xw{i%W$;7HG0qN`1y9CdAjyC?!*La`!(}?Ctpp`#8C_ZNcA4BN?Kh9X%@g; zhfJ?KJrJV(!ZpCw?kqWsn2(j)L@Y98Yl{@am7QibvVC|K&>EWfcO6M-KjyO zkCY5-?v-1wmwYw%>4AjaifHZ)BsA-{qKAyFOC@D^bc9 znsR%_kr9X?uCNDw|RujhI#+tlQ5Zvr4ojz@j8tme@PDSwyZ zhaWAT$=4pmaFH8yQt7QGKQXJdxibo|`eHFSKqUV8v(Wg9EYo69e&?FY;|f9o_OkWD zOYUNJZ8gE3H%@S4bQVbWWL{5FpeB(4L$p1hw@$Z*&+wL$-ba46FSVpXTDM!DCntK6 zh65mxfzKBsL^)V%wVoUxlX}@(A*cHA0#9dXSp5cc6R+B`^{x|7Rse>)Gr@b|Jt|#2)EN&|3Ua0+r2Fwpnz>G|& zvOMEtL!xsz55l&;wYjxlnye(!OvDt4)rQ?sj9_qN5iqEz~*+ zRDB)#UYx?ukW^h`Ue|lRKRi5KJROB2mR^)Q+apWujq3OHOqMO)ap)cjRJlHGX&T0Q z-Y}3r-(4JVi}G>?M@PyMQ*n3lQe@Tm4eHdmPIs69uzDzlgo2Ge1{CZE6eFa5Ggw@2 zSm0xqNx1|Af}?vMVOi#E|Gh$l5dhvlM$7v%u!dNdk0#C`!hBR5U-+lEX) zy))a%BO&68lO7JY)t=7Cu!RSS<0Fz&*gfrd*;&4y=`xij&=d7r6r69Unu*Oc&B>0X zi`wb?cE9UXK@8wyFZj8POBc8t-+jQN*$`;Vm#ste)pa6%M}RB9n^}x!ugJn&-Ko}i z+tavGyUGR)F0QZlPaSO$EpI+b`_Q+kmD^aoVzwU%U;O9UU&9r)uL-e-{V)ihD9AXX zXNNIrsI6**Ixi&e4BZ$iEc+|G^Nk?T7PZ0x6nz4YsmPz)sOHmmwm4=mg!RGdv9rXToRzwsO?#^^OBrLBdiwhqEn+ ztt$J4B6$Y??i-^HJBHyi{3ah&Uz&ze1@Z`lSzB}Gf~a?VJ#!zUL+Hcsz193CUkC62 zUK6AVP=|x1={M($^+C0Y8HzCi8ZW zWq>^D-|g{y*q|95@7x1_F|p93eMiaI@-L+L>rQ%P{a%_`OCAc{5OKDT+Jn2Nm2%;k%^F>JjR8m%Ntco&+> z7Pnlv!9HHN)do_2XL&~$2un2*tl2D`(_-!63_=K{_X`UhPAR!=r;YdM6T$sHqV1zAjTsxS=SVFP55 zc&ckNstznBs%{q2ov0v_dp1^Q-E(^TQThP%{Y-m|-bkqfS!lP#(}msg$xqY(7#p?( zc?4kS4;vTAbq1j-WLP&NEVB^+!(^iXOffBn=I|7@&;Mw-@1y3$GLZoKa_#wGdSzJ% z_>a7_Mb|X6o#mr++f*1pSzULe89bI=YL4s!orfB zuSkFgnLYmAz>B`JrhKebp%zM6v(I(nwI?R^oRVVKX;^8Eqk(kgP4z>q*_DKT#Mu|{ ze3`QW5%rpCP-X|8mg`!-U!?4x`#9cYPt+=SoYCRpcU*gzsV4K|XGHwIF7V(U%O_|O zg7hAsCfw_s*kn@bYVR+2^$>;@(X;LPIh^;0;!lRlJdas_^;vF-tCx5%lG4Y*5)csl z_y)-E%cOsj*TrR*BW@PO)AQ{vE2{ry!oazMSQJzw&E zFxY#jmOENUDgY}$VbiuaSs0%7KNsVD?|(~qlv4Sk^orhKx~|-iA?6Z%M$qkRT!?%> zUKi%>@P(`1^Jlt&bMhby=}YS1p(dzQvC)GS8DYMp%^Hma1xCr zjPZ|i@#Sy8CE35{+GlsGB&uixtFA^x91bS~D<)r0mc@AQNC9lQys@ujFg#%}lEEKt zmb=?+5j+6!eFKmSxfI;-=0$+FVhGmkLhJ6AH)s5suHX{le{8DLH%%-x*a4>+fT;FT zuKQy%!2H|EfwRc-gB|fJ<_84Os4UxL0e6*r2@Q&iSweP9V(_55&P*OT{*;j`G_QS^ zVT|0c4$8mRdrh}*(h?Wn>*e({xj9U}tOR|F!Z3&qW(e! zt{g;wyDzURPnb>t%DK3{cS@3DrT2V$yE)>DqCK9E`t{Wx_ywF0jv01EPJ@;~rI4&t z`bQX3Ie?1F)^ll)dhS>d0Xzp(_|MV$ar-)3$k#1-;N)p#!Bz~DM_gTggk_xW)^{*n zPEh@tX+HQcc&m@7mHRxIZ%i;1iElLcVtmb$VCh_1%68wvx2nD6a=K850rIZdLN#v> z*j!~W4Am-4zT|LpRMlZ=i;&wWb1ZqIx$Jn>2PAQi<%?Lw{;yuky_H3HAPIpE7;rGA zG3p_yHJSDS#CY=ajPvu|6GG}2Kp*-+f02;kh&pzpUe_G(rW;jbFy;iYZhgW1;OSUJ z95euy@S0bTXEM~@>S>vFRx-kI7yPV7kCzH-FMeAh&fl+lh6B~;aJbl?9sVJafpa4J z`!L5Zi`TF7$4;KD$z7`&jF;J6F8KZRf>u9=zcqn50Aj@+klT&CJaZqk`vM%!sp1v_ z2}rUef2?_)T3e`I;{$PyHTZ_oV2k&mHi|@TGD78~Q=geYf0X!#k8T73MM|VgKt-gbq){mgL`1r~8>EqLL?lE?KtSp4F6r)) z?(V*0zwbHs{D1e}uXlgI*uk1>j`@ry=31KpRj=|G$zkr%Kgb2a<*w^mlqY>jjUW)e zj5v1NxC??F8b5cYL+DZFUMF@5PFNN6M)-hsqN+dHrJaX&<$xSPle{J3>%_!pCxLsM zQ6>0Y&>^!*@RhJ*v_Bn&H4r8jj20q;mwrU1e0ad`=#{w<+fhI0tTy(dZ6mhJq=O6K zVsGQ`i$YvoQ?3gOhmMigLjc3_k2Elj9jB=t1W}lD+l!-oa3T&fPO3+>x+pA9Mz0#@ z*7|tm^zka|J{S(`ySCTk9Ghid0I-fE* zo;3(K8DAgHYzpU4N7Fc*&pQ+&Rv`ucfMh+y6+%B=>7c7i%q)b`-x?5!W4oE7X}1bg{-7nt-0 z7FKY0f&SB&+2inY`KaQr76qS|zZ_Te(cxS4-oTSH+i#EUNI(iKR_6y*sAlp(+&!!g z4HXC<10u5HyDlypc7ThYrL4Z?rQ1^U_TRfo1HZ&U6>o$7+u7R#K__>(LJa&@^VB-$ zb6i~K*SrQ8b4*om+Y{@UmF_C71joAZmMCv?H^~d}H>3Hk!Bv9fJwJI{{cqT;j|Cc9 zt20;#q2F@(3}6erTcwztdCW%Ra9RyRV<=pka_m6?uP^ByQN=p*R&2BT4?5(k@|hS9 zQOhFh$+INU_L-9C*R$u?&7((=6h8EP6e)=hgCBQ8IwY^6@}B_7^zQCP(%^Pyb2%Up zz5-|;3Yg~Y&X+a0T}uN-gyNt=;@=5(0|=~)@30)YRm(<5rF=Dz;$t9KWKr=+P+z!K z^&nRX&zhK}olD})`*v=VlUZ^ouj0)7$nfWrr}lloxp1D#hy&n5p^lNi_8wbl|lcCVh4M*+;>}&g$*)X&(1pMwg zd1!EHx|9o&23fmbYNWuaAoY?M`0*yq>@eQ7i@K6~^O6T>Qh3Z46H>c@B6|Fi;WXFe z2VRp(X+6Uk)}d2gLpb#LV&qXdD;Cny;DI%QpEKH?0R(MjB3U}S*6@mk&Y(Bdv;9T= zWQ3e+3o$`yI_~1B1xLzy=v*E*CxJWOou_Og9J*Bd{h)76 z^eA*@&Bf_>nstd{Y$k|9*h+cQQ7KC$#Sg~y)mRz0#%mJZk*PHv0QbYCDK+Ay+|O?e z*N7$)cypNAi;%jUwKJw8^1%4x<%&N{-BrY@Ln<}DiWcQ9(MAsNs=j?5#(QrM?0&Mk z3u<*4h25cgirX!Y@uX_#BZd7^Pm1EJ&=3BdjMVYv6{`B7PD!I}&Ag%4c`@YA#vY(| zzXB~pMM?i#mmF9#!&}vz`C~}h`R3|hIP4%#R}D5c*3zXf)!xH6C~<+QVNfTgX=V<^ z-5NKR%aqt!6+!b;74ZUD=ym(EO&ehhQcdNs94lKfn=A7HU_%r(g)Z36oK(6AUQcd##3W&vv@GQ z^iGe%4r}^xezJj|ltn)D`72)cb5y6E)etfyR^%NW+42(Gg8w%??c@n93!jim1YiaU zdAy6Wan@twEyI^$1$*!LD`vU(dTObV?;^7O&0gxnIOS&S-#$|XEd=>;7yc(BXSD3q)+BO5Bhl6)Ac%7LNljK*XgYA9G~UDMv$QZ_kh10#38{&iPe z=UtGir`XJbt4K)U+G4psd+1oX7y860kr@R3YBFxFQww)w4PWRBj(7}>1Ld5s*l3Zc zco&83#k0eBZiqauJxBhAdecLa5PO1x9su#2{_);S!wEyy=BE|lrL%aX0HRI)9X++>24^GXlHJ5 zJG}23ZvkY%T1~OJTF@5hCc&-B8ex$#uL&bGngetAzCIsbb_L?)3$7$#`Fd!|&o5u` zq=BhZUbE5oM&P*bm4DjTlK=$*0_WPr6?(YFN>B z*PPl_uZ%eUCcXY1yQ64BV|N&ccex5$wwy1^b)3F9aBM(Cl@&Xk9rqPm+lvbd4QloR z18jwt^|dwCTL3}3J(!QJzA^b`?1v|ukp|G9q7!P}3q3<0HY)BgNFKfEi#oJXG3@_E zE+G8ey5Ii}!G9B-ln$#RC1ZWZe*2z_Tg!d18&@i?I<_TL&JpfImB&qARSrhqJ3xJCy0knK9JRM4vU(Q@W~LU;aDxvS>2|7aIS@jsxMMb8w}L%bv3kOr-A~QCO19 ze^aft`^tE2^9yn&hC|?nfhgZVd6>*09M7~PaD5Q-!CXgZ?A1bS`%-A?&ue%p1w@d3 zAcan2dFLpYqYQmzUeiV^4E7Uj`UzpCx?|&JIcaSV-`)B*8wPS}9g2LK|CCAObpG4usNVx;;>#y$P z-aKR0$6%VtPw4p~&*B*^iT7ZotrMh8{?iBSr24KM2j`HsphT9Pil!_J;PjoGkDdwF z%3Y{TC%lY!6nBj*{Wp;c;2zMp^39evFa79&OOA%O5b*`W&)Wc}@19kEEK z$~A)@B}&pTt8=mrcdxf<`$KvHq6Q4c1YmeX<2ydWyHAiB;p3$Vl6%8*WuX7QoD^(4 zt^2~cx|0T)P9H4B^NT8;(ytL>U^$)F*s&SzzeO~^iPN2S>qtq1-W3gGA8-OBi1lTZ zbw?a`Jk&T9HBVxIE`Bhpol8AL6G)MI%^XLoG5QA56$7RPS_)BnqTaQ$8@sYTlK%+; ztJ9XCw_=10=v25A-Ip6LkByxYWc*e2Z;V&Bo^Ss&<E73<+;HMGBq*m*V4o93fQ#HwaTVC@}8`v*D9 zu4oCWy$#kN%2DVJPxjpfs_HPa;_t3IRYZduqPfwwa6e`mj!8|uDlDxfRHv?>UtxQl z(^sH7x+FUO2)oc6>6NioReweGdn1G^jnm5&A9RoHWLqIn8BNP4YH+Kkj*^| z+)A$%YZ8!3$l-TZyn@MSo#6uWdCt4Z+^O=RuiyNe0{%!#(^FAyM74aP6xicm*4Nt~ zVZxq+e54lTa$sWQc2=5sa8}&e&bf6!2)^2OIVBIQBQVMyiqO9c0w%g> z88GSX!yVfbKH-|S!x}i;&@!wje7!jJ&WHc(_6XxnI>FuP`wJj>1qgxHApORzt*ScI(AVjo9rpKfGO?K7bxra- zELKn>eor@Issy%!SppUuHjrxK(@K-QX2QslP}W{lErBUv04a7lDoY0{u{Wqpf`{2l zQPi|Nw9FJTQiUr7qZ-SFLf;}D$&J0Z9-GVp(G0x=?oVJFu!fcgf1_|Dki01ml=Fh9 zXQ#^;i)5mT5R@H~(;$5xF!S3BrBkW3As}8Jc@hLX2Y);*eB413K;yB?xn#MhI3BEM zFrtx_&LVGj4cWiX9Bho$qQqD`=1Zno7RJoM+(t4l$X%=jORo0f;2&M~*)hOTAk-VX zJc2%ge7@;>pd|m-`qCMWV&5t&QONSdR{shXbqnk9C2>o{->%=qFYd*$yKnV33WBA0hSN#CFX* zuW%n&`_cTS0#!;}xi!@Ax61a`w#lNj;{CKxx1JeuchK)h;baKeB%BtAE(7eJ(1-qn<3e_*oTClP@nR|Jt3<)>;=|EV0Ljs59^?^s(2G~Z9 z(stTi4gHKF7j0R5zow-+sW(v*7dcX}_~sy6RNU`U;+yF2pJ@;^NeDsq;JooF9knOj z)exPowdU#Q>vKFjq&cew71LE{iTfp7J5@E!Y~o!3kwpJHWW0H}aS~*IiNDlS~i|pQHQF|Um4cj)%AdZ zX6kmK%I@t?gC5I$dqPqP=eQtqkKKOjTe=ku1^ekRer;%*dX_zAUyo1CqR= zezLD&Zz?KCU7%Xshi=*T8EE2})t$Ae^P?a)oB2_Lj7mX<_x~q^)LLqJ{m}xCAU?hw zq@}q<&Atc@LiV>uM@Lgg#|W8%Wju!N7OESlrqe$rC&1m5Z_rB4 zD%=uQ_}=kK7t>nJwg2)HEv#r@kCof|qQzThL)C(SL*@JH4d=u1Bh%;2rYbE#{>dZT zbrpwqRbhuhhojZ42jRr+m(||j&rx)l1M&l*zeB|2M4jubTmv@bujzCK^PIo}>6dZ9 z2pi5OtySAuq12yKQ8av2+}k*D1-FRF*%dt_inSPH1wuV!FpW$&XOpEjP+D&EbCB@n z`bt#YfzqBO;n97dzJ5M3r zAS%j51n8{$eFWD*FUClf_V?#wpmq9NLhm6t)ddPTntEf$7|3kfSa@pb@EjTJjZ00E zP!hZW0RwwI46nYhUaIpeK%h;<&5F-BE zGEV^^ii}3={)~VKH=HU*1?rMt!((9wZ5`;benJ5YtljxUDx{b{&!#*S$boT2>X83m$;2wLD1$*E^f&#QeNQm z1RKjE<9OIR!0$Xq4J=KobL_YWQ<0mdz z;+hTONO+$FtZQBUi3d)18O`SS5D#7_%;g%XAU!J~y;)mkVRLHr2b~|Meh8f6<}rP8 zg|v2koI6VyyS8z~n$i&s9V|LXFw64*IoZIsq{4L){N@^j)Q=#iKq=XO%!Vb0Od}tQ zS|2F8FSHkc1#U22R`I;Hvt36EG{@pMQ}^*cY#s`;d)TxwKy)&C zJo6F}AA6D{D5B4wP?l5q0_45PbScZQlb_vCE%nP$1fE$dMX&C!JoES1!zPpa^nZSfS!Lt4}MWCYO6Q)Rc8a>>?u{+le&e!mzntcn^2&2QL^;$Tcs=tvKrv~x<2csON` z@xz=b6>io7usGYZeXAI^<^I||+4Ny3+p&7oXLtUz)OGDTv#{d_fr;~L!E#JZ zO^s?pA8Um98xLc`MHlMGl8?2kwdu)SnTztsgbepsMu)h93{$w~KzmzfWX&y0uywMi zu6W8YD&Mu}vMayX|Ebv_t}SL?+|`0UVklds3tGTrdz;vy~r$&qUr7!SQ zI0{GFMq66wr(Ikx&3&pmxmk!7Ivjprfqwc$awmmx>Cv;qE~uIMecm9)bB5bO)2&fX z_It2T1!)5S`as8@Ws~Um`AYG=9ZSJg@z-2sbL+EnQ(I<&4`(MQ6{{T9TWZR|^>qnmtjdEu9#i&^6pg2NCH z5%DF&7mmBSMzl9_8IBZwYsbIFEhHtlj7%N0dgu>sCRSpsg6{bXSqWIt5`aL5Wd?w4 zq_C9<3O58?+B}RuL3^2`rjxloUm-{g`j}Fi@pa4*0loG~2eG#G;|r20A)!Of9DLg~ z%WaXQ_7^FWanuSQbZigzMz7SOtg?YqFLmjCj*R(Y{3ke%#}h{KG1j+gasCPVuvy;W z!&h)Fb`K1Farg?;eiSidag!`5aqxw?cFb&z;g^h4R~Y%4R@IqeJ-w5PE$}6>4cLoH zs^tV9wR&gn*iT1FlP%k>tKEWjD`bKJ+!GVKLN4M1l3_f`^QM!!*e1wG7;ofoQ|y@A zNSNC>?A~!+|DNe|a#dizWXZK~MO~||&c*pKnAw<&rhS<)fI(U2T|m+uLqajJ$gjHV zXVvW^!z41#yzbBpH1Y`RkpvZa#W4NsFg?HgQ`K+!xYQ2f1BGv2q!dHx?w^n86FdXcs7)^R(8!`joVjmy!nsw4mxpzTN73g)L<#Uy;QLbM)Fx#0Z z-M@J)M>U}7$kne`=ZCn;uWCA+srUW2ga*BPx@1 z`0+yqM(u`8x@a~dg}nCOEZ-YMSgQhTTSy%iPn3W%gJn7YYyX?Q!- z53Y;N{;QqiT^9Py{YY^u*JZAa){_jm6n|&YXnB$bW5ovVM;Sh`oJW|vDL;4WH`TyA zC_e}+3{Wq0rB6LApLw>=F!J4&(Mm8^^C-;9*8rA)M#Cl>+DW_Pl++4)E?~#I3BXMWd0=2T~2!uEi^^Y-{KT7m7$nH zw3v;_GK^Uwnr||?E@HD`#^jh1?}Lvl3?3~viJZFAiJOG4hdj!m3N_p7BvmTz1kR_Rr3kp71&hDw%_~4Dx8-56db;_S zk4*6FP-1;$?&Iz@%KO1pk?CCea#zRq;s%>-e{b`<$3M}g$uF2DEOVcxbd@TkF#Oex zEMLBrD+QApHl;CBJj)q$+6LFUD*lomzdv9!U!W(JMRF^x6h1E?nq@w1zTWw%>;SJZ zF7O$wIfJ!AU-;KU1_tfTHjP5VVHwV>;AF;qY^Q+JR0gwegEbkxWsII;H50@iCQgTq z6F>dEz(-aO)zlSgk=$V7_$VSrt-5wL;la}v{*&)-hcD6NVZ?dEs3`NmbY>lo*{7+_384{v`fbKb#*(`tFAwf6|mUQVw$RU2tF)#WGkWa z$cWMkyL_DW=b$!*($|860mWaQb+)zPEPYxex$z{b(PdrqR-yf){B>BGcbOLD8)Jn& z@t1tr5#q;!Q=nnQoBpS4pmZ9+X_)OVb*hWi5P5y;sfx{=&B-!mqtUA}2ira#CM)dE zZj^sL;$1rNtxwV71)SXlJ|UE{#%fch-EViFiSDj%d>@YHGQ8z+B#}!ZdT;pAdC2+2 z__=FSb(X4CjfVJ;s-dflmVA3)ZO@Q=jiXK=j~H-dXt3870-#4C>!0lUW8P5 z>bs@Z(nryrH!O+>b;739P> z#$NWNe0$*I6m+r9IbwPxpopdf9KY9xQWW8^X9qy_QYY?-m7rE z7z--F?15>;3Es~15<~xf`*jodkT~qNs*r^5_6KB-U^cEl?`adsjg{ zM&3Fr^1fZaOwK2tC7kCwH@P-^04><~yBEE>goPTpyvpju~V=w60OD z3T-e>O8ij5T^Y&utiV(oAu^UMKk|43uWP*uEtu$_r*pPOE8&%2`HeGwqMnr`Z+iQ8 zKc;i&9>3#YZ+E3E6Wr1E!fDRzn}zuV_~yb|0(V}yYUyEVv3;iS$$-|%iu&xzH&i__L)n$7|;=9 zgM>q@-ZVDt8L~WjEo3vb+&o0=o<+Tfg}ZiR$D(LO)+CAzI^WEiR^WOj6A9_}AFiH0 zlCR!81#aZ$&lf`JD;;Gl!cY6oB%%)`qFtKX;8%t}hZ8H!UK6=W@Hut1_$4l)Cs9$= zrr?-0$k$FR*gPXph@gnk?Q8DL(+^4RIDgzG1{dhbp*#Z8Rfz+=Y|s}e#V*DS3)8!q zm2N%HkTOk}ly-97`IAGwx4V0qOB?XZ5t|TNk;urW$x=slbo1#-@lK^8ww|dQuy$@O zBd>&zG&;H7Rp(tmA4Wng|GPyk?aYZ5NJ}OCYtLb?UrEDTz)^fvg*y_ssZ?FpKP@UT zhtcC{Y~E2v)HVxhCS{ugbGTbOPD-34?caI!+zbc$4IK{;G5MjFzn~s9rNf=@DhUm< zMDt(uqO?gKzvbFqNBsUVtl`qZ|CkCzS&-f0wfecn02x(G`j{JMBznrM(eCw|jFXTtug)#>-|E%(s^01eQr^L$f>$uMm~;M0meI%0 z_%ZT(`0XdpHG|BF+aAmqba@LJ_w-m4yWK;OqfJqQ@xQznS%WrU3HX_1GZuK=M#NP( zY>FV2M*6PEa`gtny`4BIEb{bLY9zOoi>y4fMnf!l;%`@@f0Cgpd@fNp0`@&ehpq6+ z57?e+IZx=#alNm$Y?o|((h%^^3JqhrEp!+I zd{QyI@>T4n_$(p9JuFW#Z;RaZ=1?}h3OZuqQvHj;rMDrc*(2`1N|#H!XN4KVBB=g5 zmSUHK=IJ!W6r!R(2J;QDFQv+>Kbd77tIr3G8E+9IuEFDLb0JmZIq z55dZLXojI`nnm5&6Gm4j-G6x8bT_ockY~?NGBRQ@Z+M^B5aAxfWlYGAD_6v))&CRP zH6mRmSv;?Q*gX9caL`NstEiKN@hS&wJ99PJ+5DSJ(xUH`2&wwKvxHIa|1@G-K0>X| z%6hlX2>{@&G&zE}iO+va`<9Iyx?=Ov+sm&EJFU6*dYOd8D^3+bZlw z|1DZS$;BJgPLvB8{zZZMN&3^5VyQCkF3(NPKv=t?s zpUZH{gFMKuEv5eZkD|dVDmOR0yR^qD{`~@#2e-^0PyMiL_c_7(@g95H-XP#;%gSPH zms%{=0gkRt~-x7d@*$+MzN$VTAN8ptYmczTi6BJ0gOf79u7 z83iG-*T0-mFQXtEe`mUTGTaxp@lw6!>E&Fe!!;D(5JV@#DjD}M%AUmHm+Sa-WP6{WH5y@lWQj4qlsH;`r60?S9H^5bxb&LWw5sd z&mxJny{oMrxeWMuZvb91VTOI(!;1BM(8-BJ0v$Ym7U4q?=yZTxERLK-iKzTMsK+JU zd~>cJ-oE?L?4OGYB%O}iRLhy7?_Gj_)lfTU&4F6u50^g#n2l5I1ekx<9+O;D6g3sS z%A0um?;yF1Coz$WVuO#%@@&-wIVjmvR!{X%hv>CZ6yIS{nxhMh5$Qq^`Z>F#?xMBS zV4AtkolRSAfP|*oN6ZR~WrTZi+%y}#jQf;&ew&_rQAVT#t3%?*RTdWGeuVb^VA_U(I(q5g~8wa9@Q~ninz8m6ARJD>R z3A%zrV_+t*|9*1|=!}DcxCVKzk$(EW>?zdcCB|q@1!flctJvXEN*0C7A?ez__w!x+ zmfLTJ94bXSPC8l~koki8YiQABGoxR^y%qK?EaZCsIX;0cMQ4#x{ZQ|9L=Vsb6H3^6 zGlPoukw5TR8Aw6=`8742OQ1H%%D4m*71rc)0zLt!8Lt3m+}XQI?U%Uu?6~pB86y4C zIlnS$r*4!RYA-&ep76WCc2jY?w`#okeYMSCs`y5^;&KuYC@~>@lWD>%K115eW9$^6 zk0=PX{2|XD5mcPx+(h3$l%(OTa=nwN61NRdMW2?F@)NYd9^JA^oDzAn$)mG*hN~cG zH~hV@WyR^T>tZ~r0=cI;LNwV@7O1d)Rb*V8cLuSx#evo>mZtco*#rLha&)Na)U(iM z??oSe{ThK3LC|imAwb$T4+gSh?pii8&t6qve=x=VG?XTk>2-Ta5t*UCK8kx7d0% z)XKf;KFe%o{QX0uU!!&zv|L^Kgh5yK-AH##6nV0?2C%!6P z8$Fbs5+30^sm*Aq`Ki)DFpvEYVy9OIIexFN8^l7&O}MYKzmLstBU|JzeYfZW>t2(l zA`!$;-5DK-CBra704G|IogJ~u!G9yLNc-?#h!|dO8V&tE7+nFWBmv^}=cSY&>hon% z5+Yvi$P}0Yn{|`gU@=j`B(hg`$GP7KZLfx|;S#`!Jw%}8ms3G`;B20f9_mddrv19v z&ytmERCfzT5Cj}+B+oFyU`+eR8CzT3Z8-DHj}yL35p+$r{^r?NcJ(#OBCpk!6b4Q9 z=yB+RwFkT^e?&^}l_M($9cB9xnx;n1)iZDe9>f*m&9AYnCU`)H`rthMs76e2CZta` zUZ~l_@bVuwN`ndK4z;g`3D4huY* zd|7$Dtm4U8iTO3eNHB?_i3q(8*eMnGbTgYAFETk!aLah%;0aYsP7|Ff#FKQRpBP1_ zZ)MD8%Xl3iKN+6*I~FnL${<$E72)p!#|^ z*u28>82aAHTFm}IiRHx)S@?dGq)%<#yFY_rr`>gj`hCuiYXsJe1L-oTa1`}}sph|; z=HQ)@L7|L?WEypxNN3?XR8TZ-;d3~q`e$}C;3wh4R-)Zs=vJS7(AylkNzc@_zS3g0zg8J%$^DoDjE!0o zcX+{93!2kA9?lcn+f%A}Yj_9?(gBPW*8U#AC%#qEzhH=<e|7Ybl{JUj-9EpPB`;0- zF1!?}wm(j@4`j+=A=oCgnN&gYiY?LXS#SM{b}sm1e5Z6)Ls*uWwMSV3%?yC1;BKCL$Uc(%v| z$U@|lxxRZ+K(b^Ox-us|+qj^Nv=Bh4APqAloMdlXC*`ifnct=o%6*-+^3n=W+s;=a zxyno*A!7G|2oCb^RNQ!}GmR1x8`Ju8vk(K)!uD;<*tY|D4S+`8j=n|Zt?pSgv}V_jFTc^<9DR+vLaN^$T!=DNZVJMbIY_u0Yh(Po;G@4J*_em zM4V{GKEK+;Q9HHZ&)5FIDOX;DpfXA4ARue;B1PWx;$Pry?KUDw{zHOEb2pv_L*8s zS6{cTZtztm*SFtFaW~2D+a_LM%7|dhAHC4kil=^*D&Vj`+Fk8<@@(B&_otEsWr&&D z_HKgGz4OQ98mJ=YGBdfT6$nJQJ~I)RVsCRkp*vsGs~_k?2M(sSw{s(!;pQ?z1iFh2 z6cxzTz@?MDYUm5&tuK0b))|O;P);sZuu0b4>7j${dK+wR1Fw1PA)L_~{byr|x{;!k z_V{>5+Z^$2H&?2@TG{3xpjd}v-86vNI4aU<3U_N-49|GzS1$V0 z{>n$);o8%0%HIywwONMVZ9$EKsVB5KeSuBf760gvFAZttR8;3$@4Cl@ON0#9&gaK@ z{O(J;`D$=tdkiKDM*`=&Z5LI@a@6MS@5X_=OQv9YrRG!9*Iu=bumJ4Pr)zi>%fi1# z`Oj#{#&Hu^tU|J;6L(%z_y2b8t}x| zn(s@=0IOb9DB17(hDw5G9G4Ii~XOGdZ6`*;<`~1^*WW_)m*|%AUz238}ij8 zeJ7>2ntdlIUPlX2Z`ktFZK~XP@R9Et0@(*JR0WaVr_@=<#{=4|5$fyS%twr;r#BE2 zEmB@G=CW#@NHjzNG`xFx96bFzl7blBrlx)T)MffNHgG<1Xf0x9Brum*ulL$IJMqtI z(hF9HzcmTj?E!v;QVWT|rEo8~uD~M6yTYQ*YVGV~u)736Xgk0{-k*LTHci-DfkB#z z_{?>+;Cz@%IF8`Ev0-49+HvLgLxJmL!7-9HaUB?AG!Isv;H;0SqzIRIAT1j}+QgL1 zAV=)ohU}t1Jz7{gV{t#ksOYM!&-B6^(ucEr3kUT7bhSO|Bin&VsmEQfcdrXs9-AzE zy_VSfq{e5fpdg?M{RLohWu?Hs1I5Fkpjj|et90VfY29h27IrQCM1gKoTa(=4?Y62= zmYD9>Lz^QfFN3f8g!n5x-d&c{_hDFbhvji$yZh`(fuxl*kJpL8d4FC@qTg%$@?~36 z;J60_Pt9@Tty(#@CwUmiR(2>_y3C&7LG*gBrB>~V`KT5)W{7^#JN^|h1S(6+HM5NX z(?Uc4k~bSP0s;^F2|m%g4u^{oO~~bLtJBK{KJDr2x(c&6iA~f>YXwzj<0g%$6>*~P zovSGn`+;X{ni4|CR@IL|9k zj8$Nd^JI%wp_2jDeiHfY8~6RgtSCEi#p$FY0sJDe5GfeO~kN_%4RsP2ZjXLXc_ zW3+QLYJKI7*m>`X35sYX9FH$w9&1S_)aLWb!ff7>u0izfm&aiv6-Q)8Y|u$YiHK~* z{o6SzjrpUW-oATBR8aI0XP9 z6nkH$38LS!YlJJd*enOw&^ncIm^ZkQZTLP1NKUKUNU>G)sQ(|DLkfhdZbC#o2rCv9 zd=5WBmurD;G4}}9QlQSR8y9PQ^}LT97$9>lIs5Gm+k?A!m=*w_b|wm+z-ayPa}pid z^IVF7B|`{VJ7n7~N>`}|4(rAj#nSDSmU<7()HHQOS1e}B?~{Hc6Hg`b{2WD(^y{AB z+u-#Y^|1qTq7Jk1jUl@LKsfLBdJ{;&M(u!Mph z+DabTXN$xw{C0$r{d#7LG%2%cMC)5diI5{GcbA8K7?XGI48Vz67Y`Cec2OZs_w!mN zjl%K0(M2|>k)7W>R^|lRT`JR-t5&(*v)PN~j`$gOCAT0mCE(4sKo%zr zR{(Ud%3`fgRf?NjhU>G4o0;JNv#V=L4IOcM6Jw~=5YLwWn|=BLE$t|mX`Rvh;{0C| zohOi1`6cKFce2?xJ;Y790h^lRAB2pe=X{t6QPyC(ALiWPdVOtIDwA{a_a=6dT6DX?I zX=xZpQG)L^k#^*nV`T3Z@%6k@SrvkTTH^a2*0uh&&)Z`$W5a_Y`69I*9+Kh#FZ61H zFN)Rn1*-W5-~&Up{pmlTIAqgLoZT$KPp=~6M)b|?=kv5G7GMtb0I5jY8@G?$lv-kG z^R&Bq(T~^VnlzP=NW2Il>w~FApSgQ_C@BpT(}?}L0Mxn2?g1Jo>!&#x_{e-MPPx_K zXTOw*4wCzJsN_>M2sYDRo-2Y}x&?3Q88XBTcSZeZ`C3%;o%U>y_n$6T`9=H>#zVio zb?cn|k30x2zq<~}nRrVVx>#xsSb@;Y#3#xg-b1o$5P3k^_p~Y!9(EmNjcGu_#cAC8 zZa>Dd1_i`+!5;@I1m6m|*1ySD-oV-*)EW?UFYvib!#-XIa7OvFrJiGKuf1&3!@TeBERnV}p-}x{K0+{EVnk<${a9a&4T7Gg1*~34hh5BRm(|r|IBqLn zH-N``q;T8hJ5x-Ef{Sa-^*kD{y}K0BG$mBr&JQyd!NBqbc}W6+JcJu%Md@vjP0V%$ zerH0%e^C|iWcqf*Vr;4F=z~np{k-dg$hQ{m^mAR5a5zUBXMfdBvhl6M-fK5TeYy}BzU zs_G^3xy3^g^MA*DJu*BzuxHyqVe~B7f&bNOOeKHvR(r+hNB_rw5s2>ozC6b{*X08e zoGPk@JUv;=__Ut|Y;x}Z8y2A@-3iM%s(d1RiUMw&dxJzb4gC@-mh1~KLjH4pXdn4Q z51LDxwK#uIF|d|i-2ywC3#Hpj2rVDl)U6wiq4oLSVt z9-jGNFjPnrxEDdvC-bvx-)-slIluXNOwg}6*Jl%D1ks$kIbrB>VmttJ+`fu#J%X19u40vrk#(Ocm?&C z6>8XP2iL7Tfp1%rv~%iO`w6?$E-k)zXb+X&FTk_jWlN$PN)9(8H@nMgy;Voh5nY1M zv1sb`>xF!fMQ-Egr!H?RKD;4HqZpy5Y>kjowXr<5!F(@wh3dqC3O*L2TstwT9bN-d zP+~?}FdjOxjvv?M>hn+tQF`>Z;QXa`!`0E(eWAk(&1z=OC=kPKLgctioumg}ChwHY zUTpSc(vP>Ay5NE2Vf4b1QQ%jn7st~-t8OjdsU3|rw%CggZq7PwaOt2BZBf&fY4f9@ zY>k7>0<&pi{khI_*O;Og1AlVRORhJ*-$vuo?Z8|g&A-O~jHiKB;})PxDe_*>I-+8Q>gGH_kF7Dm) z37YxE%K9Ex^wM|>McQ_Jr&F}`&hjLO!1CjTd8*4*u%YiI8x5p?5;I<^>og`&F%8Pd zY`V zNXCBcZ@Gel5ZQgBF}N{sB-e?M&$f2~9qwoP1=UWUT{u5Upwq)8P>I z&Dgk2R8n=jn)s(rh?n1b=4;2SxwIbqG-Q)RuC*m|+DxVSXTNZ=DLFGyLZixyQh&x9 zJTugV6dx{}>N{J>7G1DO=ZR_~ZBp=pIvr=0%hb2eM^)k6&vs?azr3gh7hKbPi14>j zZLf6=TOvT}7mm2{<=O3Tc=uF!eRrsyZy$E9-HdYs-THCl8_k+2nJaIZFA62TO9VAG zAMMV$*+jV=5dq^$G*@5OE$RYU%Z&+cbenxeUib!YGAe9FCpS)Weg7Lz8Ij0=h3aFI zB2t}kMmc1r{OLx2vsO?zA$N`%>~ctaRzn9t@hd&4`VRkOs4wRZl{;QB6oYVMQ%ROV za|LSt=WfkMV`L1MBFGF6wjNv+%XfN!8} z6IH>4$?Lyte=sEundZ~u-#_^Ogg8bML6W|(diIjX&yUNVA7AJRh$|d5Hk=WLk4(RQ z>F!dK{ioBxe~vgL(byi92=3A@*#*hI=2#^I2~QKbc5Eh|Ez^F+Vu{*P?_k6%}t}5J@+luh7lR>3H`OT6Dy0E(HjF*VkG_BPNQjZVR5T*v3`)w79{f zDQI&Ihg#4#HWoMFfTwNJ{f-QB?NpizK0BiP)Pg5l8XD&{LnZX~9m`olBM=@F6(L;u z`e(TsgA^p+4|G1k&HC;H6-^=vjWU8Q>FDQ5=Idh1D%`2J z(1Cy`!tYUV|642@gA8o`sJrKOd-BIp-{iBkKeTx&mpYn}7jimuBn8mn&CeYFdO{&g zh;hL8+oEs{1EBM`H!*o4hhMo5u1&Kh3u@5E`8!*N8DzB{3dFm8{{%L@%;*eab<1x4 zJ@#(+$Kt=E0BAV7y@DrjJjds(ia~RstMp6g2*MaBsvnLAHF;OZH~?Vp@AIt`Kb<{M zm8`O-oNN2oL1w7$Q~rpz)cI5&-RAwT$Lcj=+ruVuHaVB{hqjmx=htJH0qi84@ zDpNbZYe*k+E8uW8V`juZDK*|B7Dg_iz2Dv>b2``wFX-~@#;u&?kNAngVGiql4azhv zR;_7n&*4J2`H53+(NK z&1iEHirV(u(r;ycb^zA226Gv!XfsGqgF*BKo5j0Nd>Q+g>E#q zZVLj=JXKmP&`N{bO0XbZz@w`Qk8SIk+P>!4CM2Q3zps1$7(J*C2}WyLo}dE*;>`Ab zi$(M6GI4JyV@SyJ6NUKQdj^FCq^Ul#=o873k!SLKssbV{*S{o4urv>?W1@I5@c)AF z@W2CTzVAiaho}uJt=qRuz=^a@?vBxJ2*K7Nc`tT`h?^PEW42XJ)!2u;{ zk$yj@uM2m|b2}sJz@lIzlM{kvIybqovo>fz!COELaFMSdde0KI!|CtKeY^8~ZEPa6 zQ$NarbHQ~>MjEmgOZE(+H5&K$r*w6-m#VedD4!5@jw{ZloqV@|ruR*i8px41Z0FPL z@RAw9FDE@K$*KFwE>#*Ai)K!5tRg`CsX_nQ-({n2KR_x>*)0*uU1znCbZTwflR?7yybI*RvxJD@G1 zMirl0sX=q6Hu$^!f-7Z0+E49+<_m=?OJP`vpT384QJ^1%|86NI?dv5LO9-el=W)}g zxpUljF0jCh^`rK(?T=_=EVikXZspS@c9EG-lu-fn&rhzTuO4-*2Gb~1v3tS(j3oGW zW}uDX@*2DQgUjYdXgu`Bx7fB&2qRW+tFrjm?2GBsC0owd%KVR``}}`V$QLGev~eh& z{+E2;WZpYaMl9+Yci1!$seRg>mK+xkrpIcK4V2jACAwFr|s9TU-|!jr-!I$iiJ{< zNeDg;k%eu(N2Ff(%L^F(f5#)5kUlKq9rs2ZIj6V{QgVk?|5F!|KNuJp>jp~6Ct@?% z4BXk@zyB?N@z{dnpKU5oFk!T(KdQLl$r;XpYr&kWFKiV5BVc6wW4ODw+?F=Jmweni zWk%?@p3377prmKOFLR_k^kfPi7&NvXe^}bK=^2>P0EGav6;hvRe&M)EGNW;W-uJR+ z*qD{8z3FpEb$Uja(D}9Ie7pc>%PxX~XFzt)bliFW;2}-C9`=EVttk8Qw$(4G9n+e|%tcdl9X(Wi>%QQFm=i z5ijX+DTy1ZYj_l`PRlcYG!oH>ayRvA9H%A!ZK`anf=}8O53~r5_HOPcKa~J#?cVU& zHCR(C(~Yc_(WF3-XQpfUhsxHsrG?}7BATmm4K#Y^)n#B z_?*4iHTQjJg&Z-Q#c?A}-ElNPajLJhEijVsSkWq7jZkFU(K+&o_PJV;>bN%oqBUF-&K~<4fU`qTvPiUuHH~G>44l~kzV~91>A#{A@2(|;~<<5 z(p(yEqK_#9+s@)2V$JK7kcA-3~}pM8#^%G|iv5nzrJdxQ$B z9LmVQ0H1tkzC@BKW`oh1`SwXEBYwnH(u%Cel8`xOq3!qQS!$+1xt7oW>per{`#+~@B0EqDJcOJkWMM-?k?$)?rx+zl`cUE>5%RY=?3YLknZkoxa;`-e4poa z|ABk`MZ|N?UNP64V~#QQJ}9nTOJuy7$23LtV44Tq%@}i^w5o*O?sO3Qb&hrsM+w7@ z9rb~`;#h1E0IzO(c}KNMSrQV}w`AJFlfrn^>;~n*dPU{w7#4a0a61F>H%B_ z0`!Mh!~s4MZa;my?&wrrP!o(n3tbDRk=~Ge-nO?3Bv>5wRL|^$*6m2Kf_HgaZg0^2 zTGa8CO8}0g zz2gp}I@xR19I`AHT82!FJ=6SGQ1*Bm1DR$1nAl@TmvzU;P&gdFgM@WoP%~uGdMgiS zmaSo8<1lxBvuObV6*>CC87#cAuf;~tlrk^y7@U6U$t+{B()AWnuh~M`qmg!iP8)MK zCj^U@OBK*qUg|^*!=tn)VX=NYNT($8K@)rtI7{LGSEh7qfB>xmSphHt=7M6bjF&rs zIFqWW;95a;MYxkbEu$|!pfHnb+rhi&Z;xk0Gk5l)J}xvR;EXvCC(!@z<&j>m(%Vfw zW1B_aS9X6cvls?%ECnAS0tS^iVv4vEh$S~BE?)*!*G0Kz^4BxTFSWWPE9S7vEO-~;PQ5COc3B&VK&-HGM*$O&exa@{@oe`^?KpNxU8im(=NQSJ^B}4uM zobwC2B80nSd#12*#pLz^gO=HGU)7l=Mugb+`Ra^%h8HKyJL{9EA7Ev!X$7wxc3lB% zO4RT)3U<3khvoT6y0vV4Sz}8eo0SUqnl8@*LI389RnwUol?+^?Cw_$38|hr6oxon+ z{r0?acv(9BQyTu*9$J~S!8c<_rO10IkFHaOhWil!up%p!6VF*2-@^Fu-O{6jZ>0$W znbMJX3>vb!aRcaMNSYqF*}1;T?%ekz+QRabZ*%FO)4tWJ6I<+acI><{);MONkr7fq z$x9a2y^W(_G72T__hc=gEe7-Gu>3}gQoKPxauJw2SEeZCel2RcWZ%QlT zE6p#Hf>{+-r((_%pRz0NUjl$~S5gjk1!ppT30x9{nDntVOrt;N6!>Y|8ip(wC#Hlm zppo$w#0?j!V^EY*gd+lEXoLTI`L?Pjr7V18u-vlQCfZK7)ktQUN-Je1hLNIrIFGsk z7@)bO8`~WN_DFV~Fv;)O*|`Ucu{Zk(3LUY_?;I?y(NXZm@Ii$|ZX0C+rx{&U3>a

d zItK=SiEQ=z=v3+Dns;_jz;gKD#qmN0b?tFnRb|NQjed7+5h(sdNMn+Yve#MyxqzxL zyoK5`ES*i?fZSjia*mliTq z>9X1$1Hqko<&L8EXSJPX5Gy6`!0F?c@%s7QlOdHqz@*VsJ!rZi28M@J6L}BdGvmMj z3P475gY~YosAH5cdP0F@dNETlJw>@fyaluQweoAm)qU>tj&Z8^H)C&0uJ=<>A_;c~ zJ^eo7{x7|+(UAS$F~TSs`W$ut%LJg&;GkiPo%(zD;yk>6gL$jGFOB>!3gl(M=eOs# zTbLd{wEw0TS_Qcs#K&H17FD`nQ;6ZwmyhZQ>*=}`tnM6oPaarabwlF%>tzpQbi@1v zGD2c6RMmv2cy^cyFqAc+byvaZ9aK;H-&k&GSpRPZ1#e?N$ zE*MORJ%`2$cpNKf&76SYsZ3@qfK~K>PJ12DsNhnT^|{PoQNfnE>)iaBiPXNZd!`Lk z%EaeC8V|RAZ$h~%am%_&A*5%z$$;Z?FZ%26ilzw8S+Jw|D?8&Kjs<`xmjp^mOjxB_ z5UiS>;lA!HWQc+uS2)HJcQiIF#qS_zx^zcaR}~JD@HOpvQ1{N@ZWlRJiqjCbGvL+w z-5iO_bpShMV93fN$X1jCQptD#&li$cBiUw|-|v?HR(DrOx5f(VZX!5~bY>oT4Dh2k zy;N-GQFjocQfb=aS5vogJ$*j{rJmM-4v2BwUh!Hi%=qQ zj3ld;!m%eekk9HzToNWg9SRZphOqq|V|=IS)!*u&U(&70O8*Am zwU7@(oK9vE^d_$OSwx*2E|(vQU;Gr_~`>#qkhBS3k#{(I4U^iS4@?H@SNPE}ehLX~<9}6<&(fL6xf4mkFB%W5Gr=m_2E{Lu}pTm7F} zCr8n(7&38W=A81SGs4N?WFGwJ^v5vf!4N67 zOywM(<|P)N#w~O*?X!snNjP1-130UswBPO|FxBYO5q(;&_&ijAVMGjBS_|Y^ za4Sc?mt{XK{b0fZ)Pn`U_r9vFP9OG`@$sw-!ta3Vm5dPfP!u} z4drzuUitzwnuJ?V8Dj&LUrS{0O@_zc(@NalPR)CS;$IOgAsuqio3<`Z;Ena43+T;K z!VQTSt(9{%Oo{A?kU=q36Mbdv^ZlTmrdFkIO^vz}KD~fp9`#9lolT!tWl2|O4|6OI}6DiALmb@BR zztX&-&{=+{2z8sD)e7Z|v>&iSk_eCykxUScAr(%Qeg$D@7<7#W=(mnP)^W1a+H;^r zUJ_|%E-e}N{x)ebC50qZ)ZeZVJer7+$TQkMwDJ3mT1*?n7q?J5LZf&P+FV9%>gvKL z85y-FeKdu}bW(YokWLXkGxgV z9sD*-Dy}bqa}L^XG49fPK=%~d+M}2GmzXp^DN)IMQR?shyE0}L4#kqGQ z3S8zfFPJK&XB`~Ia-qRl5xB~_?$Kp~l)+SR{z~Tz+!;x(X(J_(17$*=i7{cJMBG}OTH)R) z4fXkOg_Naop$dd+*>yP5J`|MbA%Uq;gDLZW>!}7l8Rg%4YT@^bwwM-*rsY}3hP)CIh&rxXll@vkRB^;Onw>*F;=YHUN^_e=Sct%Z@cIj{ zoY*iL05UtgX3FUYO^|KKVw|c#*!62EQWZmrL$cCC39_|U8aVxp&WQGOnHtOX;#^{q!4)`?Chs%h|8G6AvCw)_`y<{su<2MOC{M#S zc3$dnnZ7^^A&i;Zj0^c(wjx(sBX~#i3bJD|_r_n&)a`bCFWA8j+i2EftWWb~gMCjD zd_EsnH|Xk>(`eozfn=Q9G08b%AAau(HY~Kl;d~tOuC>)?e@BGQ`V|dK+0vmkIw_J6cp`-+P{ssjYD7dvvPlk}0(E@2ipNq<6 zkcttqVvOl^=;lV*IkvYO)R0}HFBMMXd-0*_wb5_8mR#0?m%NTR5$0uq$23$HH3f{E zGtl7#lZ0rf9cxnvudC;!eMQKqQTKqRgZlirhpReU5QKL^ijQP+0FMWVszCatW-7Yg zo(SyR{1Hd6Rm=skdZ55;d}{{7oogIkWgbO~ZOM~SRiv_mj7I^VRlCTa2i!FQs-BFt zB31O4?`$DBXZm)#$a@>?nXy}-n1Jhld9E=?on;n!0@r^D`45cZhPW3lXP~}pB7DdL zyba}Buz?65`M0|GDroS^cE=ZoHHZB`hg?0f8Dy9VtjT_BA@yHN6X=uu|D`7_aAmZ} z9OiDtR8JWRMQjH)0w+qw-hv>}O_t2qUT|sf4k~gv;l=Uw@so?BhzeAn&3~yWCR1Jy zmWR|5X%7=DHl9IASn0V}#}YSG+w&8zU0yB}=!Ur89fOdovvopwRnzHSFQn7#t!Fey zdJLI2g8NW=!>?*|Z8UMje^civP!4RDOcp`uDXupOXgtRrG1MDFcc*s=1P_aF^z+F? z1cNxyU)gO#GsM5^_p=~h%{*G$6myTx2r=$>%zPGwZFs)_9-Jw^1WGwbKrIG`IH9!a zYZO%|Sfbu;SpNj4twS__cRvKhTD7<;BPf>k(Jg#zlYqiYq zD_T&_XE?=*I&%p9bw9%Ks_%)`QGq=t0o2MN7m@l?xm8L}G#-bV;r`(^rsC%1mGedL zj3Yu&ky}S{(|mO2ExzhZxI<1hXY&tg+m^)xK8IDman)ab(H%RcqHpV$uF^yA^`~6* zzui9tbWUk&oPS~!*|cq;BiLHoPwUnn zWL6(~obM%8I1&qx{9XYPcdC0vT6P5|fnSUJu;-vp%QCG=OAxzXdW>4_<;cea&-~oh zf{EjykpZN?C+@NRs{K1#-6z;x@OTYyW4{yHE#w9~F}RF?+0di0y|K6;i0dZC5dRcs=1GSGkqa=_*UKQn1Iy0oap7MM zGR%f!!ORFtgkK{yUm!PyUgP^MLkZ&4_6mADue4W5g8SJzvcMbPNHGHd0lB6yRbBVW z^K8@kuY&!<)Ae#H-425K4)=cgJ=>uqzVYg_9I zG?Vj^zK7F>{VagXLJYKlsZ?0!szN${d}L;ksuCS=Oplg`mhj$UWfd6lHoHw7^~lIB zpLSY4{}ZX|C|LHDZRu#I%;dDqyz(-DpJ2(=pG@#!V2FxJW+m{oNGSfwRJ4(1B~P7d zG_zodpnmIHx{ibHoEUjT(!iUDoJ2NP?`D0PXt5@jhN|$5SlnO5I(WLhc#S47TUcf} zR3`s@itEVGBphS?(R-UA3LM?j%Y@h9?XPRaZNpY=Wbr8TC}sWJo8iQ2c(G-icZx56 z6{g}Z&Mb6B9$g_zyKJI!ffujAdw3rSZ7t5WO>^k(nwl}pk2cWJGYaw~-XANDkN%l< z_cA*YpZ0skFMsBYS|L%NF2N!Dac7c$k4(exYDZ|z|9INRj`_nL=*N>ZAHZMM5S^gy_CpXW6#TU%NZikB zy{`GBkqK9g5xZF?f=Jq}fAbY|2BzgAH4@=O9Pf1xCApJ?h1DDa?uk&N^Rr6OuPId) zks#B!)2x5w3`XFZO3G}SYv#RQANv!2d}Sgz#=~8hg8@G4$>^?YeO!J2)?*%tLEVPb z-k(vP=<4tLlD&oH?nyXaDs80HUu7Rc;H#X5p#^-7w2%B8czhncrW$qiOSgZ&YODHW z3MZ7^vJa+>=6c1!D;Q54(-;|Z86O)s_y^a?V zQIQU~r`0`lciI04xrm4vD%EMivZT!KtuBcf!)=p63OQ=9TizGlA4aIQm0WXQfhVI0 zsRZy-Sb&j>@vd%86Vl2kM|E#0>nVMW&$Vp1Pam9!NiFLr_fvpec2GA61XbIZ?tAgH z{8dNOcsS9TWnzasO1fKnG}3BtcfDb7p5ox<01eFmjTBhJq%s2QptpaBkye-T^QGG% zA?a#%mZ-O}p>H%k7(!iMQ0A18`XIx1av)q(bh*G^7!6k%8!P?a^tPJ0(zqGr6NExt zM?pA^Q2;L*xVzp#6El_ZXHgb1s|;%@{{b#j4d;((EcEwaNwNq%_Y1ZE5~mIJrWJ}2 zJ*PEMUH;6-r$@$jV#B5lQ=r*u;>X>QW?awzcs42GhotpyDa9g8D7Pj`L)fE^q>+MO zi!2ySBhxkF~^U*S?krPH{yl57T|xo?-`u@c{OXGC>I|wnMaaF46^R7X6~Oc zq?(Rb^M9WoM2}E#>x!yQJsX-@N`H>#XBBD2AzC5-viA;hWGue>ml%ONX}D#W#PVI( z_@Yjpzls%4=?xLh|DZ-Ei?tIfBC_Q!D_Z_}k6C4>b0XO@}3-rhAL#Im$zDPXJ){OG85!P&5a;jwSHCkR7;4t`Opy;WK6+z zn0h2aotH<6dWC_RRXls)`wDH_b+4-skGenJq3Qrj_wt!L>VtNer~)lf z%DT2W#qQLd=WjPLbE>w5z}NKXbvExz8^bA&?TQa5!Pq~DYnsN!;Ousaw#$x*P&8|V>bc4}V8>xvfG-Cx74zoN;f;%fbUz%JvF-cHl$61bOu#xQfloYDLJ1hQOq**BE>R2MKwch>x<%oO%-FCl>B{3}#j^ZD_Jwtx7KiU@z zmRsv1;$m?@IZEU*$An5*euO+Q|NcX!s{T7F_6+dwudq_h;9-MA zoZAW%fy$b?Z|z~O9kYMFfbo0|si%FR*;C@<8v(ZoPcxw8^lps}tga&0+#AA&mzt)u8Z{0qP0rKkYWzy`{*UTA{MgpUu7vOdQ$Kw zjr^CP;z1mEKMlC?`ONb+qu0f3vZ~`NNb4bK_cIDU&=3KIxu+oFN5Hv* z=ZG7suc;$3xw40jV|H(c)T%a=azf=;ME8H>lrN`PTmQ3E49Qn)4vSx#s0upf=}YO|PcEBg zPfva2#d9-vNM)+*0;?i|bKRs9dJzUE50ssVN#RO-eCXDR5VvFH%x7;zX)r!CLMpU4 zrxS}4gX4h;Gzdnh=7*ivFy7$dB3WJT*R!hxVZ-KEf@mk3+8r0?aUHL9)5v$hQHc5R z?USsUY%*u?X9K%dA3N-%um&i16nj%}3TLYY!gq+*PY4S;{f3~BQg@i*%3S%*X9EaO zqp6<8G|wg?K*8?;;6=iX0?Z;ZgPlOaZI|?&ZRAQ)JzHZmn4sYxB{*$aa0l>}HOwlb zx+?p4i|T217Py9si_$Pu-EXsDb-6C4*sz*k_XFI=du6Uy9i3MCqFj(UlH&Sax?L>f zY_R-Rmgr9eXrPqj79umGXIH+(^&2mG*eZR7E5nE-PkWsuepMDNqF-v>QEZ5U#QZ?g z=;@zX+Zj&%vMdv6vf5oQ1J&P`^{*|A{18*1(V}hcmAa-{It(aU{Cd0rsMF?hpKs zfarU7V1_)Ba56e9LvBxx*>K4fQrFG4FThXOgfAG-*y*_b)$WMgY=#;vesiM>HPoPa zneB8%dS+oj;`y30_exnczzLvQQ?N28fby+luDT9#ah-2PO7n#XU$gG%Sv(ky3ohYK z=Z+9IQs53-x;{bsL&cNav;66|&5E@DBEb;={ggW{4oiKn1R@h1O?X-{iElr?$qkh$ z%jz;=;Ym&8QkW>`#Xc#OD!EoR&&K|o^1z!wui~yE`yqAS%k)X90`6mtwv#@R>C{9y z7k+H-?EMT5f}~hTpM(26k~6k6wCLML@{d8Uv-T6&=5+l@tiJzSCRtgz8~&ElJuaWx z^yNRmFH;+v8RklBc1=C$fgS0$oreV#jLc{){^%p@`02y8e*tMg$;y{|~-)s87v?ZX|`%JLMg7y7^n z5J`~zCXgU9xf%kSZPJ?lzAMOWzf1eLD|X*QWt+iC zkBK#HVoW9!zO>8T7%=I%2($q}4leS%fzr^K-3y!dAhi*L&GZ{w<=d#>K1gdiQzeDk z%WE=i|E}%p>h>op9Cl+ws8%vnkjq)*e_&;`j2F4Y>C*Mh{()v9-^6SDMG<(T>>V^-Kr2Jtyj@SL48?68`JXXPyHu7B_ z?tjp*+bt*j+bkIu81k}TP4uzgoO$zM?k;hr>VBl3g04F)2nQ)J4-V5+jrC_cZzn2H z@s7NmVWc&S9_t5X7~w2xG+RL$8%e*)={232E88*rUc`Oq}?apsUd@eoiRI@5+Z~BH!48PZ)nm`h=OG z;{QQg3IOyb!4&_bnuOPc8sH!2U3cqa=cC7;L4r$RuT({MPRsb0sIfrn4fZs!{A&(L z@!wx9l)?b5w%3Sv0K_F(Ar``0m(PGhJBU;sn;a;v=Sf=Tle)L4pkJC#QoNCD)wFeC zx?f-woK&M`9qWu6)-X(T25oQfM`=Nof$uD1R@YWDzqS)y#`+}-hEu=nkm>vY;zJ)A ziN!#se2bWNXq}>-=8N_X<`!~cDxPLsQoSL>)1ul#2&TnRihvBT6#eyqs8Gc#0PDru zp1VM=}XU3(s<_=))s9QijAa1eQ@J081c+syku1jn-cv~b^q6GQ8%PH zyxnM)fA&}xG=sWgC@tP|R_3n1TV}qFSA>)$kKG_ZLEvy$l9aU&GFLOLXK6_QP(<}k zUOA7J3iQ$=L}+ol17V99k^pd6>;LUk&R{tB*jL{)5X9*!(7#)N8Fk1bZl!}&d5-~J z6`B@HFRTpzhOI}xl{Kg!K-W#C%lxqi(&c5zeEXM|K(Z+;D&b&IP;Q)?YM@he`xzJO zH3~Xevjb8JF)!X9_2qN(MST^u>P&v(iE!0HpmB2oOA1i^X-D?{ugfw+_oaj2-FOq- zj$3s31ODN#&_9WZ$xIGQI8f+WAOGI=3p1#zF_U-KEBE~d`2{qVzgC~4F8<7J%?UD)`HV`tHNp7~V{M0cP z28+w%K$0v%-$ukYILAgH|9@TpaPf_tEsRQm0j{517JDSYfj1m~Q$lLpf1mj}} z?^!&qXGt(Qwm|o^tE##m7CJ!6ts4-L4Zd#Dsb?`a1uie*zm=&gpOXQ+i{4^mO~)gR z4DD_D*H^k z@7{w3*x||<0>6+pTt@zTLpW8ZhH~q7l z(B0x$n_9VKm7gzStN0+n4LU7>S?3=`H2l{3 z#I(Zyv27MHcFXfUvxzmmX@!RLb-fXO+ab4AwhQ^3+P*BglPPbg(tEbSF#(J(scxR|nH~+9#Jr z9Zm6%vUpTi>6E7lDh&FTQFefu`fVBDU>ET8CF}8hc|q^MgbK{K2%rMmDAU=cA{zl^ zBLfn+%XUmls+6j+spUZ6cUwS5&SJz5du33)&QK8n^kMKF_t{j_`pJ=TNZAF;aS4CV z{_|t}XdUJGLKf7W$ACfg$iK#%vf;kJzr@26V@jPy8?>$kWz(EIj}w!CTg_e&#-mC_HU_qK)y_fxTo!M2GE$TpTpv#5IB zVH(6Q6lrJU@a#0yj&`P95Xb>Q0D)_{_KE_R7I~O{2$I=^l@VNaP1^~@eI2N)Gx&Z3 zh&I$z?H6!gwqPH;y&SRHvK?H#;j53N_mc3=N~`+n`iPAFz1Y1oUF$dF(;yAaWJja< zw41`X-qH9s*AMMa7NSkja>pS!7!IJ;gKbuPXY9KzX(J%`=$+oTeJ91l%lt34yT^6= z1vwd+pIWSTgDEd_FtS>dw=F|w)1ZU^$}%Xect=VTv@q#UDH!bHM(c0B^sd7uVCEeK z2dvy`&xh-@D#agOsi9n`?ah9{cnIi_rp+)G9@1rlj`sAsl=rzS*BuQ%Xc0<%M?gLP zTgT$huHe&kAt8dpFkOYnMBN+|pG#3QF-4YrVJ45kQSR-gZK!qiB^?RqiygWR8?Pd(3#mo|v zKx^+uK%orjzX=At3*hOd+v;FHgO$X~_jh~iu`Av?AflbysmRXRLk!aGS?@6v;If;m zIi5zZmncMC_M#q>P#GHNp0XD--u_X1W+N4w`8kd^lGc2$r+ui&+A{Uid=dMTRo$TD z(TKu7BI-MlURHI5sw?Z&UR4D%%4ZAPSK{KYxS_4T)&$yg&Je+%AdLbzo2dC)Aud?u zhZ@YvRl<@f=cVfg;P(?~iz8be{PR%O?xU}q*!aq(qQx+nOv8S`6_2ae_OAngHqj_E9_;|A_BUciSi(v6sR z#MF`-Jw@Nz3ver^QWRg8mKSsJujLPEzu08CP4~v8mKi!{zI?TGh}!Ot{VwGa2mJUc zHI7=$%f#&NnH(|eXYWq%u)#4IO&YXM(do=Jmq==S(J2Ip8NIFz1o1_@WEq;}Js(UgJbV-f^ z+~vdI*-qVG07Lj1Y(8?8G>F+sH4R5);(TkpHb8*jo>o_GLPd2v5)k{h?V}tN#pGga7q6C+ zFL5Ao45)pvzQ?arq_0KTwwx#JdN)7Qzh!gzM!qZ76nAHh^=3aCvXcyg^VcqHQhg)L z;8D%SqQ0ls{W6>v77ad1734d~ztsxcL0JY3A|PIk7YG(n`E4CX-{QnNL*+IiufGnf z%K_-MQ=di({{fBv3}34@Dzck^yOsAnP#WA9X9+)pGQsya!VH>Vd_n@!WIJCCKmPz7 zq?1ZcTkT4=3e9Nd-}8Df)PXw{|GJ^Ll!4h_1QodeIL;Nzs8iCZsjA`EzcqOqiuY0A=Kwwl;t5>d@DF+V0>CxE;NKJU z{^qr;fW!7eGdS0G&s(nLUg|A-Z{beN$uU9mLgTp~^)uO*o0Q4;8y$$2^8s<(^dIGR z6Sk5FeG`pK^5u2fN4d@MrLS#Jna;Z%J?Ns*r9;&XaT}Zh;)4sFC4=l;b}~hzdgNlc z_5u{Ia*Na8a>6Kw3FRJsl4tovlC9b!+dW?VYbAHe$N>ul7%JoAiGWBk(R0t|%B50Y zdC=uY3MrB72C1dD{~l(RMQl#b5oRW_G8p#PuOK+IMLh|ViFYtdi4p4Np^_vH~W1ITzRNB zm1p#_7$~S)A(z{O_5S@sEt1ElDamdkPp%boUq2a}f1G-#5R7ozW*i&MMqs6MU(5d} zZ6Fr%Ey=5LX?G0oQ*I?=_D=G@$WQ^jYE{H0K*TutmzGnFX7n#=Dk{GrB?2W+mBxMV z&x?b59z2(H!&yCHH6&1HK@DCH9D*nA(v#2AzQlE^Obv7TaQY~5@BY}zzQETEdH_@^ zw>&xm+Apwx_wd;J69ifU>rY;*#Xa2;lulC0;nu_w0A*Xm`dYLdguXDC%W_!YFweQr zupa2wFk&eFVaxPH==VPCMpO!Q$$WON0YF;axg-^KVn3fB8*Hd({H2M+(-}ZIgwlX? z)onhJMa4xP`hb0YHQJgqRUX+bE07?@>#|Q@Hhffhqpp5!jhPk;>5smKFF2-xVxm*5 zWwCg62qGEO68}6jqY>~pS1@JAE9U|iZ(GoUf-+NSAf8Cq){JK4N;QcmkI^JpIesr0 zQ^v91c5z;dW&7$7lt|4qi+crih6kh6o0&y~t78VBwR;7m$*2 zveGz9mgaIgc9QZ`zxe~tYBaZ^H4iw5oQ#oPkmjfg%Hax8`|Xo|0M&K2o;i8vX@C7*+vnRP!k zdkc`1r&$U`Hnj25J zJ9HW8+;c&H!;A$A0|j)40D9)W0idfl^0qMeSS9m9mq>Q=t7mU|&}X-o?V2N0&rzUh7@HJ^lf&&UM?eSMe7jqYwEv{= zgFQ*?iRdUyWz2mdyp-MrMc_86io8veJogYm)8Z_L!$$Cm3)v8B%#C1Z|wxbKV!06F6Ae+ORad@~vw zLBZ3`bUCrmImDHk$KB{zE(DTZ*StKG{d}rYWFpq*mg2=_f!~I|Z&LathgMVSyMXTd z#i{~ZI)SLRmADKPMoJ7lYXOky|JFG&zBP^S{Z!PU2h}Z(7c2$Rbo+u|Fb5a2`hxrD z!#&QiPx{!#<90~D$xI&2Q`d>E;!KPYqgz)v8aBc{&oQlxPSESLFaK7*D9P4?jV~Q= zzZ{lxxfa?ya@3z48p~c2PEEx!B$TE{SXJaZaLGB92%JeLL}lkG%7D!QbV=gV$<6*% zGclc}osJehBp%lokg3n$5Kt!W>G{^9#-x@SR!l#=I(+J-Op~FafYr`i|0;X z&J%@O$vSOS8NtCq1_dx0tRxlnaAKhhWo89oSs|A$K1%?C{cXCiID7gijsP^%)@KVr zV8fvEb;E*%@dAZZl~#)LlM)kPoHWlTIU)LGPD8e7l{4=p;Ld|*Y*G=H{rHq%-T@t4 z?!PStAQb4Vly`lAeD_59jml&=A_)Y&uqWoIH3If0;})gHi%f0)SiI7=KYkT&476DI zffy|>^iKiOoHABa{0dt@RtPH5QW@YXfF8NmsLC~}|Aw-|F0Xq>Wx#n8ND6r!5p;lH za{%!WSk5nI<&7l_kn>F53oq%{K=7~m$I;=L8d-I=C@7e2#m0_f;^C3|H5lk@n;EntFIzmI)+0&ts>Gcm7-($GB?>&XKq+qb!!2IXjEyQ1Ca0EtHvq;Ate+Uo zCyx6grS-qZ8Fb&rOP87f65e#0Rax_S4lO1sceTvX*4{pF^u!m^5+g03esnQKR2HQv zM~VM7nsha&kZUr>^=jHDqjODD!2SCBvanystO8MU-b}`p(F-WzGQr`-!!})~jO3pf zBF^Lh1jnhXu=2I{|N8VH6dP#mR>aS9(!N~Z!s2tO{~_c3?hWkEKn%%|-oHcEW%&Yp z5#+CD`o~YWz{(-%vOxF5Q}h5n#YJA{;bL|esAj7fDG8D*S_K;eBrxW$cJ-$Ts4H=8 zeBH32zd~~*n%#|Ud#wYK>?7of-!j#HWk*A zU*3#;unA??$I(LEnI?0^vbIDxlvMT(@JrS3=o{I=zs5$D;RV?xd>t%X>Md*HEo$P? z$;YYrZnu>*$c97H)zDiOZ<8 zV(oX727m^DS2|(82aY6yzuu95AHx83fhV^WhEEIBJ=V6zFCVn~LE*t)^jnWS1x3@b0NZrj`!>-IpWvuIQ~jSei7*l*>dx=W z?+28WmDDUtu6Xc>B!nuWFLJHEBrfJ#U49;rqlY4K+&_h1_Yk;s17~KY)Y8wKox|q6 zDfZwjXf8`U_V58j=Q5V)?tOV4AE%p2`Tce@k+F8kRkBD!6m_f;|nl)R6jd>28_eZ z!nD^z9wmS8*q11D@T9j?NkougjKKz&^!44BCspyfbZFE}dA54+0_a0v_7u)^r)Md^ zkPOPR!8$%~`K+&njBM50>HP!t^1yp!YL;y>?k&7=s29!6+uY)isEqceE z-XGKAU9%p+BPoQlOS=_vH^$We*zZtd_u4{)Tou)Q2ZxG=JhwE9M$lHy%r2L6tI?sF z+!}vDAEMN{zteyp{VT7^%HR$SxG^T@OxR~n?*h2We>#bXGZY{=0NE@80uboDil|p& zX?`6V9!jc8L4f`Sq;_(C>uS;{QTMF!ofmt&f}(yNLvwtLw`h6k|D%@_azYycdlY#6 zL@h!WC0ZOwoycs@nb{>bMzFL9AeUH-=;u_UiqYm{supa39$D_I;=X2 z5|xy7ZUQ{QlsNkNe`SaVw}5RD(Y1`b9YXSY5~BAci|V?y^B@PeTFt*>ao@@90^8YO z9iyUF-wVhy3x)v_%u^~zCnW})=KGjX5z7tOy!nkO3ffxQOxr!XsrT#=tZg~cV_HA+ zu+@Dkv9kCP3H(^8sU{#fd_*ii3-3VO&9vV0{{0+HQ*BY_^hiIJb}me*mW1Ai?klrl3+${fM*uDwlyUA^>z|FD4pU=AE~yJ%!Qm6q5cQX^Fy?dZ zPheLA#-Z;iTA$TQy8t+DY~ib85ex^ZIA`$-r1kx%g3nm>k>dIB0c?6IKS55u7A+xg zvXmYl!=wChy+_$IC74r6&88${P5{)bfVVS0ki;{+%b?EH19h*rha`vMNr{J5;<70L zE)$jUPsqcM&Fw2AKkQv%k4#dK-*Pm43%Y3%U`a@w^e!a$ln*%BFutcII1y)QJ=;ZQ zR(8?{MXX~1{7HkLow`sI=iG!-@IbrjY4^))4Zvz%oY$CkN{Y74+~lAG+bL|tdK(5V zdv}n;58~CPP-it;E;0-~-v3nI5|ZM?T1MMklH!}>dEgm<$h{J--7LU52N6fxwL1Jt zIFrH}ydSsZv7Wuu()ja1vGu;{!~%#HmqRM=Yh5qkn1Hijo%3T4z@jfu!kVg?EurY1 zuIT_(IfiNF;dGP_j0{pSW${vx!0*Gv%)VOIN5$;f`^}mLgFQ%Vy7aF+ke^mF#HC%Z zf}yDZJO_4|helOwf!kBQi{XNU+KS4Zv+_R;H*`>WkE2}cZib-kD0B!u1TrLaaM<|z z5z^Y=<~* z{tsNyXEiATE2%zEVeOsO*MRg21lnFJ9_t_q5Z^!mojsXD17^`d^jbBl++;gRYTzmsP5-fS*i43K$uue@nG`XB(k~Hz?Ht8z!-sLy1%vTE$Z&J=wd>X zaLaY5$_2U|umw_r+5pnMid*eie~UR_O_SK{P>j3wo>nb^+(Y)<5Iu9O-v#8#A#ca3 z%84@CqW8rcz)E;Ab+^w$trB&*^#@ZAu3)O1ebWRU)L#JdL7w|M-=u%r+>k>Z-(EWt zOQnoaCvIZzRS`lSYD<#z{_gsArRNPn1fvNsEKg%7cf)zg{L1nN_{FXkX9vAg`f z7y#-)eGVn#Za&Fa$9z^|5VnE^=Qr%mj0d>kvWm1u`&q?H=AZBZm)Q&S2jJiC9GUiy ze&P;y%OGTcwhl@ToSzyDk9dB+4UBaLiIhIIXS6=~;@gz*5HGY8SE@hgBY{n9dY%dR z-~+uf(xm~Xs=uo1@T}QIBe0;@ZmpzM<>@$ zw@Ua$vM%J;XTz1HN%NM!_!d?+GkbUJx4ycm1iCZ)dXENN2E}fwx93;Sz14shI(V`{ z2EIb{3bnm$%fX}3JbMP2!BHw-PPGwD&1CRph3%zXI+ayy@j|sRfT;uOJE({__2LA~ zD&q}YeSKl5?iX?X!Ox%K z7w1em_Y-LAF#O&lF_Fuq@gD+8N~pM1OeqR*x511Fb`szpCqvqe5BnP!li0xZf6wtG z0=Qq`P6J^N5@j!u3%I5jxM@VznFj3nBfQrDf+#Qh?|J)))YLLzbx%!nml@&i?*!l; z4J08Tm{3MKuUF3aGA{{Cj(+z|uLEv-$G$Ev$wginbjeRSz}NrCXX?_sYzbULt_PzV z;9i%8ODuJ@5&=uDX}fclBVgtm^v6ay{?cy@3_v{G+yMY)X$)7T_Ube$s%>Y&jyvDC zMp5qm<^EBcKEL7%y5VNm<30_y`nAn+;_vOXihy(IKH%xeAGPguiyP3mgplPRHi0G< zI_Db$yG*bN4t};MEQLiWlYxaD&<%|5ix9!5Y1@@I0K5haAQetkJ+;xw3DNVw;%kyx z;|bj8tm=1^(_T6Xgp+$;@Y^0vXir08Dr)Kswgf3zZPwbgrX~UZ1F}bd8KdtONFK~2 zW5t7QYV69oGxW2!f<^cZW2BbeD7sNH<7H zcQ;5#OE*Y&H}}l9gBQx{9`|Pvhd7gdV1<>+*KW(;u?$ZY%ydbd7 z*8(48S*mJ`fggIF?Ga|UvZpNIKlJR&qfw8|VDW2f<~+E#9n?}MC8t!}2D~(W+f)+g z2&kx#1$%;e=9^H^>3KdHGXR2(*$-%|&-!8Kz#K)`-^xB4z$<5Hn-DTwC~vy`WL}0N zEZ!%}+@}-X-~2BU0f@HWGQ&1Cn%;m5hpLH{@WvQ`dRSKLohJ~nmujhRV{?o<0%jQC z@VHRN0CGcLL*x}*X*#0o+oxPF2tane-scz~Ygx&;2YA`3Hv&CGG|>Ihm|P*FnK; zZ4|IAduH0=KYd29ms}0Ijn;kS#P#$0T};}wCbjQxJOG0~bfek5kTIFI?KraKK)A5*J5d3u`Qz?8I z9uR@2+PhUY2m&H;uF3LadcE6qW$w9;F`Qj$*|S0 zO*|iHk8`G}oSJDT0F~3Hv6N2Pf5Tumw0-uFRaH0cTf%uQYz+sMzv&ipL;Ft!VnB(V zJtT(nv>d*!M8pOvY>ws6r4=7JXX92q{PP&D5g3}y&F#nHpTYsh=vZAH+sWY6tEevL z{*kK*HNWA~p${l-08Y%!Ppr8$57W@>NWHOo@Oeo>u1x!Fd*LY_EN3_R>*dn(v)~}p z+sQt+lb#m8i%atcx^$o`1DIiP@icL?DIh`s1pg_zmqpwg84wRW!2G5qvIx{#Mxk=c)ar`c7PEDL$H-BRoD3~sX~A@0aNh9^z2a{txnHz;hE+ zzx+L_;QU^d&UPdN<0f*4?$46;+8$0V(DMRQpI1vfu@7HYjeS|EfL0!5xJIiT54m2UqCIx4v)_t~jE(qcfa zs_x%>y>$jnagba*J{GAMV6UJ!BEWI~%c_awmWq!5L1S+enzIBXB2xNlFDRCVJw3^R zz$!dE;16ecCv;q>brgq{opbGpUj@4XAMhLsYyx~^V%Ux|%ayI;b(Qe41a)UBTjl@eSBD4D3vR8g-jTQQ zD9IHrbO61dJrG6#N(@@Yg4_prU0{{eX#D|n)!c0ej6V=2Mkm$$r$2J4=5U_k`m_VT z?1NtdxDiv^^=A;3KvgD|Tl$)-G6ZubllEcE4LTuOD70;b{Wr(qztVN8X|e2Pg|LbI zV3oQMIXcko0%^7pkYWBZ`#G)oS1Cqyc)Q%18ejuJE(G``S7aSji@$uv{&JkNIQ8DO z#NQC3+Ft>{6HKf?tKP09Dw^evZx{OM%A*WVnQqgk zh>lp6c$!QRP&1yB!M1f^TCVs?4U~k(R1+{2)wKq_6WQ+tXXIh0QJm}WLDOUenylnT z0#l1?r<1-E9x%Vz@_51gzW&W>$_%y1H)2p*xSI50K_!OIp0ORxqZo-DPDcNMx+MR+ zBaMccxc^n!D?xku*nbb?)vaDjN$pfJ{&OECyhC2Bg1&pY`Cu4KBlQcmWX>FGcZ25r zwZ#-l`@u~DkV1kH+ONlx%H@r^yOt85@Fc|k%Kk;*gg#c=2;_gq#BvV4x_#VfloEFw zM*~9dcR(fs1lk>dzgT#EiUUMzfYc(Hb&LSO1$4v>jtUGYoRH0@(xZN{sn&BP)%~Ni zIiBqoc|=UOJ_J?O<`V#1fYig}77G9ufKgp5EwJqbYNghG@IcShm;;?TVYHE4$kk9@@KVUg!{QRW+wzn4a z{4fn(igvlDWI&M!Tr4J0Zz?}d2ndQn)(7gwsno2JY=A|UDhbQjSvYIe?u5?pv(IAz{+O7j15(8IXcFbh~UkJI!##i;zYuBxneudd2P)f<} z5t z#PHOByY)@OXbSaM@_i%1W;ivT^u1fCA&}8JlHc2<*xRyYS4Mg)4xKBRbb;nQZtbj8N z`nJfV>{ecw9G05?eP)Lua!S7%qR=!L_~VL!s}jjK*zBHTUgNp58S!M`XEji}w9bvwH27^qohb32M z9zkotdm!n7LPs9l<}uv zgt<6d4Ggfsq76K!6gM>i84#As8|3D88K1){d*b|$%7vy2#jTfQ1t8}^@j-}f+gF|X zKA@}s_{+?qo@(hfu)zC^3BwLnF)c@1 zrBBd#?rD~O+)HUN%5UyU;#+xl0{#IjC+m#{Feuw*pS=I)V8WGu?YCjFsF^>uegDec zUWxUPNkXwT$WLY>I$^69KmiG`G~Ud#et_{A_K*hb9`1iWSj!)ofr?lbvtI*?eEuDS z(QSXN!1}3w0(crcXgAPOIbO~zMNP*{LzD*#NdfdeRHh%Ydg>e+~V-Ja1( ztIakFUx`5HezXCc0QJ;{UBYZ>TF2WN1~88WidEpESZ!A^_V?M$Y$Ws8#t+({N(X@Q zkAP-UV$PlyAYcg2dHw(s;Qh_=ORbXU;nJFwcdsE}j1WxE> zVKA;O2U;(r@e$nZ;?OxKlpFR>iC|uGaT*siB!Cxw0lXk(DOI8ie&#gc~QLl2<$wwBNScM1=81x@*pU+0RE2{(Qc}CC5PLL ztEk^d1c1}vJ$3{0V{!cez~P6?IfVhd|Lh|SK>Wb8&OJ}#HOKe|A_hCFJBE~`l-0Gv zK>$Do$&$>jPMF$nfT8^LHBIhwaQ*-^t%I{P^3-xzKvKt&sB(L3w+BZ@n1BNVD5*^* z?5zrEZMmMFfmHqkWGM8b_M=0CttI-u;&b1w9)YW$Po)rFGnXA1%U%q>#Ci%dY*vgJu*zFr3?m$*eQx39ASw+}NEG)Bv z+PL0LR=dWW#i;kb@SVzY+?4#vQ*#MdV-bsUEOp0*J>o_;bd-cMYgy&J?^f>4z zr@f=JK9>d?o>K!W&OSkgix3G$nm5{l9BmlI(!kJ~7z4@pSPgA}V{aY}uEAt|dgZ~8 zbHf@zcua+fNofAqco{hT6k+FYg=Um#TF(sBZLS?+H9Ob}JWRu}0x>p8xBqmPr-R??X1zKZnRDZ%$-e`5|Q-$bK5 z2#6=Yp3~NU{94>Fw!ibYh=sW78}Ng!wl5;H>~;E$ z{W_-@%a&aenBB~3m!p@?E8s!K$O+2eBC6wHu8=SWZno&{9DJPl%zC_!*k$jk%!BbI z>|$6wY|T&WBg}dBIHCytOUpuz!~N0HSyt249JALf9%7UN=Xj1aC58%^>|3eS5VQ#_ zMasMa69#N;7m`^NU|d99IMNoG(XK|_nYBZFNap$V*l0FTqnLFdF#Z%8HAuUiBek(f zWnXiDB_$`>o4;a9)$vH#^VSsEow$xeY~@tfz-A#0T_sF7LVZqYs{U8WS?h-PZ^Y-j z;0@I}_)F&fob8aP1ZP+K2X-u_xOZW-7`g#Oy+?D?K1ST{AFl{?+8soCx(RD4ZKCA& z{`!~sVSLY^qZWLJ#VvnJ`I`>6YcHjObg4**tbAvThh)hWCYEv(Z?bsb-0phQxBU^B4`QBtCrC|Ki%KffPx!=($!1iOCrCpZDL13k>3T8&TXg4Xsq zGHB0x#ZtBF>A_?<9v;SNv&+$bPf}69LEQ`Jt}`elQUaPH_YBr2vcs0e{yNpLI4xXwN4xM#bI*dInF-})lePyLjP1apMR=IQ(__3Gw! zzd}AXtiRCs+#=P;=S$6jrJrAvfL-rj>HK#VdVe`MS?%Y>q~`89cHQ39jq-*he)+fI zk(?1oP0W(j6!nAZBr_#GDwmjr;y0a2rOs$9sjC09%2Ny6dd0m4>(K)aX6Pm-QOb1B zCYJ0PYYj3-(ot?t0AXC3!0HiBE{Cr4n~tDBu7h1w4A=l1`y0o0`3GG_WBmE2TPs z_giP4H|O-PFNi=eOZf|U$h`Vp{DgG{;O_;+eD6il?sic?j8$zijOTNo7sGJ_$r&$51#BXmo^Y9) zwJ3Ng@e1no#bV^&j2t#$SS@UY_|SG;@gm}Nn&%<}B{ev{2_IJx`tV4Wr$PY>p)#6S zb>qfIl>5+ky975ESN2UZkC*WMBQ~%XIGMq5cLtY=j|EQCh#tgKICNn=&wm|@`84#v zUc}cAa3I(iXh@@O1ab5Y+yBX?EDT%f_h-9r){c>Mz#Te%fAR9MDP?_l6wdXCpfdem z=e9`deaBigZJ3+S!!7Q{mVUtYlExykjF6et7ES3dLU1s+%L;2KO<1s~ljV*oA|&4* zTa+9HxTe_KA8)xLgsZQ%!rZ3o{;XB4B0XJV20#BrM4L#6QC=wj2p+l(wkI?`<$t8d z!Zd#Ae*6kzuwIoNMkc`(upsn|qpQR3PMmnP;wwfoG?!5C`^iu=xNq&G={$@gU2Fm2 z#!+e%dx1Q>4jtET0^}6Q9&B?TA9hLmCvT-ui2g6~fLe_9)%Km&?=O&h zW6=p>fQuIU7ZYyI!hy(L4N-o9OPE%OG{$L>6|M+JEG+%T*JsS9VHFE`w7F8Fl^v4W z_D|CPyWKcDwqIxH*h`h(vg0P>`8U!K-yJdhHVtna+aoNET52IIp`3Mo4`$fhtp>&y zYryGOckfAtKG&+k?`*!H1+I+S`Sb0M6_XbmR1J79F(HIg-7@q-h!EosHW9`InO%Pb zV7VWRUgeeaKn;tnOeDY+BBlLMExIY`U@+C80RHxOailZ(G#LnQ)O;LKk*csDT5ae* z+o?=Wbr9_L3AIO+oxxR_RZtb4I@iAnwa}URr zq(*s&xzuR-7?H8CR4vaa6PpnB zzOe464B6&|rPgGcP>~Igb)+nCs3DKRbhJOMK_K89SpzOT1pZ1^0>Q*)$~&+H$Q@jY~p zPwS=Hj7qZER90P1>#rs)gU%y*v@un?GsDDiGmR3oeQnlS&I>2b*o<2g;b{E|5L@ER zswiBpuZlBKNe?Y64ozSou7rbEEXZc%Vidfb#6Kc0?C%={@MUZw%Rkh29d2BO=~(my z%}c@X38~F#EKlxFuRLH?7jnR>PRY%b7Y+1LB0!J!LpgJ9<6y|>X~2ajXq#S`y8p=O zH>$?iveYk+<8Kk98?8tiWC3|a`u>X%xZMzpND)q1bhOMR@^f8Ug65QxCgU25JHZ2O zZ|v*VCsmo}M}BU2cX(=v6gaS%Z=P;3LA2Mfn50lC*R*>p53;t5p1?;NifD!(AxRt% zp^dvHOR}1hu6d=aSK*=Cn7lNU(o8QPQ)M(N{QCOpSIJndzm1F&AkP$Yn0noJoGK_E z_s!M2E4q?~Y+(jtuC|`{0ac+mCUM`Pti+SAZlfhWQ&g{Lo;&hY-Zo&dGbl z1~1B&0uqj^e(uuMQCnzZUKFpt&p?*pgU`CN6z8NyJ$uF9bUK4>l@rNEL%O8HL0|ff z0UUM$EU;|h*pbI$cn{BC{UOouPY{#FRt%62u4+N1$PU-l=Au%hQEx8GIJl=$swHx) zI97&K3B@&eI)DPqHG_YcvwdwKLCm{x+gief;qN*eNVJN&5~w2{1@u9Ti0(mTew3jg+up}iO0Zo7aszx#UB z=PZ4cLU+99Ti?dSQ|j%=Tk{N%pJvf)yT$WR3WQOWAsy}BYf`=BP(QZwL4s85&8r+* zJ-1!G1lK0jxXZBbVy}6XF?3G~QZ~l^@}Gn=WQ2RcSkijiFHxzW4no`8+h`};ii92@ z$-o3`v6Bwo>ddhTFe=S`_d$VRTeu}FYbW#m%O?M5jtv~bZrF}#6JhpsMv;uXv=kx) z#N*m?7|rD-uv~K?fhV5?;52G%e8)US?F{}|MCjhobJTL$d(SxK(dOrZqM8q*Ldi`R z)M<;(O>huC3~N>Tg|Zg+(Sc7rRS#d?f-Qc6$`M79qq1xiCoLpohqSIyu)}XMTJ`5p zFnMMR35NX2wxvG5WehKu*FtjS4VE!@_`~wRXFFm9&%u~7Ke@_)6;j-f|3?du^9x_Y z5xCB*;P$mA)T-P3D_LcNYMDvh;!Js14u}S+ag1cQg1v*|{P9~}e6 zIr|{BGh&`FXv2b?fM$1~M?X4xwj+?Jj_x%#R*<*-WrFP6CWhiP=a}N%@+Vm-iHY+~ zDdHA6IZ0LCdKe0R$~##Xk}`~EyGGn_^{9I zDfG4r{Vi-K3IEs-Ra5Tds*ZLMI=Fa7ec;kpx#%h`%ME;rYszjHQO`GhEPJnjmdjmO zf?4oqBRg09_H5HNqoHJ~nrqzc)}UUXzbRT7wjvv-pRTKSn-2ec=c5KY->7x z=LEJb0fEHFh3PH^0n8-9+@{p{ou=i5u-k|TbX2T7iiht(hle(SH@?}Mk<>D){CMke zS2?ezuj1Is_qVs0_b|K`V}Gl5E{2>J8u<#9s?8T5{4MGgl!F>xLrr$e{G_||$-hU2 zgS<&~Yel&JXitlXf2B=vvGr$ho-X`3pq+Te;i~nv!P5c0tWlUph-vh;m37`|5L2F*zD70H zU_)#8!ccX#g)G-=MPwte&cFcv?3M-^hA1b{TK$JE^VCR*nc=tm_EFI#aVOWYo!qgx zl>)BLohqg{(9>fTKY`UW@r8cFjsV;L{lsdO5m0+Tw>f~RZ4G`GtNiy#br*)f19zy`&QB1Cv(a98DwRwi zh+U#zy2AeVVRQ+s&@Wolk0^Nh*8fYR1pMC*>-!f0`nmtTa=t?QlmERGd7&Hhf3L!{ z|1Y-x|Mh2xKyJywRJW8O`Y7^|GC>jsa`|V!d$uP8U3nm?RY(MaDvi3c_t&Hwte#>r zHVrNH&bTtfHz)|A#!MuVdHq`SI1t=w4yJ-Zhu6X41*)g*vKpv(i8PbC?fYIwIFRTb z95N+0lN3a!_mY^=4LfJ3mK`^0rmI1G_hHlxEI*$u?>Bl3Va5f?2NoqGmm%sX${bI6 zQyAqK*`Ex1JMK+ac9Gp%B??p=Z^I#-JZhWj&19srEx6&&FVrxeKi(>iEg-%D}evUMYQ{__!?K$ksku-3K%I$*H_Ys*6G-MtKL~B;o+*YsJpfB z79&(Nth@iMVd@YT{Ibt45|go)S(&Bi)8#4X`_r;Ro-2aiuBZnwp*AN^0$0Imw&y8H zql6R)R9utTDOw*PLXrwdT_3wECvN#~7Fz-i@4Za|yk5U!f46o}1AmlQ1_yDxw9Jvc z=eo(0v+3#WqwcsS14i;Ik0!@2*9sXmlQGE!&QXMQ#sfK9usS3R|7QZ}9O&w3F zK6zaf`Em%vw^Gr@(R=|msP^*oH51CyHQ8EKGv=azm;FKgC85PiaHP13B-Ek3gp`yg zDqKh2fy;~_G>fLO!ly^KkC)2hxw^OaZ1yG_3h3+f$pKJsK5R8++^7zbJ} z`jMTbCfC<-uwD{I1P;|vZra2Dm;z4e7{qU|lm*kC>B||K4|1tnB9Cqc(yB1jTyc(PG68X z;Ag$e>BM;>6i~)ssxq8Xz4gioEt_m_#mQ8_(`)MqiU0P+_$}ExKJ!6@Rv-IM+OH)o zUV%HZOEW-BCz}6P75%UE(cgN}9RlnvToA@-HWn@(*71x>(7}ZLmo|5xC{^!w zRera1EIe$F|JXkxP-4*5TP8x)pTo_IYY>ADBfT%Yhv1gQVPS~C@9Ft>&>u_W@k%R1 zjruRTt5@f3eHBrZcj!mD1`3R{4EQrFx)R`aA)0M1Z2=x4Z*cMVIOjQQ*iNdSZnv^f zQPG3n2*Xd_{j^MUn|Ww~g*&3=mqwFjlomT-2cKq(WivN1!~li>n}yiL>j7+P#gaNf zg!#+2Cy$5-i~WbGE-F9T!Ip!dq&M4)UbTW}=}pq&-Vpf(LBiEoI)1}!#KgviR*czk zRI#dCzdiQw&{kL#0dCTj&QF7R zq|v|^O_R=NNn;Xw7lwigc9V2HCF4oSBEyr=Y&)tnD?cAuh)a}8Yt!a^{e)mK@(P7We-@-3l+GzB;tiNiJq>OdZ{zk-9e;QY%Y7;5MLi0vX=8C z;0h8l@l3auW3eL3jV%dI(h2KC3tR`HzQ-#`^(Gjf|72#Lr4W32%euxcA(@B0h;rt7 z9&<7y%aq!VodX{`aYn3W6Eyjd9LPXRCH1m%&LVWMk@|yt{lT2dua6X>D$XDV;*Abk zqu-5%V#ne6%&?c{a+F`kBtzm+F|8N(!r+2OH)g-=SAZ2ex}n$aCMaL@gKJjn1PhSA zYkwLVww5h5Gl7rDIy|Cw*oIg07>DT7F(Tme2G;E>YNAcIpKgiDS$l6Uozcy=K!*Ef z&q&ld8y(=x>RE(-@5jk%1c$O#(-F)}G7S%?YW_+T|uHj?+%B))#fIFAr_zP`APuIz z=d^ZZ`0>rQuMb3eGp&Y}pZ))^5JNv_?L6lzzefn3PPa{EbHDbX%AnX8-9|&Ro@vTz z8N)p?<4v=@fWyJB`G4hUF2Nd1;>S;|xUD<39=BZs_AH{vRA{88!CpQG{tK5l8VyEm zzht=@S2x!tk3F+8IXoG_eh{WcbpXEtso$roxIY=)nJT4*-upxaeOSg1QI*Qa>#b(N zGI(`H7)YqfAi~dg)KW5i1P;@%kiJMFU)soFl*jowR3D+=8%V!&*Wj-mSa>R|@R(A$ zM88TA(fTx^?7AHdfWR=mvWyI(cxRK7a@zYFIA|m6_Kvz&PjMr7NC;wk){?x%UaKDqS1`%}tfXz>^A%1eB0aR4>x@I5N} z@IGU!^KqsA*z(Wx^_99ukDT^DFGH|JPtFt&~Q%l zG(~QSuWPo45Ja2a6q@}t=SO&m=6Hb3z6%6WcQltb#a=FPY@5VuUnf;wQ*}YL8>ocI z582^w;6`vEEhfxhEdS|#ScM363g!SU9`D_X3A`(PG{hFQqPogNmy2-^ksrl?#`tH$RO zOq!~r;*PadyV8qin7vj}VeP+>h{5P?&Q)I}8oj*4>b017(&&cB&;9J5Z`Jk>F|9CX zg+nkp@%fdqe@P^Q?@kgt7i&TT@RvgP0ICcCD_OsPjZGgkH@7}RIuu?rauFf3crFMF zN>IGXAhZJvhmvebxli4A*Zg72*7x2nB4fD)<{B&Tdg@mULALW(=Fzt43t72`uO(lL zp(@Z7we~967xuPerb+Gksygh zf(tvdgvjQ$>3C6d6wK~Zc$Lz$L)(0sT<=&W^>VbU3iTr2|9KkvlULC&adG4J=XmXw!UtOCU`o5ZjfzTyMW4upH|9Qwmey$rLk+Q zEIU#-b71~bzJ6$sz;Cfb7dTzTAzcXqF$u0eNaUc%uNBb>jwTMv&OKnYpZ^ByVLynP zIIPd7Zi-YZB$B0Z-MGj_ofF3V^ru_g@snV8%O!<~iqS_hiH=D|q22b7-x^{DOaj!( zu`9nSfR`}5*nw!`O^>Aig1o`Z6>#2TfEb)|$Elu`Yu1`agc^TfKJy@&Dld^@;Q^^A zEY0#vXui4bi?s?d>b9JFxij-f%$WO6r?FABA8}VFM6-~X^3!*wqf=5|xr+hu@lM@mmMSLERAsX$LbyC@==;$!`KJ2d`xIq;kPYZIWYD)9u3AUT)B2Q~M>Ns0J}NDZ%lp*1#GVHunuD^mG^ zbLhg|;IRnm&fSxShUT$AI3{MI;&i>y9?7e<5FhrBpE82Ww70n|^(RN`aITt~@OA|> z>n)3&n_A38)uvN9#LpJ3K%J<3pVC%Gx>BMWe(#okU=TcXB*=5opDEQktrPv*yJm>E z<>Hy2(D&DMW>AVE6||#Uc*}^osHbp()61$>$)in^Ne+>GjSZ>lumwr%BK&mYO>D|2 zN^rB%+9ke7*%9X0L0C2FzBxvBzkdbFGZ=w-@OKM%EQzBnYBC*=DYqEs1*wGm|4PfL&{U=Oq8;nKaN?Hv*t78q!XQ!7UQn#Qqg z*xvgZ-QFJ7UsUBzs)&FT5>l*EM1@B~i~1V7Xg=eeXuhzBh;N_+UFRg@f7;OaB%RGH zRk5E5+k$Moysa9-ER7h4DR2;fG67~skI2j>T=f0a4#Q!CURGUZX4wd0UI_m+N$`Fw zYK841`iR~Iv@p86JW~0U=3;p)>u_uadLpBY$r~4zQ3z?mx9eRoL{>?OFLFd)|65Yq zQxeUu8z2#@eC_#1GxoCMr?y0zpfmpZq9NuPndaN?dZ5rPK|gx|qVug73X?V$bA`IL z@LqEAx~A)E_lw;q`CxR0DXYx|hQj*U@avnub?@DmfvqJp%z3D&bEdX7Q9ZEr@-8{+ ztG)q?ts8CFKTY*|2ES&^aK;hk@ft=~3dNOOL1~NpO6Ui&xZS@8eT`O&7bEFSNJ986 zL(<-5J^fog^7f#?{##p;TLi2tVswtMB$&w-J429zsYCxSH||5l^YvN&EDs--9!=$DhI4?&6ECzGG&x zBF8Ju!^x^#W4ow&hRtCw1U3;9cw)cY;Mw}zm3eu*^yqje2C#SkK=sjV&6Snd(|fl| zbzZk-8Uj0cm(wtAOX8i*CRKJZF^zl=>^OM{MAL}v2e14_G>i)EG95W-2Z_mR9C(P& zN^AcQisIExRbJg|5*9mNP}4y4=gj7LxFw`O$t=b?Jld~+xbHn*ffFYo*Pm>>z3f50 zuz8ewpTB>5v~Xcx&FV4_E4GH@rihT4>N0i30ic8a@H_&B$rnUbD=A9Ke^(m7ipAfu zRSo>%b-7(stvz31SyUj5VmAp*$2{ae8yT!1R)0h)5=GH%O<@pLGRC2HJe+%FR7sfO zv-IsgbB1#4rNK@irPNSuy+!JR_j572qZEzOa5^8kF(N+&lZw~a_|&i{)w5DBrT&{mC2l1^Y_yXQ=?yQ#FpPZObfghi?oxdM3+#6vuQPoW4cqg)= zmma_61h6syQBw2K!pzt%_JY!R2eeDSv%L}$hYnk}z4#SuFrPua7Ljtn40=ER22UlP zfdCgx&AEoJhjsI&M*tg$_M2ogowoF5Efl@ELu{RKiWBC8^+ko)-9G4BBj=#l?y1L_ za(ir-cY)@2-t&7Ay@kf@ArZLGZ=hB|`>o9Ev}3Lr6?Z7kCt$&ERV9y;2fiwCj`;UMS*I|Zv61Ol+e);^9PbyuZrLMj!9e?kxav} zDc^(WbK4Z)xzDk~do`*}NJrA+X>|1}Jlp2UY|r!@ShSrt#3oQL4kmT;QA$Bn;s9F^ zG=x8D7gVP053V;r#ukA)zq1S;7>slw$>mvDUoKvJ13-(<1MctuLNIa7oA;v03G3b{ zUZ;Hi`l(!-$=itx!lHBaSwuM*T6jX6>2v2fPPe9w3Z-3LPLDnfNXOP8qH6JP1OfN3 z*7oLvz556EAJ*pET#3{TTW;zW%V2X=T0Iw0l{@_V{?-5#K;vx4I|3%Ou*V zag;xo1~ms;rxL4;DlW90`L2NSHU^Y9_PGd!7vJ_5q`4kc%$IMj{__``rlvY_FKzRNsgo-c38eWGxuc_ z?9|mL&aUt(%j3a=CAoIy=&b@#5)UL7dXe-*@zlVQ+6B6-7=WkOd z`po08#^kDdEe?>gh*t-`1MGC2HEr~;kI$XniEuGed3JZ7KgWg>Z~iUPmWUvvD|yY= zy~Y^fLJ~mc)g;1@(DgWK#p89r1Th%jdl8o5^)dV9$`{>8(A*m?+>s)IogA|k5Sttu zDsmr1#kxukYSoZlqkmU7zdmr4FNK+{#U_FFT0EjUSmDjL{=dX_?T)0N6B=sa)&;An zQRj@+rrm;ZZpO1wxNkqj&fXcGXR_FG6iIY^O3H|fa(eEU$!c+MEVry5>-s%e zx5L8p`5rpS2Ty1x`c!Xd;dd|acL0lEiG`AhgQWW|B4}ecwBl$CA;%Rfs) z#?5f0xwgj`J;KjssP(8BJ8X7*_#^4?sTgV}mzZ5a%!pYuxC~dh9FE6Ek-aN2ec+w3 zFNJYTE2*JaImR1nNA?7L;Iz`yT$)gnqt_G$pY$&UQlK< z+)qOr`%0rn?1UC4P%lFH6drzNztKPZ8eBg4N)+Z%)i|H|M;WdJ%a*E$et!NrTkS=T zamXJAoQmPF+yCFWs=p*CS^~oNouR1Tww#{A1riQQ;LXaNDfruRsZkzfb{r@^HMCtW z3S1rjg&;h|(7^V^ADJm`ZspBWMrjP?ei3zb0_~YE>Lla<8NLF&)JZi=((Z|Btiyi_ zuDN}>lZ|;B9*;9vSkGs1-~m!YgsFsv%innc2O+p}37mU2x~f}oW~0zl0To~o#kwPd zO*G#8w~TF0TCmY2+-FxgeuktXp7pmrVJMZi4uT+5NqsUd9z2#8NGezOUOBgxvX#K80HCmRx(u zQ=LJjh`0g^H8mV;9({ypdnFNA+U9H-|Y(A&>It>ReT%w!9oQ60pOoGr$ zwl|5h;7r`k4-hVW7hcG7fH#kHfBj|QuJ}UGK>CO^X;$7@8K;itb_CJtk>o zGs>bFU|AXty&=kJYMo0PEch(SWb_70P4wOln!PPaDpUYnXC}|k-Q53-$3_0{j;N!K zvK!HO=!krlNR&KeHS6VjWVf)LLP9_oaj=$SN|$`LM8$7m=Ga7jLT-qD#uhaPrb};f zR(N%FL$&+j9lKu6Pg~hq;e3}rqr<;@Y~Jd>M-Bf5D8ZkXvg7%|vVJ1Oy+&*;0-i7N zi2xB%JD%&pO*qXas$=~=Pl-NapGf#b?QJ+f#9iGMKY|J{LN;P7U&0J5IVC$a&fD+b z6RzZ$@I~Tdko<(Qe$OBZReFPO!4u;I#8%9Ylpn|UaicS3iUKPiiO(*l-;D(WL~;?J zQJ=aCx~7KVXW+_OX6)r-^)7X4X#r(K?6WKcXn{Bpe2MLnU5dT;)1c~YUq1CAZ}Xcx8OP!>KBr$q<|hA6e| zfD+!4)AnWDE{E+%gg7UvF~M}Kg9_bna=v_SM}-L_t1!#+Jy3A4R7i@7<{e0My3>k> zWu!w}9D8J^1f_rHmzS2s=yw9h;a!~aA6^6Gr&!Dw?O@>rb||hrGL<=UAS+<43lF#J zpMaGYSkA6b`in{kxlQStTq=WR5?YV{P6LP>#I=v2CQRSsCJn4jLbytq#z2z$*t+a? zD?i)vNHJYz*9Bbm1q)-Wr2S!^?zi_*zM@ots^NV*e@j~jUurrXUSCy0gJWEQqS&&2 zP8Z?hEmfLh^OfF=EW)m28JTb4-uz$v`=@e@=$^~E4Pa6D^G!D`%EI^o*8{JpR z?{@}S^3dg;i;Ftbf;R{Nzi_iLVOWQNHSPh0K2~i-C;?RhoU|43;v;~Y0D_4O!VlC% z)0>=n{BYWT%qfflzmvH+aDXVNXbQM~ZlBNa78P}qB@)DQ`Em=>VFV$Tc5`Tl+XtfOaZz;-A6g!QHv@A)$DOp(eKC;;BX->8pRfcLeZtpS)%e zQzaqrow2c(s~OioyJ|iRCr%!?Eb_T)l5w@QJQZz1QBMof1w;5{rTqom;a#;D5NwFZ zs4#Jr-C<>@e|e*tR|?`V;2ZjX-LemI5TvS5iJ{_XzWbc500y84It=$*$=-mae;-Wi zYXaG7e10n{3z}MNs+THw@b{Ub;0oU(iT2_G#`~P$i(hj=xzd*=0Bo_2TEMntaJI=X zGD!5l?)Y_E05o?nmUE&0o+){(?&af=HA!)1R-soivsjQb-8s zrc3*!fK?N_xcMt6Ta1liP?;Yb>Di$dTHbH^t=f5+s1EQ|zP1sEVg@m1))PRi~wq{P9?pfQDmC1fWxO zHG8|kBd@hAuMfP3# zSAn{sUaG};*=73H?aT&TDa!uIsWcQlp|M{w>0%ok1>=KgXbsULOZ=UAdb)K;K#VXh zAs0{4l}3N=paeO( z-Ux*d|Ixkd_zc=rASa+;;mTnC2rlaaD-K%oKJ3qg=HwMa7GX$O`j>7ir4kr01IPq= z<26g%?4UOnE15?qXh^w^^SwBJ$|+7@w~&Gi!e{!~gzNOOEgaym3X$zb*x(Yq;@h+@ z1Qy@wm%~!Pb_hXs|26Q>LCVxhwVcknS6L06m?=S|+!BV~@kYrD;x*?^92z$$UBVCo z-(e1+>=dR1ooE+6C0M>Yr1aYjCTRJ21lMN=$w8^mmjC+oYoAJyl2{X<6#*Ij&ky=d zotM8~9L&2E_mrm|e3Cf+j;=`=ifky~RqK$)yR+Deqa6sOJv-D;eotTaiH}YqMfpWX z#G*%g*gCy0121p6&2x)8s=#!=gT>!4;HcQM zRj5!WJnEAWdGp)+0eTBn=?$o(fQOX&!b6;A%3b{h_MrMBL2E!#vjJ}R)Au)@;C$&I zh!9>pN0(7JLcoccyI!rM1B&wnLhXg?OTgg|->i6i%Lr(T>7#p%EVciiZxn<0@<0L} zY!Ow^k?#mVL^EPThVBK-;o2dxB;L+XDH%V?U^K<}JOJwLnph2!20JDnpJeU%aG_t$ z?_#Z*-+}C=k#XcbP?ngh!f{Km}x8 z6j#j+Mpv6|yqcOuFutu2gj`R?Mtn6AU(j>*(7jgO<(}HTBrhmC^TozeEV{Grr;M<( z8*#g>y7=#sQNwn;g1`Y{vKzI4rLFF0tSRhMYm@aW=!K(5fxUN`Q_I`zGq3&ca(_k@ z!h}$Pgy#}c!Z!zzgRsU(<#W-E>WOfoqhB0M|3FL%36Y8i33Za2*kStlMmt%T`&C}< zbpm_`o>24uwKHd?i{%|ZzxUiJq415s#X?SMpHzXitO@{5Z$5tglKaSYz3PzWH`o7j z0!f!Xb%}q{y+7;RA z>wk0!W2EWiR)&D!@aJFen|Uw+xHfI?bl_BlPV`$P+oDIC7TBx*Js|`9kzos!re@uKbUz^LB zA)CR6eL-Nbd-SfNlRO8nZ$9SSY%e7Y+WeKCl5|+?`W*RfJNMOn^-|JU4=Tw+F0nnE zcOs18!m`-c*Nvb1uD=9qfpN6Tuz^-qhRljn*?VAq-t>*7f4@!7)O>lhSY4iJ1Iwm+ z4D!a_C+bY(SQwst^;OZHe(nNr6%@Ob>6XeO0l|q}z@;AlEJ{VSulk<*jvEy*dq0Tl&DlBS&x5!L zd47I=SMNmLV|d2=VSS44CGi7hPyS`Ztew@iC4YV2rNG0#GH>y}Ji2g$i_6rQ5+5eN zxx3pQFaPhg`>}Y)VkK~TxU_`zK{mr-hLUOiYnDao9tVz|^2k|j$lC3fn;3Ci0N69T z83pvJOX~+<>)M60^TWP}pHIDgar4Ur_x|jsGv~5CFiP3W;1~KRZPNb?8^KK_I#LV) zTeGgt0Ztxv$g1B6eSd8yZ%=O)a5>&W;Cd-jpx1#Dc@N)Jnm<4Auz2Nv8;+~%SAdI_ zHCLDogc@S+Nvm4=`koEnn)Q0W(W>kzmk;dOw_x4M!cx9|U5ew>(kFKZmgNH z;`#c=Nsmo~gMzjc6`y%JmG6M;0Z)eXj~}1T(bjWj_{MOI@de*a{*zoz3!iB>N9{@q zp6tH_Sch^-&A57X-8$7Zr_z9{Wj^fj@T``U6jiSill*R_rMB)?_mdFalk6pTQzrDC zdhM3Uk-})LeWLA1(H%x$90We>U^v3aai;db4yH(dhAMN}6HH}?E9AXgj=WBDJ**%b zx1|W0SG)@{an^LB{Ts5<2s2z literal 0 HcmV?d00001 From e45144f7e1dd608db3b4985bd5adc136bd4d30c3 Mon Sep 17 00:00:00 2001 From: Robert Krawitz Date: Mon, 10 Jun 2019 21:40:14 -0400 Subject: [PATCH 24/31] Graduate node PIDS limiting to beta (#14425) --- .../en/docs/reference/command-line-tools-reference/kubelet.md | 2 +- content/en/docs/setup/release/notes.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/kubelet.md b/content/en/docs/reference/command-line-tools-reference/kubelet.md index f19da3d42e8a2..4d4ec8f4b9f88 100644 --- a/content/en/docs/reference/command-line-tools-reference/kubelet.md +++ b/content/en/docs/reference/command-line-tools-reference/kubelet.md @@ -545,7 +545,7 @@ kubelet [flags] --feature-gates mapStringBool - A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ReadOnlyAPIDataVolumes=true|false
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
ServiceProxyAllowExternalIPs=true|false
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
VolumeScheduling=true|false (ALPHA - default=false)
VolumeSubpath=true|false
+ A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ReadOnlyAPIDataVolumes=true|false
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
ServiceProxyAllowExternalIPs=true|false
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (ALPHA - default=false)
SupportNodePidsLimit=true|false (BETA - default=true)
TaintBasedEvictions=true|false (BETA - default=true)
TaintNodesByCondition=true|false (BETA - default=true)
VolumeScheduling=true|false (ALPHA - default=false)
VolumeSubpath=true|false
diff --git a/content/en/docs/setup/release/notes.md b/content/en/docs/setup/release/notes.md index fb2bb8d38583c..8f774f88e796d 100644 --- a/content/en/docs/setup/release/notes.md +++ b/content/en/docs/setup/release/notes.md @@ -411,7 +411,7 @@ New "APPLY" value for the "verb" metric label which indicates a PATCH with "Cont - Fixed dockershim panic issues when deleting docker images. ([#75367](https://github.com/kubernetes/kubernetes/pull/75367), [@feiskyer](https://github.com/feiskyer)) - Kubelet no longer watches `ConfigMaps` and `Secrets` for terminated pods, in worst scenario causing it to not be able to send other requests to kube-apiserver ([#74809](https://github.com/kubernetes/kubernetes/pull/74809), [@oxddr](https://github.com/oxddr)) - A new `TaintNodesByCondition` admission plugin taints newly created Node objects as "not ready", to fix a race condition that could cause pods to be scheduled on new nodes before their taints were updated to accurately reflect their reported conditions. This admission plugin is enabled by default if the `TaintNodesByCondition` feature is enabled. ([#73097](https://github.com/kubernetes/kubernetes/pull/73097), [@bsalamat](https://github.com/bsalamat)) -- kubelet now accepts `pid=` in the `--system-reserved` and `--kube-reserved` options to ensure that the specified number of process IDs will be reserved for the system as a whole and for Kubernetes system daemons respectively. Please reference `Kube Reserved` and `System Reserved` in `Reserve Compute Resources for System Daemons` in the Kubernetes documentation for general discussion of resource reservation. To utilize this functionality, you must set the feature gate `SupportNodePidsLimit=true` ([#73651](https://github.com/kubernetes/kubernetes/pull/73651) +- Node-level support for pid limiting via `pid=` in the `--system-reserved` and `--kube-reserved` has been graduated to beta and no longer requires setting the feature gate `SupportNodePidsLimit=true` explicitly. Please reference `Kube Reserved` and `System Reserved` in `Reserve Compute Resources for System Daemons` in the Kubernetes documentation for general discussion of resource reservation. ([#73651](https://github.com/kubernetes/kubernetes/pull/73651) ### Scheduling @@ -891,7 +891,7 @@ filename | sha512 hash * Breaking changes in client-go: ([#72214](https://github.com/kubernetes/kubernetes/pull/72214), [@caesarxuchao](https://github.com/caesarxuchao)) * The disk-cached discovery client is moved from k8s.io/client-go/discovery to k8s.io/client-go/discovery/cached/disk. * The memory-cached discovery client is moved from k8s.io/client-go/discovery/cached to k8s.io/client-go/discovery/cached/memory. -* kubelet now accepts `pid=` in the `--system-reserved` and `--kube-reserved` options to ensure that the specified number of process IDs will be reserved for the system as a whole and for Kubernetes system daemons respectively. Please reference `Kube Reserved` and `System Reserved` in `Reserve Compute Resources for System Daemons` in the Kubernetes documentation for general discussion of resource reservation. To utilize this functionality, you must set the feature gate `SupportNodePidsLimit=true` ([#73651](https://github.com/kubernetes/kubernetes/pull/73651), [@RobertKrawitz](https://github.com/RobertKrawitz)) +* Node-level support for pid limiting via `pid=` in the `--system-reserved` and `--kube-reserved` has been graduated to beta and no longer requires setting the feature gate `SupportNodePidsLimit=true` explicitly. Please reference `Kube Reserved` and `System Reserved` in `Reserve Compute Resources for System Daemons` in the Kubernetes documentation for general discussion of resource reservation ([#73651](https://github.com/kubernetes/kubernetes/pull/73651), [@RobertKrawitz](https://github.com/RobertKrawitz)) * The apiserver, including both the kube-apiserver and apiservers built with the generic apiserver library, will now return 413 RequestEntityTooLarge error if a json patch contains more than 10,000 operations. ([#74000](https://github.com/kubernetes/kubernetes/pull/74000), [@caesarxuchao](https://github.com/caesarxuchao)) * kubeadm: allow the usage of --kubeconfig-dir and --config flags on kubeadm init ([#73998](https://github.com/kubernetes/kubernetes/pull/73998), [@yagonobre](https://github.com/yagonobre)) * when pleg channel is full, discard events and record its count ([#72709](https://github.com/kubernetes/kubernetes/pull/72709), [@changyaowei](https://github.com/changyaowei)) From 3d1d270e3c8cfced6889f9aae2ee44c465a91982 Mon Sep 17 00:00:00 2001 From: Qiming Date: Tue, 11 Jun 2019 09:44:16 +0800 Subject: [PATCH 25/31] Drop .travis.yml from dev-1.15 branch (#14812) --- .travis.yml | 22 ---------------------- 1 file changed, 22 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 0fe57ab80e470..0000000000000 --- a/.travis.yml +++ /dev/null @@ -1,22 +0,0 @@ -env: - global: - - KUBE_VERSION: 1.13 -language: go -go: - - 1.11.5 - -jobs: - include: - - name: "Testing examples" - cache: - directories: - - $HOME/.cache/go-build - install: - - bash scripts/test_examples.sh install - script: - - bash scripts/test_examples.sh run - - name: "Hugo build" - install: - - make travis-hugo-build - script: - - hugo From 2a0f39fb41a60e1b705ce942090a1c0e4173cb19 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 11 Jun 2019 04:56:16 +0300 Subject: [PATCH 26/31] kubeadm: update the reference documentation for 1.15 (#14596) * kubeadm-ref: update placeholder generated files * kubeadm-ref: include reset-phase.md document Also update reset.md to reflect the addition of reset phases. * kubeadm-ref: update authored content * kubeadm-ref: remove imports for removed "config upload" commands * kubeadm-ref: add page for kubeadm upgrade phases --- .../setup-tools/kubeadm/generated/kubeadm.md | 30 +- .../kubeadm/generated/kubeadm_alpha.md | 43 +-- .../kubeadm/generated/kubeadm_alpha_certs.md | 43 +-- .../kubeadm_alpha_certs_certificate-key.md | 29 ++ .../kubeadm_alpha_certs_check-expiration.md | 25 ++ .../generated/kubeadm_alpha_certs_renew.md | 45 +-- .../kubeadm_alpha_certs_renew_admin.conf.md | 35 +++ .../kubeadm_alpha_certs_renew_all.md | 95 +------ ...alpha_certs_renew_apiserver-etcd-client.md | 99 ++----- ...ha_certs_renew_apiserver-kubelet-client.md | 99 ++----- .../kubeadm_alpha_certs_renew_apiserver.md | 99 ++----- ...pha_certs_renew_controller-manager.conf.md | 35 +++ ...pha_certs_renew_etcd-healthcheck-client.md | 99 ++----- .../kubeadm_alpha_certs_renew_etcd-peer.md | 99 ++----- .../kubeadm_alpha_certs_renew_etcd-server.md | 99 ++----- ...dm_alpha_certs_renew_front-proxy-client.md | 99 ++----- ...ubeadm_alpha_certs_renew_scheduler.conf.md | 35 +++ .../generated/kubeadm_alpha_kubeconfig.md | 43 +-- .../kubeadm_alpha_kubeconfig_user.md | 97 +------ .../generated/kubeadm_alpha_kubelet.md | 43 +-- .../generated/kubeadm_alpha_kubelet_config.md | 43 +-- .../kubeadm_alpha_kubelet_config_download.md | 69 +---- ...adm_alpha_kubelet_config_enable-dynamic.md | 75 +---- .../generated/kubeadm_alpha_selfhosting.md | 45 +-- .../kubeadm_alpha_selfhosting_pivot.md | 89 +----- .../kubeadm/generated/kubeadm_completion.md | 47 +--- .../kubeadm/generated/kubeadm_config.md | 53 +--- .../generated/kubeadm_config_images.md | 55 +--- .../generated/kubeadm_config_images_list.md | 80 +----- .../generated/kubeadm_config_images_pull.md | 88 +----- .../generated/kubeadm_config_migrate.md | 70 +---- .../kubeadm/generated/kubeadm_config_print.md | 51 +--- .../kubeadm_config_print_init-defaults.md | 59 +--- .../kubeadm_config_print_join-defaults.md | 59 +--- .../generated/kubeadm_config_upload.md | 61 ----- .../kubeadm_config_upload_from-file.md | 74 ----- .../kubeadm_config_upload_from-flags.md | 144 ---------- .../kubeadm/generated/kubeadm_config_view.md | 53 +--- .../kubeadm/generated/kubeadm_init.md | 258 ++++-------------- .../kubeadm/generated/kubeadm_init_phase.md | 47 +--- .../generated/kubeadm_init_phase_addon.md | 45 +-- .../generated/kubeadm_init_phase_addon_all.md | 127 ++------- .../kubeadm_init_phase_addon_coredns.md | 103 +------ .../kubeadm_init_phase_addon_kube-proxy.md | 103 +------ .../kubeadm_init_phase_bootstrap-token.md | 69 +---- .../generated/kubeadm_init_phase_certs.md | 43 +-- .../generated/kubeadm_init_phase_certs_all.md | 95 +------ ..._init_phase_certs_apiserver-etcd-client.md | 79 +----- ...it_phase_certs_apiserver-kubelet-client.md | 79 +----- .../kubeadm_init_phase_certs_apiserver.md | 111 ++------ .../generated/kubeadm_init_phase_certs_ca.md | 63 +---- .../kubeadm_init_phase_certs_etcd-ca.md | 63 +---- ...nit_phase_certs_etcd-healthcheck-client.md | 79 +----- .../kubeadm_init_phase_certs_etcd-peer.md | 79 +----- .../kubeadm_init_phase_certs_etcd-server.md | 79 +----- ...kubeadm_init_phase_certs_front-proxy-ca.md | 63 +---- ...adm_init_phase_certs_front-proxy-client.md | 79 +----- .../generated/kubeadm_init_phase_certs_sa.md | 48 +--- .../kubeadm_init_phase_control-plane.md | 45 +-- .../kubeadm_init_phase_control-plane_all.md | 152 ++--------- ...eadm_init_phase_control-plane_apiserver.md | 115 +------- ..._phase_control-plane_controller-manager.md | 91 +----- ...eadm_init_phase_control-plane_scheduler.md | 83 +----- .../generated/kubeadm_init_phase_etcd.md | 45 +-- .../kubeadm_init_phase_etcd_local.md | 71 +---- .../kubeadm_init_phase_kubeconfig.md | 45 +-- .../kubeadm_init_phase_kubeconfig_admin.md | 87 +----- .../kubeadm_init_phase_kubeconfig_all.md | 95 +------ ...nit_phase_kubeconfig_controller-manager.md | 87 +----- .../kubeadm_init_phase_kubeconfig_kubelet.md | 95 +------ ...kubeadm_init_phase_kubeconfig_scheduler.md | 87 +----- .../kubeadm_init_phase_kubelet-start.md | 73 +---- .../kubeadm_init_phase_mark-control-plane.md | 59 +--- .../generated/kubeadm_init_phase_preflight.md | 65 +---- .../kubeadm_init_phase_upload-certs.md | 2 +- .../kubeadm_init_phase_upload-config.md | 45 +-- .../kubeadm_init_phase_upload-config_all.md | 63 +---- ...ubeadm_init_phase_upload-config_kubeadm.md | 65 +---- ...ubeadm_init_phase_upload-config_kubelet.md | 65 +---- .../kubeadm/generated/kubeadm_join.md | 173 +++--------- .../kubeadm/generated/kubeadm_join_phase.md | 4 +- .../kubeadm_join_phase_control-plane-join.md | 4 +- ...beadm_join_phase_control-plane-join_all.md | 5 +- ...eadm_join_phase_control-plane-join_etcd.md | 1 + ...e_control-plane-join_mark-control-plane.md | 1 + ..._phase_control-plane-join_update-status.md | 1 + ...ubeadm_join_phase_control-plane-prepare.md | 4 +- ...dm_join_phase_control-plane-prepare_all.md | 5 +- ..._join_phase_control-plane-prepare_certs.md | 5 +- ...ase_control-plane-prepare_control-plane.md | 5 +- ...se_control-plane-prepare_download-certs.md | 5 +- ..._phase_control-plane-prepare_kubeconfig.md | 5 +- .../kubeadm_join_phase_kubelet-start.md | 4 +- .../generated/kubeadm_join_phase_preflight.md | 1 + .../kubeadm/generated/kubeadm_reset.md | 95 ++----- .../kubeadm/generated/kubeadm_reset_phase.md | 19 ++ .../kubeadm_reset_phase_cleanup-node.md | 25 ++ .../kubeadm_reset_phase_preflight.md | 25 ++ .../kubeadm_reset_phase_remove-etcd-member.md | 24 ++ ...beadm_reset_phase_update-cluster-status.md | 23 ++ .../kubeadm/generated/kubeadm_token.md | 60 +--- .../kubeadm/generated/kubeadm_token_create.md | 109 +------- .../kubeadm/generated/kubeadm_token_delete.md | 65 +---- .../generated/kubeadm_token_generate.md | 61 +---- .../kubeadm/generated/kubeadm_token_list.md | 61 +---- .../kubeadm/generated/kubeadm_upgrade.md | 47 +--- .../generated/kubeadm_upgrade_apply.md | 151 ++-------- .../kubeadm/generated/kubeadm_upgrade_diff.md | 83 +----- .../kubeadm/generated/kubeadm_upgrade_node.md | 56 ++-- .../generated/kubeadm_upgrade_node_config.md | 86 ------ ...upgrade_node_experimental-control-plane.md | 86 ------ .../generated/kubeadm_upgrade_node_phase.md | 19 ++ ...ubeadm_upgrade_node_phase_control-plane.md | 25 ++ ...beadm_upgrade_node_phase_kubelet-config.md | 26 ++ .../kubeadm/generated/kubeadm_upgrade_plan.md | 103 +------ .../kubeadm/generated/kubeadm_version.md | 51 +--- .../setup-tools/kubeadm/kubeadm-alpha.md | 22 ++ .../setup-tools/kubeadm/kubeadm-config.md | 6 - .../setup-tools/kubeadm/kubeadm-init-phase.md | 2 +- .../setup-tools/kubeadm/kubeadm-init.md | 61 ++++- .../setup-tools/kubeadm/kubeadm-join.md | 7 +- .../kubeadm/kubeadm-reset-phase.md | 53 ++++ .../setup-tools/kubeadm/kubeadm-reset.md | 13 +- .../kubeadm/kubeadm-upgrade-phase.md | 26 ++ .../setup-tools/kubeadm/kubeadm-upgrade.md | 19 +- 125 files changed, 1609 insertions(+), 5876 deletions(-) create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_certificate-key.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_check-expiration.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_admin.conf.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_controller-manager.conf.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_scheduler.conf.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_preflight.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_update-cluster-status.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md delete mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_control-plane.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset-phase.md create mode 100644 content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade-phase.md diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md index 3d114781675a5..3758917b6b353 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md @@ -5,7 +5,6 @@ kubeadm: easily bootstrap a secure Kubernetes cluster - ┌──────────────────────────────────────────────────────────┐ │ KUBEADM │ │ Easily bootstrap a secure Kubernetes cluster │ @@ -38,29 +37,8 @@ Example usage: ### Options - - - - - - - - - - - - - - - - - - - - - - -
-h, --help
help for kubeadm
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + -h, --help help for kubeadm + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md index ced9d4b528ee8..4211c0b57e002 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha.md @@ -3,48 +3,17 @@ Kubeadm experimental sub-commands ### Synopsis - Kubeadm experimental sub-commands ### Options - - - - - - - - - - - - - - - -
-h, --help
help for alpha
- - +``` + -h, --help help for alpha +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md index 8548203f74cc8..8ddf61d005ff0 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs.md @@ -3,48 +3,17 @@ Commands related to handling kubernetes certificates ### Synopsis - Commands related to handling kubernetes certificates ### Options - - - - - - - - - - - - - - - -
-h, --help
help for certs
- - +``` + -h, --help help for certs +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_certificate-key.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_certificate-key.md new file mode 100644 index 0000000000000..c42340618fac5 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_certificate-key.md @@ -0,0 +1,29 @@ + +Generate certificate keys + +### Synopsis + + +This command will print out a secure randomly-generated certificate key that can be used with +the "init" command. + +You can also use "kubeadm init --experimental-upload-certs" without specifying a certificate key and it will +generate and print one for you. + + +``` +kubeadm alpha certs certificate-key [flags] +``` + +### Options + +``` + -h, --help help for certificate-key +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_check-expiration.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_check-expiration.md new file mode 100644 index 0000000000000..3cf14558b0d28 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_check-expiration.md @@ -0,0 +1,25 @@ + +Check certificates expiration for a Kubernetes cluster + +### Synopsis + +Checks expiration for the certificates in the local PKI managed by kubeadm. + +``` +kubeadm alpha certs check-expiration [flags] +``` + +### Options + +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for check-expiration +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md index f212b9e12be0f..4aba5d96187bf 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew.md @@ -1,9 +1,8 @@ -Renews certificates for a Kubernetes cluster +Renew certificates for a Kubernetes cluster ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm alpha certs renew [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for renew
- - +``` + -h, --help help for renew +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_admin.conf.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_admin.conf.md new file mode 100644 index 0000000000000..fafd0c7641aaf --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_admin.conf.md @@ -0,0 +1,35 @@ + +Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself + +### Synopsis + +Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself. + +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. + +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. + +``` +kubeadm alpha certs renew admin.conf [flags] +``` + +### Options + +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for admin.conf + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md index 60fa27fb569eb..ba6e1bb75a8e5 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_all.md @@ -1,10 +1,9 @@ -renew all available certificates +Renew all available certificates ### Synopsis - -Renews all known certificates necessary to run the control plan. Renewals are run unconditionally, regardless of expiration date. Renewals can also be run individually for more control. +Renew all known certificates necessary to run the control plane. Renewals are run unconditionally, regardless of expiration date. Renewals can also be run individually for more control. ``` kubeadm alpha certs renew all [flags] @@ -12,85 +11,19 @@ kubeadm alpha certs renew all [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for all
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for all + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md index 3489a9d6773ab..e7c791ed1b9c8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md @@ -1,12 +1,15 @@ -Generates the client apiserver uses to access etcd +Renew the certificate the apiserver uses to access etcd ### Synopsis +Renew the certificate the apiserver uses to access etcd. -Renews the client apiserver uses to access etcd, and saves them into apiserver-etcd-client.cert and apiserver-etcd-client.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew apiserver-etcd-client [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew apiserver-etcd-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for apiserver-etcd-client
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for apiserver-etcd-client + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md index b6e5f7406f76e..064e0c24db66e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md @@ -1,12 +1,15 @@ -Generates the Client certificate for the API server to connect to kubelet +Renew the certificate for the API server to connect to kubelet ### Synopsis +Renew the certificate for the API server to connect to kubelet. -Renews the Client certificate for the API server to connect to kubelet, and saves them into apiserver-kubelet-client.cert and apiserver-kubelet-client.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew apiserver-kubelet-client [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew apiserver-kubelet-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for apiserver-kubelet-client
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for apiserver-kubelet-client + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md index d45c3ca73a6be..92b62d83d0168 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_apiserver.md @@ -1,12 +1,15 @@ -Generates the certificate for serving the Kubernetes API +Renew the certificate for serving the Kubernetes API ### Synopsis +Renew the certificate for serving the Kubernetes API. -Renews the certificate for serving the Kubernetes API, and saves them into apiserver.cert and apiserver.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew apiserver [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew apiserver [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for apiserver
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for apiserver + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_controller-manager.conf.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_controller-manager.conf.md new file mode 100644 index 0000000000000..7339fc04b8629 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_controller-manager.conf.md @@ -0,0 +1,35 @@ + +Renew the certificate embedded in the kubeconfig file for the controller manager to use + +### Synopsis + +Renew the certificate embedded in the kubeconfig file for the controller manager to use. + +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. + +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. + +``` +kubeadm alpha certs renew controller-manager.conf [flags] +``` + +### Options + +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for controller-manager.conf + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md index 3c798047f3975..f1b796a08b56b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md @@ -1,12 +1,15 @@ -Generates the client certificate for liveness probes to healtcheck etcd +Renew the certificate for liveness probes to healtcheck etcd ### Synopsis +Renew the certificate for liveness probes to healtcheck etcd. -Renews the client certificate for liveness probes to healtcheck etcd, and saves them into etcd/healthcheck-client.cert and etcd/healthcheck-client.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew etcd-healthcheck-client [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew etcd-healthcheck-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for etcd-healthcheck-client
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for etcd-healthcheck-client + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md index b8cf86bad1444..e5c46d5245c5b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-peer.md @@ -1,12 +1,15 @@ -Generates the credentials for etcd nodes to communicate with each other +Renew the certificate for etcd nodes to communicate with each other ### Synopsis +Renew the certificate for etcd nodes to communicate with each other. -Renews the credentials for etcd nodes to communicate with each other, and saves them into etcd/peer.cert and etcd/peer.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew etcd-peer [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew etcd-peer [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for etcd-peer
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for etcd-peer + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md index 123111cd5557a..997a607bd8dcc 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_etcd-server.md @@ -1,12 +1,15 @@ -Generates the certificate for serving etcd +Renew the certificate for serving etcd ### Synopsis +Renew the certificate for serving etcd. -Renews the certificate for serving etcd, and saves them into etcd/server.cert and etcd/server.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew etcd-server [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew etcd-server [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for etcd-server
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for etcd-server + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md index ed9f18495a7ea..2945b45b2f733 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_front-proxy-client.md @@ -1,12 +1,15 @@ -Generates the client for the front proxy +Renew the certificate for the front proxy client ### Synopsis +Renew the certificate for the front proxy client. -Renews the client for the front proxy, and saves them into front-proxy-client.cert and front-proxy-client.key files. +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. -Extra attributes such as SANs will be based on the existing certificates, there is no need to resupply them. +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. ``` kubeadm alpha certs renew front-proxy-client [flags] @@ -14,85 +17,19 @@ kubeadm alpha certs renew front-proxy-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for front-proxy-client
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--use-api
Use the Kubernetes certificate API to renew certificates
- - +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for front-proxy-client + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_scheduler.conf.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_scheduler.conf.md new file mode 100644 index 0000000000000..58dc3900a95c1 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_certs_renew_scheduler.conf.md @@ -0,0 +1,35 @@ + +Renew the certificate embedded in the kubeconfig file for the scheduler manager to use + +### Synopsis + +Renew the certificate embedded in the kubeconfig file for the scheduler manager to use. + +Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them. + +Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request. + +After renewal, in order to make changes effective, is is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere. + +``` +kubeadm alpha certs renew scheduler.conf [flags] +``` + +### Options + +``` + --cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for scheduler.conf + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --use-api Use the Kubernetes certificate API to renew certificates +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md index 24b0f4d901330..0481bb1d2a8ff 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig.md @@ -3,50 +3,19 @@ Kubeconfig file utilities ### Synopsis - Kubeconfig file utilities. Alpha Disclaimer: this command is currently alpha. ### Options - - - - - - - - - - - - - - - -
-h, --help
help for kubeconfig
- - +``` + -h, --help help for kubeconfig +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md index c12f50d3cf718..5bcc47139c2ef 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubeconfig_user.md @@ -1,10 +1,9 @@ -Outputs a kubeconfig file for an additional user +Output a kubeconfig file for an additional user ### Synopsis - -Outputs a kubeconfig file for an additional user. +Output a kubeconfig file for an additional user. Alpha Disclaimer: this command is currently alpha. @@ -15,91 +14,25 @@ kubeadm alpha kubeconfig user [flags] ### Examples ``` - # Outputs a kubeconfig file for an additional user named foo + # Output a kubeconfig file for an additional user named foo kubeadm alpha kubeconfig user --client-name=foo ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API server is accessible on
--apiserver-bind-port int32     Default: 6443
The port the API server is accessible on
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are stored
--client-name string
The name of user. It will be used as the CN if client certificates are created
-h, --help
help for user
--org stringSlice
The orgnizations of the client certificate. It will be used as the O if client certificates are created
--token string
The token that should be used as the authentication mechanism for this kubeconfig, instead of client certificates
- - +``` + --apiserver-advertise-address string The IP address the API server is accessible on + --apiserver-bind-port int32 The port the API server is accessible on (default 6443) + --cert-dir string The path where certificates are stored (default "/etc/kubernetes/pki") + --client-name string The name of user. It will be used as the CN if client certificates are created + -h, --help help for user + --org strings The orgnizations of the client certificate. It will be used as the O if client certificates are created + --token string The token that should be used as the authentication mechanism for this kubeconfig, instead of client certificates +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md index 5479cf594e084..d975aae9633c3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet.md @@ -3,48 +3,17 @@ Commands related to handling the kubelet ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ### Options - - - - - - - - - - - - - - - -
-h, --help
help for kubelet
- - +``` + -h, --help help for kubelet +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md index be9d4fcc26935..0040242ff2a8c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config.md @@ -3,48 +3,17 @@ Utilities for kubelet configuration ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ### Options - - - - - - - - - - - - - - - -
-h, --help
help for config
- - +``` + -h, --help help for config +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md index 20ad0bbf4c53c..ca6e808504e33 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_download.md @@ -1,10 +1,9 @@ -Downloads the kubelet configuration from the cluster ConfigMap kubelet-config-1.X, where X is the minor version of the kubelet. +Download the kubelet configuration from the cluster ConfigMap kubelet-config-1.X, where X is the minor version of the kubelet ### Synopsis - -Downloads the kubelet configuration from a ConfigMap of the form "kubelet-config-1.X" in the cluster, where X is the minor version of the kubelet. Either kubeadm autodetects the kubelet version by exec-ing "kubelet --version" or respects the --kubelet-version parameter. +Download the kubelet configuration from a ConfigMap of the form "kubelet-config-1.X" in the cluster, where X is the minor version of the kubelet. Either kubeadm autodetects the kubelet version by exec-ing "kubelet --version" or respects the --kubelet-version parameter. Alpha Disclaimer: this command is currently alpha. @@ -15,66 +14,24 @@ kubeadm alpha kubelet config download [flags] ### Examples ``` - # Downloads the kubelet configuration from the ConfigMap in the cluster. Autodetects the kubelet version. + # Download the kubelet configuration from the ConfigMap in the cluster. Autodetect the kubelet version. kubeadm alpha phase kubelet config download - # Downloads the kubelet configuration from the ConfigMap in the cluster. Uses a specific desired kubelet version. - kubeadm alpha phase kubelet config download --kubelet-version v1.12.0 + # Download the kubelet configuration from the ConfigMap in the cluster. Use a specific desired kubelet version. + kubeadm alpha phase kubelet config download --kubelet-version 1.14.0 ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-h, --help
help for download
--kubeconfig string     Default: "/etc/kubernetes/kubelet.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubelet-version string
The desired version for the kubelet. Defaults to being autodetected from 'kubelet --version'.
- - +``` + -h, --help help for download + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubelet-version string The desired version for the kubelet. Defaults to being autodetected from 'kubelet --version'. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md index fdad69928b9a0..3d772d05b97d6 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_kubelet_config_enable-dynamic.md @@ -1,10 +1,9 @@ -EXPERIMENTAL: Enables or updates dynamic kubelet configuration for a Node +EXPERIMENTAL: Enable or update dynamic kubelet configuration for a Node ### Synopsis - -Enables or updates dynamic kubelet configuration for a Node, against the kubelet-config-1.X ConfigMap in the cluster, where X is the minor version of the desired kubelet version. +Enable or update dynamic kubelet configuration for a Node, against the kubelet-config-1.X ConfigMap in the cluster, where X is the minor version of the desired kubelet version. WARNING: This feature is still experimental, and disabled by default. Enable only if you know what you are doing, as it may have surprising side-effects at this stage. @@ -17,8 +16,8 @@ kubeadm alpha kubelet config enable-dynamic [flags] ### Examples ``` - # Enables dynamic kubelet configuration for a Node. - kubeadm alpha phase kubelet enable-dynamic-config --node-name node-1 --kubelet-version v1.12.0 + # Enable dynamic kubelet configuration for a Node. + kubeadm alpha phase kubelet enable-dynamic-config --node-name node-1 --kubelet-version 1.14.0 WARNING: This feature is still experimental, and disabled by default. Enable only if you know what you are doing, as it may have surprising side-effects at this stage. @@ -26,64 +25,16 @@ kubeadm alpha kubelet config enable-dynamic [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-h, --help
help for enable-dynamic
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubelet-version string
The desired version for the kubelet
--node-name string
Name of the node that should enable the dynamic kubelet configuration
- - +``` + -h, --help help for enable-dynamic + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubelet-version string The desired version for the kubelet + --node-name string Name of the node that should enable the dynamic kubelet configuration +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting.md index a4543c9ee780f..95951d4bbef03 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting.md @@ -1,50 +1,19 @@ -Makes a kubeadm cluster self-hosted +Make a kubeadm cluster self-hosted ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ### Options - - - - - - - - - - - - - - - -
-h, --help
help for selfhosting
- - +``` + -h, --help help for selfhosting +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md index 4f04e48126a6b..eaa946f2c1465 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_alpha_selfhosting_pivot.md @@ -1,10 +1,9 @@ -Converts a static Pod-hosted control plane into a self-hosted one +Convert a static Pod-hosted control plane into a self-hosted one ### Synopsis - -Converts static Pod files for control plane components into self-hosted DaemonSets configured via the Kubernetes API. +Convert static Pod files for control plane components into self-hosted DaemonSets configured via the Kubernetes API. See the documentation for self-hosting limitations. @@ -17,85 +16,25 @@ kubeadm alpha selfhosting pivot [flags] ### Examples ``` - # Converts a static Pod-hosted control plane into a self-hosted one. + # Convert a static Pod-hosted control plane into a self-hosted one. kubeadm alpha phase self-hosting convert-from-staticpods ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where certificates are stored
--config string
Path to a kubeadm config file. WARNING: Usage of a configuration file is experimental
-f, --force
Pivot the cluster without prompting for confirmation
-h, --help
help for pivot
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
-s, --store-certs-in-secrets
Enable storing certs in secrets
- - +``` + --cert-dir string The path where certificates are stored (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -f, --force Pivot the cluster without prompting for confirmation + -h, --help help for pivot + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + -s, --store-certs-in-secrets Enable storing certs in secrets +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_completion.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_completion.md index 37481780b1652..4618befd9bd23 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_completion.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_completion.md @@ -1,10 +1,9 @@ -Output shell completion code for the specified shell (bash or zsh). +Output shell completion code for the specified shell (bash or zsh) ### Synopsis - Output shell completion code for the specified shell (bash or zsh). The shell code must be evaluated to provide interactive completion of kubeadm commands. This can be done by sourcing it from @@ -39,7 +38,7 @@ source $HOME/.bash_profile # Load the kubeadm completion code for bash into the current shell source <(kubeadm completion bash) -# Write bash completion code to a file and source if from .bash_profile +# Write bash completion code to a file and source it from .bash_profile kubeadm completion bash > ~/.kube/kubeadm_completion.bash.inc printf "\n# Kubeadm shell completion\nsource '$HOME/.kube/kubeadm_completion.bash.inc'\n" >> $HOME/.bash_profile source $HOME/.bash_profile @@ -50,43 +49,13 @@ source <(kubeadm completion zsh) ### Options - - - - - - - - - - - - - - - -
-h, --help
help for completion
- - +``` + -h, --help help for completion +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md index 6c615daacd689..bc085de08a9e7 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config.md @@ -1,10 +1,9 @@ -Manage configuration for a kubeadm cluster persisted in a ConfigMap in the cluster. +Manage configuration for a kubeadm cluster persisted in a ConfigMap in the cluster ### Synopsis - There is a ConfigMap in the kube-system namespace called "kubeadm-config" that kubeadm uses to store internal configuration about the cluster. kubeadm CLI v1.8.0+ automatically creates this ConfigMap with the config used with 'kubeadm init', but if you initialized your cluster using kubeadm v1.7.x or lower, you must use the 'config upload' command to create this @@ -17,50 +16,14 @@ kubeadm config [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - -
-h, --help
help for config
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - +``` + -h, --help help for config + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md index 4b1073d17b69d..6c9be5ac3cb4b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images.md @@ -1,10 +1,9 @@ -Interact with container images used by kubeadm. +Interact with container images used by kubeadm ### Synopsis - -Interact with container images used by kubeadm. +Interact with container images used by kubeadm ``` kubeadm config images [flags] @@ -12,50 +11,14 @@ kubeadm config images [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for images
- - +``` + -h, --help help for images +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md index fac2d052f2a47..28a2f0a7dcb39 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_list.md @@ -1,10 +1,9 @@ -Print a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customized. +Print a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customized ### Synopsis - -Print a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customized. +Print a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customized ``` kubeadm config images list [flags] @@ -12,71 +11,18 @@ kubeadm config images list [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for list
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
- - +``` + --config string Path to kubeadm config file. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for list + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md index 7cb2173a265b1..611524df9c2b8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_images_pull.md @@ -1,10 +1,9 @@ -Pull images used by kubeadm. +Pull images used by kubeadm ### Synopsis - -Pull images used by kubeadm. +Pull images used by kubeadm ``` kubeadm config images pull [flags] @@ -12,78 +11,19 @@ kubeadm config images pull [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file.
--cri-socket string     Default: "/var/run/dockershim.sock"
Specify the CRI socket to connect to.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for pull
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
- - +``` + --config string Path to kubeadm config file. + --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for pull + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md index 11bf770de8cf0..ba1361453ee7b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_migrate.md @@ -1,13 +1,13 @@ -Read an older version of the kubeadm configuration API types from a file, and output the similar config object for the newer version. +Read an older version of the kubeadm configuration API types from a file, and output the similar config object for the newer version ### Synopsis - This command lets you convert configuration objects of older versions to the latest supported version, locally in the CLI tool without ever touching anything in the cluster. In this version of kubeadm, the following API versions are supported: + - kubeadm.k8s.io/v1beta1 - kubeadm.k8s.io/v1beta2 @@ -26,64 +26,16 @@ kubeadm config migrate [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-h, --help
help for migrate
--new-config string
Path to the resulting equivalent kubeadm config file using the new API version. Optional, if not specified output will be sent to STDOUT.
--old-config string
Path to the kubeadm config file that is using an old API version and should be converted. This flag is mandatory.
- - +``` + -h, --help help for migrate + --new-config string Path to the resulting equivalent kubeadm config file using the new API version. Optional, if not specified output will be sent to STDOUT. + --old-config string Path to the kubeadm config file that is using an old API version and should be converted. This flag is mandatory. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md index ea98b1fa30b67..159b87581dca7 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print.md @@ -3,7 +3,6 @@ Print configuration ### Synopsis - This command prints configurations for subcommands provided. ``` @@ -12,50 +11,14 @@ kubeadm config print [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for print
- - +``` + -h, --help help for print +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md index afdc7c3d4efe9..239a969a827f8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_init-defaults.md @@ -4,7 +4,6 @@ Print default init configuration, that can be used for 'kubeadm init' ### Synopsis - This command prints objects such as the default init configuration that is used for 'kubeadm init'. Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like {"abcdef.0123456789abcdef" "" "nil" [] []} in order to pass validation but @@ -17,57 +16,15 @@ kubeadm config print init-defaults [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - -
--component-configs stringSlice
A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed.
-h, --help
help for init-defaults
- - +``` + --component-configs strings A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed. + -h, --help help for init-defaults +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md index d26a83dc92fda..57d5024988964 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_print_join-defaults.md @@ -4,7 +4,6 @@ Print default join configuration, that can be used for 'kubeadm join' ### Synopsis - This command prints objects such as the default join configuration that is used for 'kubeadm join'. Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like {"abcdef.0123456789abcdef" "" "nil" [] []} in order to pass validation but @@ -17,57 +16,15 @@ kubeadm config print join-defaults [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - -
--component-configs stringSlice
A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed.
-h, --help
help for join-defaults
- - +``` + --component-configs strings A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed. + -h, --help help for join-defaults +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md deleted file mode 100644 index b4e825099a6ef..0000000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload.md +++ /dev/null @@ -1,61 +0,0 @@ - -Upload configuration about the current state, so that 'kubeadm upgrade' can later know how to configure the upgraded cluster. - -### Synopsis - - -Upload configuration about the current state, so that 'kubeadm upgrade' can later know how to configure the upgraded cluster. - -``` -kubeadm config upload [flags] -``` - -### Options - - - - - - - - - - - - - - - - -
-h, --help
help for upload
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md deleted file mode 100644 index 51d7987479234..0000000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-file.md +++ /dev/null @@ -1,74 +0,0 @@ - -Upload a configuration file to the in-cluster ConfigMap for kubeadm configuration. - -### Synopsis - - - -Using this command, you can upload configuration to the ConfigMap in the cluster using the same config file you gave to 'kubeadm init'. -If you initialized your cluster using a v1.7.x or lower kubeadm client and used the --config option, you need to run this command with the -same config file before upgrading to v1.8 using 'kubeadm upgrade'. - -The configuration is located in the "kube-system" namespace in the "kubeadm-config" ConfigMap. - - -``` -kubeadm config upload from-file [flags] -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to a kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for from-file
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md deleted file mode 100644 index 08c1adb2f1d9c..0000000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_upload_from-flags.md +++ /dev/null @@ -1,144 +0,0 @@ - -Create the in-cluster configuration file for the first time from using flags. - -### Synopsis - - - -Using this command, you can upload configuration to the ConfigMap in the cluster using the same flags you gave to 'kubeadm init'. -If you initialized your cluster using a v1.7.x or lower kubeadm client and set certain flags, you need to run this command with the -same flags before upgrading to v1.8 using 'kubeadm upgrade'. - -The configuration is located in the "kube-system" namespace in the "kubeadm-config" ConfigMap. - - -``` -kubeadm config upload from-flags [flags] -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--apiserver-cert-extra-sans stringSlice
Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--cri-socket string     Default: "/var/run/dockershim.sock"
Specify the CRI socket to connect to.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for from-flags
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--node-name string
Specify the node name.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md index a389c5d947e0e..a311347b32b6f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_config_view.md @@ -1,10 +1,9 @@ -View the kubeadm configuration stored inside the cluster. +View the kubeadm configuration stored inside the cluster ### Synopsis - Using this command, you can view the ConfigMap in the cluster where the configuration for kubeadm is located. The configuration is located in the "kube-system" namespace in the "kubeadm-config" ConfigMap. @@ -16,50 +15,14 @@ kubeadm config view [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for view
- - +``` + -h, --help help for view +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - -
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md index ebd50c07e22a5..c7faf7064008d 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init.md @@ -1,43 +1,46 @@ -Run this command in order to set up the Kubernetes control plane. +Run this command in order to set up the Kubernetes control plane ### Synopsis -The `init` command executes the following phases: +Run this command in order to set up the Kubernetes control plane + +The "init" command executes the following phases: ``` -preflight Run master pre-flight checks -kubelet-start Writes kubelet settings and (re)starts the kubelet +preflight Run pre-flight checks +kubelet-start Write kubelet settings and (re)start the kubelet certs Certificate generation - /ca Generates the self-signed Kubernetes CA to provision identities for other Kubernetes components - /apiserver Generates the certificate for serving the Kubernetes API - /apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet - /front-proxy-ca Generates the self-signed CA to provision identities for front proxy - /front-proxy-client Generates the client for the front proxy - /etcd-ca Generates the self-signed CA to provision identities for etcd - /etcd-server Generates the certificate for serving etcd - /etcd-peer Generates the credentials for etcd nodes to communicate with each other - /etcd-healthcheck-client Generates the client certificate for liveness probes to healtcheck etcd - /apiserver-etcd-client Generates the client apiserver uses to access etcd - /sa Generates a private key for signing service account tokens along with its public key -kubeconfig Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file - /admin Generates a kubeconfig file for the admin to use and for kubeadm itself - /kubelet Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes - /controller-manager Generates a kubeconfig file for the controller manager to use - /scheduler Generates a kubeconfig file for the scheduler to use -control-plane Generates all static Pod manifest files necessary to establish the control plane + /etcd-ca Generate the self-signed CA to provision identities for etcd + /apiserver-etcd-client Generate the certificate the apiserver uses to access etcd + /etcd-healthcheck-client Generate the certificate for liveness probes to healtcheck etcd + /etcd-server Generate the certificate for serving etcd + /etcd-peer Generate the certificate for etcd nodes to communicate with each other + /ca Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components + /apiserver Generate the certificate for serving the Kubernetes API + /apiserver-kubelet-client Generate the certificate for the API server to connect to kubelet + /front-proxy-ca Generate the self-signed CA to provision identities for front proxy + /front-proxy-client Generate the certificate for the front proxy client + /sa Generate a private key for signing service account tokens along with its public key +kubeconfig Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file + /admin Generate a kubeconfig file for the admin to use and for kubeadm itself + /kubelet Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes + /controller-manager Generate a kubeconfig file for the controller manager to use + /scheduler Generate a kubeconfig file for the scheduler to use +control-plane Generate all static Pod manifest files necessary to establish the control plane /apiserver Generates the kube-apiserver static Pod manifest /controller-manager Generates the kube-controller-manager static Pod manifest /scheduler Generates the kube-scheduler static Pod manifest -etcd Generates static Pod manifest file for local etcd. - /local Generates the static Pod manifest file for a local, single-node local etcd instance. -upload-config Uploads the kubeadm and kubelet configuration to a ConfigMap - /kubeadm Uploads the kubeadm ClusterConfiguration to a ConfigMap - /kubelet Uploads the kubelet component config to a ConfigMap +etcd Generate static Pod manifest file for local etcd + /local Generate the static Pod manifest file for a local, single-node local etcd instance +upload-config Upload the kubeadm and kubelet configuration to a ConfigMap + /kubeadm Upload the kubeadm ClusterConfiguration to a ConfigMap + /kubelet Upload the kubelet component config to a ConfigMap +upload-certs Upload certificates to kubeadm-certs mark-control-plane Mark a node as a control-plane bootstrap-token Generates bootstrap tokens used to join a node to a cluster -addon Installs required addons for passing Conformance tests - /coredns Installs the CoreDNS addon to a Kubernetes cluster - /kube-proxy Installs the kube-proxy addon to a Kubernetes cluster +addon Install required addons for passing Conformance tests + /coredns Install the CoreDNS addon to a Kubernetes cluster + /kube-proxy Install the kube-proxy addon to a Kubernetes cluster ``` @@ -47,176 +50,35 @@ kubeadm init [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--apiserver-cert-extra-sans stringSlice
Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--cri-socket string     Default: "/var/run/dockershim.sock"
Specify the CRI socket to connect to.
--dry-run
Don't apply any changes; just output what would be done.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for init
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--node-name string
Specify the node name.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
--skip-phases stringSlice
List of phases to be skipped
--skip-token-print
Skip printing of the default bootstrap token generated by 'kubeadm init'.
--token string
The token to use for establishing bidirectional trust between nodes and masters. The format is [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
--token-ttl duration     Default: 24h0m0s
The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --apiserver-cert-extra-sans strings Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names. + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --certificate-key string Key used to encrypt the control-plane certificates in the kubeadm-certs Secret. + --config string Path to a kubeadm configuration file. + --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. + --dry-run Don't apply any changes; just output what would be done. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for init + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --node-name string Specify the node name. + --pod-network-cidr string Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node. + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") + --service-dns-domain string Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local") + --skip-certificate-key-print Don't print the key used to encrypt the control-plane certificates. + --skip-phases strings List of phases to be skipped + --skip-token-print Skip printing of the default bootstrap token generated by 'kubeadm init'. + --token string The token to use for establishing bidirectional trust between nodes and control-plane nodes. The format is [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef + --token-ttl duration The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire (default 24h0m0s) + --upload-certs Upload control-plane certificates to the kubeadm-certs Secret. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md index b2946caf1fa02..c469017dfd905 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase.md @@ -1,50 +1,19 @@ -use this command to invoke single phase of the init workflow +Use this command to invoke single phase of the init workflow ### Synopsis - -use this command to invoke single phase of the init workflow +Use this command to invoke single phase of the init workflow ### Options - - - - - - - - - - - - - - - -
-h, --help
help for phase
- - +``` + -h, --help help for phase +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md index 1ddc02f451112..53aea81144903 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon.md @@ -1,9 +1,8 @@ -Installs required addons for passing Conformance tests +Install required addons for passing Conformance tests ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm init phase addon [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for addon
- - +``` + -h, --help help for addon +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md index 76e1055467042..f27dd956996bb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_all.md @@ -1,10 +1,9 @@ -Installs all the addons +Install all the addons ### Synopsis - -Installs all the addons +Install all the addons ``` kubeadm init phase addon all [flags] @@ -12,113 +11,23 @@ kubeadm init phase addon all [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for all
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --config string Path to a kubeadm configuration file. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for all + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --pod-network-cidr string Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node. + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") + --service-dns-domain string Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md index a358676d53070..1e7c742f97931 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_coredns.md @@ -1,10 +1,9 @@ -Installs the CoreDNS addon to a Kubernetes cluster +Install the CoreDNS addon to a Kubernetes cluster ### Synopsis - -Installs the CoreDNS addon components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed. +Install the CoreDNS addon components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed. ``` kubeadm init phase addon coredns [flags] @@ -12,92 +11,20 @@ kubeadm init phase addon coredns [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for coredns
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
- - +``` + --config string Path to a kubeadm configuration file. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for coredns + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") + --service-dns-domain string Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md index f967ae1647d0e..2af54fcf9d0fa 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_addon_kube-proxy.md @@ -1,10 +1,9 @@ -Installs the kube-proxy addon to a Kubernetes cluster +Install the kube-proxy addon to a Kubernetes cluster ### Synopsis - -Installs the kube-proxy addon components via the API server. +Install the kube-proxy addon components via the API server. ``` kubeadm init phase addon kube-proxy [flags] @@ -12,92 +11,20 @@ kubeadm init phase addon kube-proxy [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for kube-proxy
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --config string Path to a kubeadm configuration file. + -h, --help help for kube-proxy + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --pod-network-cidr string Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md index c853d7d2fb37e..da037aa1248e3 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_bootstrap-token.md @@ -3,7 +3,6 @@ Generates bootstrap tokens used to join a node to a cluster ### Synopsis - Bootstrap tokens are used for establishing bidirectional trust between a node joining the cluster and a the control-plane node. This command makes all the configurations required to make bootstrap tokens works and then creates an initial token. @@ -15,71 +14,23 @@ kubeadm init phase bootstrap-token [flags] ### Examples ``` - # Makes all the bootstrap token configurations and creates an initial token, functionally + # Make all the bootstrap token configurations and create an initial token, functionally # equivalent to what generated by kubeadm init. kubeadm init phase bootstrap-token ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for bootstrap-token
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
--skip-token-print
Skip printing of the default bootstrap token generated by 'kubeadm init'.
- - +``` + --config string Path to a kubeadm configuration file. + -h, --help help for bootstrap-token + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --skip-token-print Skip printing of the default bootstrap token generated by 'kubeadm init'. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md index 538a7a1b33934..0c7c04d9ef3b1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs.md @@ -3,7 +3,6 @@ Certificate generation ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm init phase certs [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for certs
- - +``` + -h, --help help for certs +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md index 910b94afbeda5..6798a92d1ee04 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_all.md @@ -1,10 +1,9 @@ -Generates all certificates +Generate all certificates ### Synopsis - -Generates all certificates +Generate all certificates ``` kubeadm init phase certs all [flags] @@ -12,85 +11,19 @@ kubeadm init phase certs all [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-cert-extra-sans stringSlice
Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for all
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-cert-extra-sans strings Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names. + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for all + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") + --service-dns-domain string Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md index 01e798e144127..03af7712c04e7 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-etcd-client.md @@ -1,10 +1,9 @@ -Generates the client apiserver uses to access etcd +Generate the certificate the apiserver uses to access etcd ### Synopsis - -Generates the client apiserver uses to access etcd, and saves them into apiserver-etcd-client.cert and apiserver-etcd-client.key files. +Generate the certificate the apiserver uses to access etcd, and save them into apiserver-etcd-client.cert and apiserver-etcd-client.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,71 +15,17 @@ kubeadm init phase certs apiserver-etcd-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for apiserver-etcd-client
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for apiserver-etcd-client +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md index b3e30655c4649..3ef086552157e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver-kubelet-client.md @@ -1,10 +1,9 @@ -Generates the Client certificate for the API server to connect to kubelet +Generate the certificate for the API server to connect to kubelet ### Synopsis - -Generates the Client certificate for the API server to connect to kubelet, and saves them into apiserver-kubelet-client.cert and apiserver-kubelet-client.key files. +Generate the certificate for the API server to connect to kubelet, and save them into apiserver-kubelet-client.cert and apiserver-kubelet-client.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,71 +15,17 @@ kubeadm init phase certs apiserver-kubelet-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for apiserver-kubelet-client
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for apiserver-kubelet-client +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md index a36c16aee3da7..9562ffd8fedaa 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_apiserver.md @@ -1,10 +1,9 @@ -Generates the certificate for serving the Kubernetes API +Generate the certificate for serving the Kubernetes API ### Synopsis - -Generates the certificate for serving the Kubernetes API, and saves them into apiserver.cert and apiserver.key files. +Generate the certificate for serving the Kubernetes API, and save them into apiserver.cert and apiserver.key files. Default SANs are kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, 10.96.0.1, 127.0.0.1 @@ -18,99 +17,21 @@ kubeadm init phase certs apiserver [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-cert-extra-sans stringSlice
Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for apiserver
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-cert-extra-sans strings Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names. + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for apiserver + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") + --service-dns-domain string Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md index 257da0ea6a1cd..cc5d479e7bddb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_ca.md @@ -1,10 +1,9 @@ -Generates the self-signed Kubernetes CA to provision identities for other Kubernetes components +Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components ### Synopsis - -Generates the self-signed Kubernetes CA to provision identities for other Kubernetes components, and saves them into ca.cert and ca.key files. +Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components, and save them into ca.cert and ca.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,57 +15,15 @@ kubeadm init phase certs ca [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for ca
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for ca +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md index 210b0d1dd2dca..48345f6b01fa2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-ca.md @@ -1,10 +1,9 @@ -Generates the self-signed CA to provision identities for etcd +Generate the self-signed CA to provision identities for etcd ### Synopsis - -Generates the self-signed CA to provision identities for etcd, and saves them into etcd/ca.cert and etcd/ca.key files. +Generate the self-signed CA to provision identities for etcd, and save them into etcd/ca.cert and etcd/ca.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,57 +15,15 @@ kubeadm init phase certs etcd-ca [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for etcd-ca
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for etcd-ca +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md index 65e5c41bedfa7..c4428459b747b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-healthcheck-client.md @@ -1,10 +1,9 @@ -Generates the client certificate for liveness probes to healtcheck etcd +Generate the certificate for liveness probes to healtcheck etcd ### Synopsis - -Generates the client certificate for liveness probes to healtcheck etcd, and saves them into etcd/healthcheck-client.cert and etcd/healthcheck-client.key files. +Generate the certificate for liveness probes to healtcheck etcd, and save them into etcd/healthcheck-client.cert and etcd/healthcheck-client.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,71 +15,17 @@ kubeadm init phase certs etcd-healthcheck-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for etcd-healthcheck-client
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for etcd-healthcheck-client +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md index 2e045ae1e1ac7..fd9f355a6abe4 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-peer.md @@ -1,10 +1,9 @@ -Generates the credentials for etcd nodes to communicate with each other +Generate the certificate for etcd nodes to communicate with each other ### Synopsis - -Generates the credentials for etcd nodes to communicate with each other, and saves them into etcd/peer.cert and etcd/peer.key files. +Generate the certificate for etcd nodes to communicate with each other, and save them into etcd/peer.cert and etcd/peer.key files. Default SANs are localhost, 127.0.0.1, 127.0.0.1, ::1 @@ -18,71 +17,17 @@ kubeadm init phase certs etcd-peer [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for etcd-peer
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for etcd-peer +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md index a960d01e4f77b..19489e9f16cdc 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_etcd-server.md @@ -1,10 +1,9 @@ -Generates the certificate for serving etcd +Generate the certificate for serving etcd ### Synopsis - -Generates the certificate for serving etcd, and saves them into etcd/server.cert and etcd/server.key files. +Generate the certificate for serving etcd, and save them into etcd/server.cert and etcd/server.key files. Default SANs are localhost, 127.0.0.1, 127.0.0.1, ::1 @@ -18,71 +17,17 @@ kubeadm init phase certs etcd-server [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for etcd-server
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for etcd-server +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md index bff25d78cd7d7..dcf4e16da259d 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-ca.md @@ -1,10 +1,9 @@ -Generates the self-signed CA to provision identities for front proxy +Generate the self-signed CA to provision identities for front proxy ### Synopsis - -Generates the self-signed CA to provision identities for front proxy, and saves them into front-proxy-ca.cert and front-proxy-ca.key files. +Generate the self-signed CA to provision identities for front proxy, and save them into front-proxy-ca.cert and front-proxy-ca.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,57 +15,15 @@ kubeadm init phase certs front-proxy-ca [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for front-proxy-ca
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for front-proxy-ca +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md index 9a051340a1c95..8594356941b57 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_front-proxy-client.md @@ -1,10 +1,9 @@ -Generates the client for the front proxy +Generate the certificate for the front proxy client ### Synopsis - -Generates the client for the front proxy, and saves them into front-proxy-client.cert and front-proxy-client.key files. +Generate the certificate for the front proxy client, and save them into front-proxy-client.cert and front-proxy-client.key files. If both files already exist, kubeadm skips the generation step and existing files will be used. @@ -16,71 +15,17 @@ kubeadm init phase certs front-proxy-client [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--csr-dir string
The path to output the CSRs and private keys to
--csr-only
Create CSRs instead of generating certificates
-h, --help
help for front-proxy-client
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --csr-dir string The path to output the CSRs and private keys to + --csr-only Create CSRs instead of generating certificates + -h, --help help for front-proxy-client +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md index 951b773eac6e0..d820c1db4a9a1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_certs_sa.md @@ -1,10 +1,9 @@ -Generates a private key for signing service account tokens along with its public key +Generate a private key for signing service account tokens along with its public key ### Synopsis - -Generates the private key for signing service account tokens along with its public key, and saves them into sa.key and sa.pub files. If both files already exist, kubeadm skips the generation step and existing files will be used. +Generate the private key for signing service account tokens along with its public key, and save them into sa.key and sa.pub files. If both files already exist, kubeadm skips the generation step and existing files will be used. Alpha Disclaimer: this command is currently alpha. @@ -14,43 +13,14 @@ kubeadm init phase certs sa [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for sa
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + -h, --help help for sa +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md index 38cc40ed905ca..e3b47df73fd31 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane.md @@ -1,9 +1,8 @@ -Generates all static Pod manifest files necessary to establish the control plane +Generate all static Pod manifest files necessary to establish the control plane ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm init phase control-plane [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for control-plane
- - +``` + -h, --help help for control-plane +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md index 322a54531818e..d5a229d47e4cb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_all.md @@ -1,138 +1,46 @@ -Generates all static Pod manifest files +Generate all static Pod manifest files ### Synopsis - -Generates all static Pod manifest files +Generate all static Pod manifest files ``` kubeadm init phase control-plane all [flags] ``` -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +### Examples - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--apiserver-extra-args mapStringString
A set of extra flags to pass to the API Server or override default ones in form of <flagname>=<value>
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--controller-manager-extra-args mapStringString
A set of extra flags to pass to the Controller Manager or override default ones in form of <flagname>=<value>
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for all
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--scheduler-extra-args mapStringString
A set of extra flags to pass to the Scheduler or override default ones in form of <flagname>=<value>
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
+``` + # Generates all static Pod manifest files for control plane components, + # functionally equivalent to what is generated by kubeadm init. + kubeadm init phase control-plane all + + # Generates all static Pod manifest files using options read from a configuration file. + kubeadm init phase control-plane all --config config.yaml +``` +### Options +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --apiserver-extra-args mapStringString A set of extra flags to pass to the API Server or override default ones in form of = + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --controller-manager-extra-args mapStringString A set of extra flags to pass to the Controller Manager or override default ones in form of = + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for all + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --pod-network-cidr string Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node. + --scheduler-extra-args mapStringString A set of extra flags to pass to the Scheduler or override default ones in form of = + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md index 6ebc0d2724758..a73f6979debb4 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_apiserver.md @@ -3,7 +3,6 @@ Generates the kube-apiserver static Pod manifest ### Synopsis - Generates the kube-apiserver static Pod manifest ``` @@ -12,106 +11,22 @@ kubeadm init phase control-plane apiserver [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--apiserver-extra-args mapStringString
A set of extra flags to pass to the API Server or override default ones in form of <flagname>=<value>
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for apiserver
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --apiserver-extra-args mapStringString A set of extra flags to pass to the API Server or override default ones in form of = + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for apiserver + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --service-cidr string Use alternative range of IP address for service VIPs. (default "10.96.0.0/12") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md index 19fb762a7e786..a12e82f3c96ea 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_controller-manager.md @@ -3,7 +3,6 @@ Generates the kube-controller-manager static Pod manifest ### Synopsis - Generates the kube-controller-manager static Pod manifest ``` @@ -12,85 +11,19 @@ kubeadm init phase control-plane controller-manager [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--controller-manager-extra-args mapStringString
A set of extra flags to pass to the Controller Manager or override default ones in form of <flagname>=<value>
-h, --help
help for controller-manager
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + --controller-manager-extra-args mapStringString A set of extra flags to pass to the Controller Manager or override default ones in form of = + -h, --help help for controller-manager + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --pod-network-cidr string Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md index 9cf579ab56e93..13219cef3662c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_control-plane_scheduler.md @@ -3,7 +3,6 @@ Generates the kube-scheduler static Pod manifest ### Synopsis - Generates the kube-scheduler static Pod manifest ``` @@ -12,78 +11,18 @@ kubeadm init phase control-plane scheduler [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for scheduler
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--scheduler-extra-args mapStringString
A set of extra flags to pass to the Scheduler or override default ones in form of <flagname>=<value>
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for scheduler + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") + --kubernetes-version string Choose a specific Kubernetes version for the control plane. (default "stable-1") + --scheduler-extra-args mapStringString A set of extra flags to pass to the Scheduler or override default ones in form of = +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md index fa2a91f055085..e6c93bc7e2ecc 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd.md @@ -1,9 +1,8 @@ -Generates static Pod manifest file for local etcd. +Generate static Pod manifest file for local etcd ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm init phase etcd [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for etcd
- - +``` + -h, --help help for etcd +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md index ced1517a3b812..c07f3d39ca345 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_etcd_local.md @@ -1,10 +1,9 @@ -Generates the static Pod manifest file for a local, single-node local etcd instance. +Generate the static Pod manifest file for a local, single-node local etcd instance ### Synopsis - -Generates the static Pod manifest file for a local, single-node local etcd instance. +Generate the static Pod manifest file for a local, single-node local etcd instance ``` kubeadm init phase etcd local [flags] @@ -24,64 +23,16 @@ kubeadm init phase etcd local [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for local
--image-repository string     Default: "k8s.gcr.io"
Choose a container registry to pull control plane images from
- - +``` + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for local + --image-repository string Choose a container registry to pull control plane images from (default "k8s.gcr.io") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md index 8931be52e7964..e903238a9effb 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig.md @@ -1,9 +1,8 @@ -Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file +Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm init phase kubeconfig [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for kubeconfig
- - +``` + -h, --help help for kubeconfig +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md index a10f9aaf656d0..4173da018fb03 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_admin.md @@ -1,10 +1,9 @@ -Generates a kubeconfig file for the admin to use and for kubeadm itself +Generate a kubeconfig file for the admin to use and for kubeadm itself ### Synopsis - -Generates the kubeconfig file for the admin and for kubeadm itself, and saves it to admin.conf file. +Generate the kubeconfig file for the admin and for kubeadm itself, and save it to admin.conf file. ``` kubeadm init phase kubeconfig admin [flags] @@ -12,78 +11,18 @@ kubeadm init phase kubeconfig admin [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for admin
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for admin + --kubeconfig-dir string The path where to save the kubeconfig file. (default "/etc/kubernetes") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md index 698541d2733ad..5182f62548269 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_all.md @@ -1,10 +1,9 @@ -Generates all kubeconfig files +Generate all kubeconfig files ### Synopsis - -Generates all kubeconfig files +Generate all kubeconfig files ``` kubeadm init phase kubeconfig all [flags] @@ -12,85 +11,19 @@ kubeadm init phase kubeconfig all [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for all
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
--node-name string
Specify the node name.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for all + --kubeconfig-dir string The path where to save the kubeconfig file. (default "/etc/kubernetes") + --node-name string Specify the node name. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md index 3a27c9a950b5f..5e6514887abf0 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_controller-manager.md @@ -1,10 +1,9 @@ -Generates a kubeconfig file for the controller manager to use +Generate a kubeconfig file for the controller manager to use ### Synopsis - -Generates the kubeconfig file for the controller manager to use and saves it to controller-manager.conf file +Generate the kubeconfig file for the controller manager to use and save it to controller-manager.conf file ``` kubeadm init phase kubeconfig controller-manager [flags] @@ -12,78 +11,18 @@ kubeadm init phase kubeconfig controller-manager [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for controller-manager
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for controller-manager + --kubeconfig-dir string The path where to save the kubeconfig file. (default "/etc/kubernetes") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md index 2ba61026cb99e..4d08ce4bcfd17 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_kubelet.md @@ -1,10 +1,9 @@ -Generates a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes +Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes ### Synopsis - -Generates the kubeconfig file for the kubelet to use and saves it to kubelet.conf file. +Generate the kubeconfig file for the kubelet to use and save it to kubelet.conf file. Please note that this should only be used for cluster bootstrapping purposes. After your control plane is up, you should request all kubelet credentials from the CSR API. @@ -14,85 +13,19 @@ kubeadm init phase kubeconfig kubelet [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for kubelet
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
--node-name string
Specify the node name.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for kubelet + --kubeconfig-dir string The path where to save the kubeconfig file. (default "/etc/kubernetes") + --node-name string Specify the node name. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md index 892e77c829f19..72784a950809d 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubeconfig_scheduler.md @@ -1,10 +1,9 @@ -Generates a kubeconfig file for the scheduler to use +Generate a kubeconfig file for the scheduler to use ### Synopsis - -Generates the kubeconfig file for the scheduler to use and saves it to scheduler.conf file. +Generate the kubeconfig file for the scheduler to use and save it to scheduler.conf file. ``` kubeadm init phase kubeconfig scheduler [flags] @@ -12,78 +11,18 @@ kubeadm init phase kubeconfig scheduler [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. Specify '0.0.0.0' to use the address of the default network interface.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save and store the certificates.
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for scheduler
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
- - +``` + --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443) + --cert-dir string The path where to save and store the certificates. (default "/etc/kubernetes/pki") + --config string Path to a kubeadm configuration file. + -h, --help help for scheduler + --kubeconfig-dir string The path where to save the kubeconfig file. (default "/etc/kubernetes") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md index 69a28ad329fdf..4ee94fe1d0a87 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_kubelet-start.md @@ -1,10 +1,9 @@ -Writes kubelet settings and (re)starts the kubelet +Write kubelet settings and (re)start the kubelet ### Synopsis - -Writes a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)starts kubelet. +Write a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)start kubelet. ``` kubeadm init phase kubelet-start [flags] @@ -14,69 +13,21 @@ kubeadm init phase kubelet-start [flags] ``` # Writes a dynamic environment file with kubelet flags from a InitConfiguration file. - kubeadm init phase kubelet-start --config masterconfig.yaml + kubeadm init phase kubelet-start --config config.yaml ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
--cri-socket string     Default: "/var/run/dockershim.sock"
Specify the CRI socket to connect to.
-h, --help
help for kubelet-start
--node-name string
Specify the node name.
- - +``` + --config string Path to a kubeadm configuration file. + --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. + -h, --help help for kubelet-start + --node-name string Specify the node name. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md index 4c0d119be5cda..f30046cc55b7f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_mark-control-plane.md @@ -3,7 +3,6 @@ Mark a node as a control-plane ### Synopsis - Mark a node as a control-plane ``` @@ -22,57 +21,15 @@ kubeadm init phase mark-control-plane [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for mark-control-plane
--node-name string
Specify the node name.
- - +``` + --config string Path to a kubeadm configuration file. + -h, --help help for mark-control-plane + --node-name string Specify the node name. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md index b5f2626e72bae..499c88dfcd440 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_preflight.md @@ -1,10 +1,9 @@ -Run master pre-flight checks +Run pre-flight checks ### Synopsis - -Run master pre-flight checks, functionally equivalent to what implemented by kubeadm init. +Run pre-flight checks for kubeadm init. ``` kubeadm init phase preflight [flags] @@ -13,63 +12,21 @@ kubeadm init phase preflight [flags] ### Examples ``` - # Run master pre-flight checks using a config file. + # Run pre-flight checks for kubeadm init using a config file. kubeadm init phase preflight --config kubeadm-config.yml ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for preflight
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
- - +``` + --config string Path to a kubeadm configuration file. + -h, --help help for preflight + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md index aed553b8c3d1e..d30c66253789e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-certs.md @@ -14,9 +14,9 @@ kubeadm init phase upload-certs [flags] ``` --certificate-key string Key used to encrypt the control-plane certificates in the kubeadm-certs Secret. --config string Path to a kubeadm configuration file. - --experimental-upload-certs Upload control-plane certificates to the kubeadm-certs Secret. -h, --help help for upload-certs --skip-certificate-key-print Don't print the key used to encrypt the control-plane certificates. + --upload-certs Upload control-plane certificates to the kubeadm-certs Secret. ``` ### Options inherited from parent commands diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md index 0f20e04b16fc0..c44ed6054818c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config.md @@ -1,9 +1,8 @@ -Uploads the kubeadm and kubelet configuration to a ConfigMap +Upload the kubeadm and kubelet configuration to a ConfigMap ### Synopsis - This command is not meant to be run on its own. See list of available subcommands. ``` @@ -12,43 +11,13 @@ kubeadm init phase upload-config [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for upload-config
- - +``` + -h, --help help for upload-config +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md index 8e697d5c5b930..d6ccc063d36b1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_all.md @@ -1,10 +1,9 @@ -Uploads all configuration to a config map +Upload all configuration to a config map ### Synopsis - -Uploads all configuration to a config map +Upload all configuration to a config map ``` kubeadm init phase upload-config all [flags] @@ -12,57 +11,15 @@ kubeadm init phase upload-config all [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for all
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - +``` + --config string Path to a kubeadm configuration file. + -h, --help help for all + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md index 986d5ac0ad817..cec14b629f0e8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubeadm.md @@ -1,10 +1,9 @@ -Uploads the kubeadm ClusterConfiguration to a ConfigMap +Upload the kubeadm ClusterConfiguration to a ConfigMap ### Synopsis - -Uploads the kubeadm ClusterConfiguration to a ConfigMap called kubeadm-config in the kube-system namespace. This enables correct configuration of system components and a seamless user experience when upgrading. +Upload the kubeadm ClusterConfiguration to a ConfigMap called kubeadm-config in the kube-system namespace. This enables correct configuration of system components and a seamless user experience when upgrading. Alternatively, you can use kubeadm config. @@ -15,63 +14,21 @@ kubeadm init phase upload-config kubeadm [flags] ### Examples ``` - # uploads the configuration of your cluster + # upload the configuration of your cluster kubeadm init phase upload-config --config=myConfig.yaml ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for kubeadm
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - +``` + --config string Path to a kubeadm configuration file. + -h, --help help for kubeadm + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md index 8eb48c01ec0a1..549703c9702c8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_init_phase_upload-config_kubelet.md @@ -1,10 +1,9 @@ -Uploads the kubelet component config to a ConfigMap +Upload the kubelet component config to a ConfigMap ### Synopsis - -Uploads kubelet configuration extracted from the kubeadm InitConfiguration object to a ConfigMap of the form kubelet-config-1.X in the cluster, where X is the minor version of the current (API Server) Kubernetes version. +Upload kubelet configuration extracted from the kubeadm InitConfiguration object to a ConfigMap of the form kubelet-config-1.X in the cluster, where X is the minor version of the current (API Server) Kubernetes version. ``` kubeadm init phase upload-config kubelet [flags] @@ -13,63 +12,21 @@ kubeadm init phase upload-config kubelet [flags] ### Examples ``` - # Uploads the kubelet configuration from the kubeadm Config file to a ConfigMap in the cluster. + # Upload the kubelet configuration from the kubeadm Config file to a ConfigMap in the cluster. kubeadm init phase upload-config kubelet --config kubeadm.yaml ``` ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file. WARNING: Usage of a configuration file is experimental.
-h, --help
help for kubelet
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - +``` + --config string Path to a kubeadm configuration file. + -h, --help help for kubelet + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md index 5116ad9850109..8e8c47444e857 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join.md @@ -4,11 +4,10 @@ Run this on any machine you wish to join an existing cluster ### Synopsis - When joining a kubeadm initialized cluster, we need to establish bidirectional trust. This is split into discovery (having the Node -trust the Kubernetes Master) and TLS bootstrap (having the Kubernetes -Master trust the Node). +trust the Kubernetes Control Plane) and TLS bootstrap (having the +Kubernetes Control Plane trust the Node). There are 2 main schemes for discovery. The first is to use a shared token along with the IP address of the API server. The second is to @@ -23,8 +22,8 @@ the connection. If you use a shared token for discovery, you should also pass the --discovery-token-ca-cert-hash flag to validate the public key of the -root certificate authority (CA) presented by the Kubernetes Master. The -value of this flag is specified as ":", +root certificate authority (CA) presented by the Kubernetes Control Plane. +The value of this flag is specified as ":", where the supported hash type is "sha256". The hash is calculated over the bytes of the Subject Public Key Info (SPKI) object (as in RFC7469). This value is available in the output of "kubeadm init" or can be @@ -34,12 +33,12 @@ may be repeated multiple times to allow more than one public key. If you cannot know the CA public key hash ahead of time, you can pass the --discovery-token-unsafe-skip-ca-verification flag to disable this verification. This weakens the kubeadm security model since other nodes -can potentially impersonate the Kubernetes Master. +can potentially impersonate the Kubernetes Control Plane. The TLS bootstrap mechanism is also driven via a shared token. This is -used to temporarily authenticate with the Kubernetes Master to submit a +used to temporarily authenticate with the Kubernetes Control Plane to submit a certificate signing request (CSR) for a locally created key pair. By -default, kubeadm will set up the Kubernetes Master to automatically +default, kubeadm will set up the Kubernetes Control Plane to automatically approve these signing requests. This token is passed in with the --tls-bootstrap-token abcdef.1234567890abcdef flag. @@ -47,133 +46,51 @@ Often times the same token is used for both parts. In this case, the --token flag can be used instead of specifying each token individually. +The "join [api-server-endpoint]" command executes the following phases: ``` -kubeadm join [flags] +preflight Run join pre-flight checks +control-plane-prepare Prepare the machine for serving a control plane + /download-certs [EXPERIMENTAL] Download certificates shared among control-plane nodes from the kubeadm-certs Secret + /certs Generate the certificates for the new control plane components + /kubeconfig Generate the kubeconfig for the new control plane components + /control-plane Generate the manifests for the new control plane components +kubelet-start Write kubelet settings, certificates and (re)start the kubelet +control-plane-join Join a machine as a control plane instance + /etcd Add a new local etcd member + /update-status Register the new control-plane node into the ClusterStatus maintained in the kubeadm-config ConfigMap + /mark-control-plane Mark a node as a control-plane ``` -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--apiserver-advertise-address string
If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on.
--apiserver-bind-port int32     Default: 6443
If the node should host a new control plane instance, the port for the API Server to bind to.
--config string
Path to kubeadm config file.
--cri-socket string     Default: "/var/run/dockershim.sock"
Specify the CRI socket to connect to.
--discovery-file string
A file or URL from which to load cluster information.
--discovery-token string
A token used to validate cluster information fetched from the API server.
--discovery-token-ca-cert-hash stringSlice
For token-based discovery, validate that the root CA public key matches this hash (format: "<type>:<value>").
--discovery-token-unsafe-skip-ca-verification
For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.
--experimental-control-plane
Create a new control plane instance on this node
-h, --help
help for join
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--node-name string
Specify the node name.
--token string
Use this token for both discovery-token and tls-bootstrap-token when those values are not provided.
+``` +kubeadm join [api-server-endpoint] [flags] +``` +### Options +``` + --apiserver-advertise-address string If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used. + --apiserver-bind-port int32 If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443) + --certificate-key string Use this key to decrypt the certificate secrets uploaded by init. + --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node + --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. + --discovery-file string For file-based discovery, a file or URL from which to load cluster information. + --discovery-token string For token-based discovery, the token used to validate cluster information fetched from the API server. + --discovery-token-ca-cert-hash strings For token-based discovery, validate that the root CA public key matches this hash (format: ":"). + --discovery-token-unsafe-skip-ca-verification For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning. + --experimental-control-plane Create a new control plane instance on this node + -h, --help help for join + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + --node-name string Specify the node name. + --skip-phases strings List of phases to be skipped + --tls-bootstrap-token string Specify the token used to temporarily authenticate with the Kubernetes Control Plane while joining the node. + --token string Use this token for both discovery-token and tls-bootstrap-token when those values are not provided. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase.md index a5562c5dc4e6c..23872078a6acd 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase.md @@ -1,9 +1,9 @@ -use this command to invoke single phase of the join workflow +Use this command to invoke single phase of the join workflow ### Synopsis -use this command to invoke single phase of the join workflow +Use this command to invoke single phase of the join workflow ### Options diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md index e65c5248f44d4..6885cc60bb554 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join.md @@ -1,9 +1,9 @@ -Joins a machine as a control plane instance +Join a machine as a control plane instance ### Synopsis -Joins a machine as a control plane instance +Join a machine as a control plane instance ``` kubeadm join phase control-plane-join [flags] diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md index d2f288fd98c59..327f3353c8c9b 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_all.md @@ -1,9 +1,9 @@ -Joins a machine as a control plane instance +Join a machine as a control plane instance ### Synopsis -Joins a machine as a control plane instance +Join a machine as a control plane instance ``` kubeadm join phase control-plane-join all [flags] @@ -14,6 +14,7 @@ kubeadm join phase control-plane-join all [flags] ``` --apiserver-advertise-address string If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --experimental-control-plane Create a new control plane instance on this node -h, --help help for all --node-name string Specify the node name. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md index 05ebd37d41c17..d4282b445bef7 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_etcd.md @@ -14,6 +14,7 @@ kubeadm join phase control-plane-join etcd [flags] ``` --apiserver-advertise-address string If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --experimental-control-plane Create a new control plane instance on this node -h, --help help for etcd --node-name string Specify the node name. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_mark-control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_mark-control-plane.md index 9a06263e3876b..eb8f6e1de36da 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_mark-control-plane.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_mark-control-plane.md @@ -13,6 +13,7 @@ kubeadm join phase control-plane-join mark-control-plane [flags] ``` --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --experimental-control-plane Create a new control plane instance on this node -h, --help help for mark-control-plane --node-name string Specify the node name. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_update-status.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_update-status.md index 00a10bb606939..f01287ded917f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_update-status.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-join_update-status.md @@ -14,6 +14,7 @@ kubeadm join phase control-plane-join update-status [flags] ``` --apiserver-advertise-address string If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --experimental-control-plane Create a new control plane instance on this node -h, --help help for update-status --node-name string Specify the node name. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md index 1ed4d231ba2e9..10535c0c41a09 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare.md @@ -1,9 +1,9 @@ -Prepares the machine for serving a control plane. +Prepare the machine for serving a control plane ### Synopsis -Prepares the machine for serving a control plane. +Prepare the machine for serving a control plane ``` kubeadm join phase control-plane-prepare [flags] diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md index 30e3351584f55..11707f8006a69 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_all.md @@ -1,9 +1,9 @@ -Prepares the machine for serving a control plane. +Prepare the machine for serving a control plane ### Synopsis -Prepares the machine for serving a control plane. +Prepare the machine for serving a control plane ``` kubeadm join phase control-plane-prepare all [api-server-endpoint] [flags] @@ -16,6 +16,7 @@ kubeadm join phase control-plane-prepare all [api-server-endpoint] [flags] --apiserver-bind-port int32 If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443) --certificate-key string Use this key to decrypt the certificate secrets uploaded by init. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --discovery-file string For file-based discovery, a file or URL from which to load cluster information. --discovery-token string For token-based discovery, the token used to validate cluster information fetched from the API server. --discovery-token-ca-cert-hash strings For token-based discovery, validate that the root CA public key matches this hash (format: ":"). diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md index f429b7536cf6e..2392b884de53a 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_certs.md @@ -1,9 +1,9 @@ -Generates the certificates for the new control plane components +Generate the certificates for the new control plane components ### Synopsis -Generates the certificates for the new control plane components +Generate the certificates for the new control plane components ``` kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags] @@ -14,6 +14,7 @@ kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags] ``` --apiserver-advertise-address string If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --discovery-file string For file-based discovery, a file or URL from which to load cluster information. --discovery-token string For token-based discovery, the token used to validate cluster information fetched from the API server. --discovery-token-ca-cert-hash strings For token-based discovery, validate that the root CA public key matches this hash (format: ":"). diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_control-plane.md index cecc4b2a80ae8..1a856cd3c5e8c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_control-plane.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_control-plane.md @@ -1,9 +1,9 @@ -Generates the manifests for the new control plane components +Generate the manifests for the new control plane components ### Synopsis -Generates the manifests for the new control plane components +Generate the manifests for the new control plane components ``` kubeadm join phase control-plane-prepare control-plane [flags] @@ -15,6 +15,7 @@ kubeadm join phase control-plane-prepare control-plane [flags] --apiserver-advertise-address string If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used. --apiserver-bind-port int32 If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443) --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --experimental-control-plane Create a new control plane instance on this node -h, --help help for control-plane ``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_download-certs.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_download-certs.md index cb87677c20600..473a8eac4600e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_download-certs.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_download-certs.md @@ -1,9 +1,9 @@ -[EXPERIMENTAL] Downloads certificates shared among control-plane nodes from the kubeadm-certs Secret +[EXPERIMENTAL] Download certificates shared among control-plane nodes from the kubeadm-certs Secret ### Synopsis -[EXPERIMENTAL] Downloads certificates shared among control-plane nodes from the kubeadm-certs Secret +[EXPERIMENTAL] Download certificates shared among control-plane nodes from the kubeadm-certs Secret ``` kubeadm join phase control-plane-prepare download-certs [api-server-endpoint] [flags] @@ -14,6 +14,7 @@ kubeadm join phase control-plane-prepare download-certs [api-server-endpoint] [f ``` --certificate-key string Use this key to decrypt the certificate secrets uploaded by init. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --discovery-file string For file-based discovery, a file or URL from which to load cluster information. --discovery-token string For token-based discovery, the token used to validate cluster information fetched from the API server. --discovery-token-ca-cert-hash strings For token-based discovery, validate that the root CA public key matches this hash (format: ":"). diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_kubeconfig.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_kubeconfig.md index 558ed7fd33ccb..187f2ff3bbaf4 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_kubeconfig.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_control-plane-prepare_kubeconfig.md @@ -1,9 +1,9 @@ -Generates the kubeconfig for the new control plane components +Generate the kubeconfig for the new control plane components ### Synopsis -Generates the kubeconfig for the new control plane components +Generate the kubeconfig for the new control plane components ``` kubeadm join phase control-plane-prepare kubeconfig [api-server-endpoint] [flags] @@ -14,6 +14,7 @@ kubeadm join phase control-plane-prepare kubeconfig [api-server-endpoint] [flags ``` --certificate-key string Use this key to decrypt the certificate secrets uploaded by init. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --discovery-file string For file-based discovery, a file or URL from which to load cluster information. --discovery-token string For token-based discovery, the token used to validate cluster information fetched from the API server. --discovery-token-ca-cert-hash strings For token-based discovery, validate that the root CA public key matches this hash (format: ":"). diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md index 6120e664bb255..ef670419cfc66 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_kubelet-start.md @@ -1,9 +1,9 @@ -Writes kubelet settings, certificates and (re)starts the kubelet +Write kubelet settings, certificates and (re)start the kubelet ### Synopsis -Writes a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)starts kubelet. +Write a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)start kubelet. ``` kubeadm join phase kubelet-start [api-server-endpoint] [flags] diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_preflight.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_preflight.md index 70643a0da341a..14f7d4b1003bd 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_preflight.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_join_phase_preflight.md @@ -23,6 +23,7 @@ kubeadm join phase preflight [api-server-endpoint] [flags] --apiserver-bind-port int32 If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443) --certificate-key string Use this key to decrypt the certificate secrets uploaded by init. --config string Path to kubeadm config file. + --control-plane Create a new control plane instance on this node --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. --discovery-file string For file-based discovery, a file or URL from which to load cluster information. --discovery-token string For token-based discovery, the token used to validate cluster information fetched from the API server. diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md index ee319608530e4..8ce1e6dfba8ab 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset.md @@ -1,10 +1,18 @@ -Run this to revert any changes made to this host by 'kubeadm init' or 'kubeadm join'. +Run this to revert any changes made to this host by 'kubeadm init' or 'kubeadm join' ### Synopsis +Run this to revert any changes made to this host by 'kubeadm init' or 'kubeadm join' + +The "reset" command executes the following phases: +``` +preflight Run reset pre-flight checks +update-cluster-status Remove this node from the ClusterStatus object. +remove-etcd-member Remove a local etcd member. +cleanup-node Run cleanup node. +``` -Run this to revert any changes made to this host by 'kubeadm init' or 'kubeadm join'. ``` kubeadm reset [flags] @@ -12,78 +20,19 @@ kubeadm reset [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--cert-dir string     Default: "/etc/kubernetes/pki"
The path to the directory where the certificates are stored. If specified, clean this directory.
--cri-socket string     Default: "/var/run/dockershim.sock"
The path to the CRI socket to use with crictl when cleaning up containers.
-f, --force
Reset the node without prompting for confirmation.
-h, --help
help for reset
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - +``` + --cert-dir string The path to the directory where the certificates are stored. If specified, clean this directory. (default "/etc/kubernetes/pki") + --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. + -f, --force Reset the node without prompting for confirmation. + -h, --help help for reset + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --skip-phases strings List of phases to be skipped +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase.md new file mode 100644 index 0000000000000..cec28480c045f --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase.md @@ -0,0 +1,19 @@ + +Use this command to invoke single phase of the reset workflow + +### Synopsis + +Use this command to invoke single phase of the reset workflow + +### Options + +``` + -h, --help help for phase +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md new file mode 100644 index 0000000000000..2d7c4e99268ca --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_cleanup-node.md @@ -0,0 +1,25 @@ + +Run cleanup node. + +### Synopsis + +Run cleanup node. + +``` +kubeadm reset phase cleanup-node [flags] +``` + +### Options + +``` + --cert-dir string The path to the directory where the certificates are stored. If specified, clean this directory. (default "/etc/kubernetes/pki") + --cri-socket string Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket. + -h, --help help for cleanup-node +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_preflight.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_preflight.md new file mode 100644 index 0000000000000..e40c9707123f7 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_preflight.md @@ -0,0 +1,25 @@ + +Run reset pre-flight checks + +### Synopsis + +Run pre-flight checks for kubeadm reset. + +``` +kubeadm reset phase preflight [flags] +``` + +### Options + +``` + -f, --force Reset the node without prompting for confirmation. + -h, --help help for preflight + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md new file mode 100644 index 0000000000000..7642494a5bec2 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_remove-etcd-member.md @@ -0,0 +1,24 @@ + +Remove a local etcd member. + +### Synopsis + +Remove a local etcd member for a control plane node. + +``` +kubeadm reset phase remove-etcd-member [flags] +``` + +### Options + +``` + -h, --help help for remove-etcd-member + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_update-cluster-status.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_update-cluster-status.md new file mode 100644 index 0000000000000..08b87e73ae125 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_reset_phase_update-cluster-status.md @@ -0,0 +1,23 @@ + +Remove this node from the ClusterStatus object. + +### Synopsis + +Remove this node from the ClusterStatus object if the node is a control plane node. + +``` +kubeadm reset phase update-cluster-status [flags] +``` + +### Options + +``` + -h, --help help for update-cluster-status +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md index 0f57b4a9fd01b..85aecc6f46f16 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token.md @@ -1,10 +1,9 @@ -Manage bootstrap tokens. +Manage bootstrap tokens ### Synopsis - This command manages bootstrap tokens. It is optional and needed only for advanced use cases. In short, bootstrap tokens are used for establishing bidirectional trust between a client and a server. @@ -20,7 +19,7 @@ What is a bootstrap token more exactly? - The name of the Secret must be named "bootstrap-token-(token-id)". You can read more about bootstrap tokens here: - /docs/admin/bootstrap-tokens/ + https://kubernetes.io/docs/admin/bootstrap-tokens/ ``` @@ -29,54 +28,15 @@ kubeadm token [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Whether to enable dry-run mode or not
-h, --help
help for token
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - +``` + --dry-run Whether to enable dry-run mode or not + -h, --help help for token + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` ### Options inherited from parent commands - - - - - - - - - - - - - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md index dfb557fcc6eaf..6c35a2166bc91 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_create.md @@ -1,10 +1,9 @@ -Create bootstrap tokens on the server. +Create bootstrap tokens on the server ### Synopsis - This command will create a bootstrap token for you. You can specify the usages for this token, the "time to live" and an optional human friendly description. @@ -19,99 +18,21 @@ kubeadm token create [token] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--config string
Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
--description string
A human friendly description of how this token is used.
--groups stringSlice     Default: [system:bootstrappers:kubeadm:default-node-token]
Extra groups that this token will authenticate as when used for authentication. Must match "\\Asystem:bootstrappers:[a-z0-9:-]{0,255}[a-z0-9]\\z"
-h, --help
help for create
--print-join-command
Instead of printing only the token, print the full 'kubeadm join' flag needed to join the cluster using the token.
--ttl duration     Default: 24h0m0s
The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire
--usages stringSlice     Default: [signing,authentication]
Describes the ways in which this token can be used. You can pass --usages multiple times or provide a comma separated list of options. Valid options: [signing,authentication]
- - +``` + --config string Path to a kubeadm configuration file. + --description string A human friendly description of how this token is used. + --groups strings Extra groups that this token will authenticate as when used for authentication. Must match "\\Asystem:bootstrappers:[a-z0-9:-]{0,255}[a-z0-9]\\z" (default [system:bootstrappers:kubeadm:default-node-token]) + -h, --help help for create + --print-join-command Instead of printing only the token, print the full 'kubeadm join' flag needed to join the cluster using the token. + --ttl duration The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire (default 24h0m0s) + --usages strings Describes the ways in which this token can be used. You can pass --usages multiple times or provide a comma separated list of options. Valid options: [signing,authentication] (default [signing,authentication]) +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Whether to enable dry-run mode or not
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --dry-run Whether to enable dry-run mode or not + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md index 415ce6e1078a7..8dd1908b43ce2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_delete.md @@ -1,73 +1,30 @@ -Delete bootstrap tokens on the server. +Delete bootstrap tokens on the server ### Synopsis - -This command will delete a given bootstrap token for you. +This command will delete a list of bootstrap tokens for you. The [token-value] is the full Token of the form "[a-z0-9]{6}.[a-z0-9]{16}" or the Token ID of the form "[a-z0-9]{6}" to delete. ``` -kubeadm token delete [token-value] +kubeadm token delete [token-value] ... ``` ### Options - - - - - - - - - - - - - - - -
-h, --help
help for delete
- - +``` + -h, --help help for delete +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Whether to enable dry-run mode or not
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --dry-run Whether to enable dry-run mode or not + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md index 2194e642031d9..a49dfb185810c 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_generate.md @@ -1,10 +1,9 @@ -Generate and print a bootstrap token, but do not create it on the server. +Generate and print a bootstrap token, but do not create it on the server ### Synopsis - This command will print out a randomly-generated bootstrap token that can be used with the "init" and "join" commands. @@ -22,57 +21,15 @@ kubeadm token generate [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for generate
- - +``` + -h, --help help for generate +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Whether to enable dry-run mode or not
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --dry-run Whether to enable dry-run mode or not + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md index d49fb72c86740..b4d2699d13f6e 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_token_list.md @@ -1,10 +1,9 @@ -List bootstrap tokens on the server. +List bootstrap tokens on the server ### Synopsis - This command will list all bootstrap tokens for you. @@ -14,57 +13,15 @@ kubeadm token list [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for list
- - +``` + -h, --help help for list +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Whether to enable dry-run mode or not
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --dry-run Whether to enable dry-run mode or not + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md index dd3d5f9d897ec..fd29befad9ea0 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade.md @@ -1,10 +1,9 @@ -Upgrade your cluster smoothly to a newer version with this command. +Upgrade your cluster smoothly to a newer version with this command ### Synopsis - -Upgrade your cluster smoothly to a newer version with this command. +Upgrade your cluster smoothly to a newer version with this command ``` kubeadm upgrade [flags] @@ -12,43 +11,13 @@ kubeadm upgrade [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for upgrade
- - +``` + -h, --help help for upgrade +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md index b199ac406fdf3..7a0ad92572ac8 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_apply.md @@ -1,10 +1,9 @@ -Upgrade your Kubernetes cluster to the specified version. +Upgrade your Kubernetes cluster to the specified version ### Synopsis - -Upgrade your Kubernetes cluster to the specified version. +Upgrade your Kubernetes cluster to the specified version ``` kubeadm upgrade apply [version] @@ -12,134 +11,26 @@ kubeadm upgrade apply [version] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--allow-experimental-upgrades
Show unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes.
--allow-release-candidate-upgrades
Show release candidate versions of Kubernetes as an upgrade alternative and allow upgrading to a release candidate versions of Kubernetes.
--config string
Path to a kubeadm configuration file.
--cri-socket string     Default: "/var/run/dockershim.sock"
Specify the CRI socket to connect to.
--dry-run
Do not change any state, just output what actions would be performed.
--etcd-upgrade     Default: true
Perform the upgrade of etcd.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-f, --force
Force upgrading although some requirements might not be met. This also implies non-interactive mode.
-h, --help
help for apply
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--image-pull-timeout duration     Default: 15m0s
The maximum amount of time to wait for the control plane pods to be downloaded.
--kubeconfig string     Default: "/Users/tim/.kube/config"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--print-config
Specifies whether the configuration file that will be used in the upgrade should be printed or not.
-y, --yes
Perform the upgrade and do not prompt for confirmation (non-interactive mode).
- - +``` + --allow-experimental-upgrades Show unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes. + --allow-release-candidate-upgrades Show release candidate versions of Kubernetes as an upgrade alternative and allow upgrading to a release candidate versions of Kubernetes. + --certificate-renewal Perform the renewal of certificates used by component changed during upgrades. (default true) + --config string Path to a kubeadm configuration file. + --dry-run Do not change any state, just output what actions would be performed. + --etcd-upgrade Perform the upgrade of etcd. (default true) + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -f, --force Force upgrading although some requirements might not be met. This also implies non-interactive mode. + -h, --help help for apply + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + --image-pull-timeout duration The maximum amount of time to wait for the control plane pods to be downloaded. (default 15m0s) + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --print-config Specifies whether the configuration file that will be used in the upgrade should be printed or not. + -y, --yes Perform the upgrade and do not prompt for confirmation (non-interactive mode). +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md index 0403034bdfaab..950b91d7eaca5 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_diff.md @@ -3,7 +3,6 @@ Show what differences would be applied to existing static pod manifests. See als ### Synopsis - Show what differences would be applied to existing static pod manifests. See also: kubeadm upgrade apply --dry-run ``` @@ -12,78 +11,18 @@ kubeadm upgrade diff [version] [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--api-server-manifest string     Default: "/etc/kubernetes/manifests/kube-apiserver.yaml"
path to API server manifest
--config string
Path to a kubeadm configuration file.
-c, --context-lines int     Default: 3
How many lines of context in the diff
--controller-manager-manifest string     Default: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
path to controller manifest
-h, --help
help for diff
--scheduler-manifest string     Default: "/etc/kubernetes/manifests/kube-scheduler.yaml"
path to scheduler manifest
- - +``` + --api-server-manifest string path to API server manifest (default "/etc/kubernetes/manifests/kube-apiserver.yaml") + --config string Path to a kubeadm configuration file. + -c, --context-lines int How many lines of context in the diff (default 3) + --controller-manager-manifest string path to controller manifest (default "/etc/kubernetes/manifests/kube-controller-manager.yaml") + -h, --help help for diff + --scheduler-manifest string path to scheduler manifest (default "/etc/kubernetes/manifests/kube-scheduler.yaml") +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md index 0c76e50de5b8a..0c58916493e76 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node.md @@ -1,10 +1,16 @@ -Upgrade commands for a node in the cluster. Currently only supports upgrading the configuration, not the kubelet itself. +Upgrade commands for a node in the cluster ### Synopsis +Upgrade commands for a node in the cluster + +The "node" command executes the following phases: +``` +control-plane Upgrade the control plane instance deployed on this node, if any +kubelet-config Upgrade the kubelet configuration for this node +``` -Upgrade commands for a node in the cluster. Currently only supports upgrading the configuration, not the kubelet itself. ``` kubeadm upgrade node [flags] @@ -12,43 +18,17 @@ kubeadm upgrade node [flags] ### Options - - - - - - - - - - - - - - - -
-h, --help
help for node
- - +``` + --dry-run Do not change any state, just output the actions that would be performed. + -h, --help help for node + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubelet-version string The *desired* version for the kubelet config after the upgrade. If not specified, the KubernetesVersion from the kubeadm-config ConfigMap will be used + --skip-phases strings List of phases to be skipped +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md deleted file mode 100644 index 85c5ccd8fec68..0000000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_config.md +++ /dev/null @@ -1,86 +0,0 @@ - -Downloads the kubelet configuration from the cluster ConfigMap kubelet-config-1.X, where X is the minor version of the kubelet. - -### Synopsis - - -Downloads the kubelet configuration from a ConfigMap of the form "kubelet-config-1.X" in the cluster, where X is the minor version of the kubelet. kubeadm uses the --kubelet-version parameter to determine what the desired kubelet version is. Give - -``` -kubeadm upgrade node config [flags] -``` - -### Examples - -``` - # Downloads the kubelet configuration from the ConfigMap in the cluster. Uses a specific desired kubelet version. - kubeadm upgrade node config --kubelet-version v1.13.0 - - # Simulates the downloading of the kubelet configuration from the ConfigMap in the cluster with a specific desired - # version. Does not change any state locally on the node. - kubeadm upgrade node config --kubelet-version v1.13.0 --dry-run -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Do not change any state, just output the actions that would be performed.
-h, --help
help for config
--kubeconfig string     Default: "/etc/kubernetes/kubelet.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--kubelet-version string
The *desired* version for the kubelet after the upgrade.
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md deleted file mode 100644 index 7b85992770d3d..0000000000000 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_experimental-control-plane.md +++ /dev/null @@ -1,86 +0,0 @@ - -Upgrades the control plane instance deployed on this node. IMPORTANT. This command should be executed after executing `kubeadm upgrade apply` on another control plane instance - -### Synopsis - - -Downloads the kubelet configuration from a ConfigMap of the form "kubelet-config-1.X" in the cluster, where X is the minor version of the kubelet. kubeadm uses the --kubelet-version parameter to determine what the desired kubelet version is. Give - -``` -kubeadm upgrade node experimental-control-plane [flags] -``` - -### Examples - -``` - # Downloads the kubelet configuration from the ConfigMap in the cluster. Uses a specific desired kubelet version. - kubeadm upgrade node config --kubelet-version v1.13.0 - - # Simulates the downloading of the kubelet configuration from the ConfigMap in the cluster with a specific desired - # version. Does not change any state locally on the node. - kubeadm upgrade node config --kubelet-version v1.13.0 --dry-run -``` - -### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--dry-run
Do not change any state, just output the actions that would be performed.
--etcd-upgrade     Default: true
Perform the upgrade of etcd.
-h, --help
help for experimental-control-plane
--kubeconfig string     Default: "/etc/kubernetes/kubelet.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
- - - -### Options inherited from parent commands - - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - - diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md new file mode 100644 index 0000000000000..872e9f593d9ae --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase.md @@ -0,0 +1,19 @@ + +Use this command to invoke single phase of the node workflow + +### Synopsis + +Use this command to invoke single phase of the node workflow + +### Options + +``` + -h, --help help for phase +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_control-plane.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_control-plane.md new file mode 100644 index 0000000000000..ed0d571997e00 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_control-plane.md @@ -0,0 +1,25 @@ + +Upgrade the control plane instance deployed on this node, if any + +### Synopsis + +Upgrade the control plane instance deployed on this node, if any + +``` +kubeadm upgrade node phase control-plane [flags] +``` + +### Options + +``` + --dry-run Do not change any state, just output the actions that would be performed. + -h, --help help for control-plane + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md new file mode 100644 index 0000000000000..dee8786af7632 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_node_phase_kubelet-config.md @@ -0,0 +1,26 @@ + +Upgrade the kubelet configuration for this node + +### Synopsis + +Download the kubelet configuration from a ConfigMap of the form "kubelet-config-1.X" in the cluster, where X is the minor version of the kubelet. kubeadm uses the KuberneteVersion field in the kubeadm-config ConfigMap to determine what the desired kubelet version is, but the user can override this by using the --kubelet-version parameter. + +``` +kubeadm upgrade node phase kubelet-config [flags] +``` + +### Options + +``` + --dry-run Do not change any state, just output the actions that would be performed. + -h, --help help for kubelet-config + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --kubelet-version string The *desired* version for the kubelet config after the upgrade. If not specified, the KubernetesVersion from the kubeadm-config ConfigMap will be used +``` + +### Options inherited from parent commands + +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` + diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md index a58b22d6ba5f2..62d73bd99a9cf 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_upgrade_plan.md @@ -1,10 +1,9 @@ -Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter. +Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter ### Synopsis - -Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter. +Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter ``` kubeadm upgrade plan [version] [flags] @@ -12,92 +11,20 @@ kubeadm upgrade plan [version] [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
--allow-experimental-upgrades
Show unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes.
--allow-release-candidate-upgrades
Show release candidate versions of Kubernetes as an upgrade alternative and allow upgrading to a release candidate versions of Kubernetes.
--config string
Path to a kubeadm configuration file.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
-h, --help
help for plan
--ignore-preflight-errors stringSlice
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--kubeconfig string     Default: "/Users/tim/.kube/config"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations are searched for an existing KubeConfig file.
--print-config
Specifies whether the configuration file that will be used in the upgrade should be printed or not.
- - +``` + --allow-experimental-upgrades Show unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes. + --allow-release-candidate-upgrades Show release candidate versions of Kubernetes as an upgrade alternative and allow upgrading to a release candidate versions of Kubernetes. + --config string Path to a kubeadm configuration file. + --feature-gates string A set of key=value pairs that describe feature gates for various features. No feature gates are available in this release. + -h, --help help for plan + --ignore-preflight-errors strings A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks. + --kubeconfig string The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file. (default "/etc/kubernetes/admin.conf") + --print-config Specifies whether the configuration file that will be used in the upgrade should be printed or not. +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md index 9e824f238c491..df927d0759627 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md +++ b/content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm_version.md @@ -3,7 +3,6 @@ Print the version of kubeadm ### Synopsis - Print the version of kubeadm ``` @@ -12,50 +11,14 @@ kubeadm version [flags] ### Options - - - - - - - - - - - - - - - - - - - - - - -
-h, --help
help for version
-o, --output string
Output format; available options are 'yaml', 'json' and 'short'
- - +``` + -h, --help help for version + -o, --output string Output format; available options are 'yaml', 'json' and 'short' +``` ### Options inherited from parent commands - - - - - - - - - - - - - - - -
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
- - +``` + --rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem. +``` diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md index b5114e0861c59..35e38af3f723f 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-alpha.md @@ -14,19 +14,41 @@ weight: 90 ## kubeadm alpha certs renew {#cmd-certs-renew} You can renew all Kubernetes certificates using the `all` subcommand or renew them selectively. +For more details about certificate expiration and renewal see the [certificate management documentation](docs/tasks/administer-cluster/kubeadm/kubeadm-certs). {{< tabs name="tab-certs-renew" >}} {{< tab name="renew" include="generated/kubeadm_alpha_certs_renew.md" />}} {{< tab name="all" include="generated/kubeadm_alpha_certs_renew_all.md" />}} +{{< tab name="admin.conf" include="generated/kubeadm_alpha_certs_renew_admin.conf.md" />}} {{< tab name="apiserver-etcd-client" include="generated/kubeadm_alpha_certs_renew_apiserver-etcd-client.md" />}} {{< tab name="apiserver-kubelet-client" include="generated/kubeadm_alpha_certs_renew_apiserver-kubelet-client.md" />}} {{< tab name="apiserver" include="generated/kubeadm_alpha_certs_renew_apiserver.md" />}} +{{< tab name="controller-manager.conf" include="generated/kubeadm_alpha_certs_renew_controller-manager.conf.md" />}} {{< tab name="etcd-healthcheck-client" include="generated/kubeadm_alpha_certs_renew_etcd-healthcheck-client.md" />}} {{< tab name="etcd-peer" include="generated/kubeadm_alpha_certs_renew_etcd-peer.md" />}} {{< tab name="etcd-server" include="generated/kubeadm_alpha_certs_renew_etcd-server.md" />}} {{< tab name="front-proxy-client" include="generated/kubeadm_alpha_certs_renew_front-proxy-client.md" />}} +{{< tab name="scheduler.conf" include="generated/kubeadm_alpha_certs_renew_scheduler.conf.md" />}} {{< /tabs >}} +## kubeadm alpha certs certificate-key {#cmd-certs-certificate-key} + +This command can be used to generate a new control-plane certificate key. +The key can be passed as `--certificate-key` to `kubeadm init` and `kubeadm join` +to enable the automatic copy of certificates when joining additional control-plane nodes. + +{{< tabs name="tab-certs-certificate-key" >}} +{{< tab name="certificate-key" include="generated/kubeadm_alpha_certs_certificate-key.md" />}} +{{< /tabs >}} + +## kubeadm alpha certs check-expiration {#cmd-certs-check-expiration} + +This command checks expiration for the certificates in the local PKI managed by kubeadm. +For more details about certificate expiration and renewal see the [certificate management documentation](docs/tasks/administer-cluster/kubeadm/kubeadm-certs). + +{{< tabs name="tab-certs-check-expiration" >}} +{{< tab name="check-expiration" include="generated/kubeadm_alpha_certs_check-expiration.md" />}} +{{< /tabs >}} ## kubeadm alpha kubeconfig user {#cmd-phase-kubeconfig} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md index e7ea1eaefe02d..b740d18e9cfd2 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-config.md @@ -28,12 +28,6 @@ has to be used. {{% /capture %}} {{% capture body %}} -## kubeadm config upload from-file {#cmd-config-from-file} -{{< include "generated/kubeadm_config_upload_from-file.md" >}} - -## kubeadm config upload from-flags {#cmd-config-from-flags} -{{< include "generated/kubeadm_config_upload_from-flags.md" >}} - ## kubeadm config view {#cmd-config-view} {{< include "generated/kubeadm_config_view.md" >}} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md index c65da6fddb1d9..7cbbfcbddb1c9 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init-phase.md @@ -32,7 +32,7 @@ Can be used to create all required certificates by kubeadm. {{< tab name="etcd-server" include="generated/kubeadm_init_phase_certs_etcd-server.md" />}} {{< tab name="front-proxy-ca" include="generated/kubeadm_init_phase_certs_front-proxy-ca.md" />}} {{< tab name="front-proxy-client" include="generated/kubeadm_init_phase_certs_front-proxy-client.md" />}} -{{< tab name="certs_sa" include="generated/kubeadm_init_phase_certs_sa.md" />}} +{{< tab name="sa" include="generated/kubeadm_init_phase_certs_sa.md" />}} {{< /tabs >}} ## kubeadm init phase kubeconfig {#cmd-phase-kubeconfig} diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md index 07d797b708e01..5a0a1fddf0662 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-init.md @@ -92,7 +92,7 @@ You can also use `--help` to see the list of sub-phases for a certain parent pha sudo kubeadm init phase control-plane --help ``` -`kubeadm init` also expose a flag called `--skip-phases` that can be used to skip certain phases. The flag accepts a list of phase names and the names can be taken from the above ordered list. +`kubeadm init` also exposes a flag called `--skip-phases` that can be used to skip certain phases. The flag accepts a list of phase names and the names can be taken from the above ordered list. An example: @@ -154,6 +154,30 @@ Allowed customization are: Please note that the configuration field `kubernetesVersion` or the command line flag `--kubernetes-version` affect the version of the images. +### Uploading control-plane certificates to the cluster + +By adding the flag `--upload-certs` to `kubeadm init` you can temporary upload +the control-plane certificates to a Secret in the cluster. Please note that this Secret +will expire automatically after 2 hours. The certificates are encrypted using +a 32byte key that can be specified using `--certificate-key`. The same key can be used +to download the certificates when additional control-plane nodes are joining, by passing +`--control-plane` and `--certificate-key` to `kubeadm join`. + +The following phase command can be used to re-upload the certificates after expiration: + +``` +kubeadm init phase upload-certs --upload-certs --certificate-key=SOME_VALUE +``` + +If the flag `--certificate-key` is not passed to `kubeadm init` and +`kubeadm init phase upload-certs` a new key will be generated automatically. + +The following command can be used to generate a new key on demand: + +``` +kubeadm alpha certs certificate-key +``` + ### Using custom certificates {#custom-certificates} By default, kubeadm generates all the certificates needed for a cluster to run. @@ -191,14 +215,18 @@ This is what it looks like: ``` [Service] -Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" -Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true" -Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" -Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local" -Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt" -Environment="KUBELET_CADVISOR_ARGS=" -Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki" -ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS +Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf +--kubeconfig=/etc/kubernetes/kubelet.conf" +Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" +# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating +the KUBELET_KUBEADM_ARGS variable dynamically +EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env +# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, +#the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. +# KUBELET_EXTRA_ARGS should be sourced from this file. +EnvironmentFile=-/etc/default/kubelet +ExecStart= +ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS ``` Here's a breakdown of what/why: @@ -269,7 +297,7 @@ using an external CRI implementation. ### Setting the node name By default, `kubeadm` assigns a node name based on a machine's host address. You can override this setting with the `--node-name`flag. -The flag passes the appropriate [`--hostname-override`](/docs/reference/command-line-tools-reference/kubelet/#options) +The flag passes the appropriate [`--hostname-override`](/docs/reference/command-line-tools-reference/kubelet/#options) to the kubelet. Be aware that overriding the hostname can [interfere with cloud providers](https://github.com/kubernetes/website/pull/8873). @@ -305,9 +333,16 @@ know the IP address that the control-plane node will have after it is started. kubeadm token generate ``` -1. Start both the control-plane node and the worker nodes concurrently with this token. - As they come up they should find each other and form the cluster. The same - `--token` argument can be used on both `kubeadm init` and `kubeadm join`. +1. Start both the control-plane node and the worker nodes concurrently with this token. + As they come up they should find each other and form the cluster. The same + `--token` argument can be used on both `kubeadm init` and `kubeadm join`. + +1. Similar can be done for `--certificate-key` when joining additional control-plane + nodes. The key can be generated using: + + ```shell + kubeadm alpha certs certificate-key + ``` Once the cluster is up, you can grab the admin credentials from the control-plane node at `/etc/kubernetes/admin.conf` and use that to talk to the cluster. diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md index 7852e16af1e0e..563f6390ffe30 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-join.md @@ -98,9 +98,12 @@ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert For control-plane nodes: ```shell -kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef --experimental-control-plane 1.2.3.4:6443 +kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef --control-plane 1.2.3.4:6443 ``` +You can also call `join` for a control-plane node with `--certificate-key` to copy certificates to this node, +if the `kubeadm init` command was called with `--upload-certs`. + **Advantages:** - Allows bootstrapping nodes to securely discover a root of trust for the @@ -250,7 +253,7 @@ contain a `JoinConfiguration` structure. To print the default values of `JoinConfiguration` run the following command: ```shell -kubeadm config print-default --api-objects=JoinConfiguration +kubeadm config print join-defaults ``` For details on individual fields in `JoinConfiguration` see [the godoc](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#JoinConfiguration). diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset-phase.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset-phase.md new file mode 100644 index 0000000000000..6d17e73c09c42 --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset-phase.md @@ -0,0 +1,53 @@ +--- +title: kubeadm reset phase +weight: 90 +--- +In v1.15.0, kubeadm introduces the `kubeadm reset phase` command with the aim of making kubeadm more modular. This modularity enables you to invoke atomic sub-steps of the reset process. +Hence, you can let kubeadm do some parts and fill in yourself where you need customizations. + +`kubeadm reset phase` is consistent with the [kubeadm reset workflow](/docs/reference/setup-tools/kubeadm/kubeadm-reset/#reset-workflow), +and behind the scene both use the same code. + +## kubeadm reset phase {#cmd-reset-phase} + +{{< tabs name="tab-phase" >}} +{{< tab name="phase" include="generated/kubeadm_reset_phase.md" />}} +{{< /tabs >}} + +## kubeadm reset phase preflight {#cmd-reset-phase-preflight} + +Using this phase you can execute preflight checks on a node that is being reset. + +{{< tabs name="tab-preflight" >}} +{{< tab name="preflight" include="generated/kubeadm_reset_phase_preflight.md" />}} +{{< /tabs >}} + +## kubeadm reset phase update-cluster-status {#cmd-reset-phase-update-cluster-status} + +Using this phase you can remove this control-plane node from the ClusterStatus object. + +{{< tabs name="tab-update-cluster-status" >}} +{{< tab name="update-cluster-status" include="generated/kubeadm_reset_phase_update-cluster-status.md" />}} +{{< /tabs >}} + +## kubeadm reset phase remove-etcd-member {#cmd-reset-phase-remove-etcd-member} + +Using this phase you can remove this control-plane node's etcd member from the etcd cluster. + +{{< tabs name="tab-remove-etcd-member" >}} +{{< tab name="remove-etcd-member" include="generated/kubeadm_reset_phase_remove-etcd-member.md" />}} +{{< /tabs >}} + +## kubeadm reset phase cleanup-node {#cmd-reset-phase-cleanup-node} + +Using this phase you can perform cleanup on this node. + +{{< tabs name="tab-cleanup-node" >}} +{{< tab name="cleanup-node" include="generated/kubeadm_reset_phase_cleanup-node.md" />}} +{{< /tabs >}} + +## What's next +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node +* [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) to connect a node to the cluster +* [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) to revert any changes made to this host by `kubeadm init` or `kubeadm join` +* [kubeadm alpha](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/) to try experimental functionality diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md index 94ecaeb09805f..5847e6209dd58 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-reset.md @@ -14,7 +14,18 @@ This command reverts any changes made by `kubeadm init` or `kubeadm join`. {{% capture body %}} {{< include "generated/kubeadm_reset.md" >}} -### External etcd clean up! +### Reset workflow {#reset-workflow} + +`kubeadm reset` is responsible for cleaning up a node local file system from files that were created using +the `kubeadm init` or `kubeadm join` commands. For control-plane nodes `reset` also removes the local stacked +etcd member of this node from the etcd cluster and also removes this node's information from the kubeadm +`ClusterStatus` object. `ClusterStatus` is a kubeadm managed Kubernetes API object that holds a list of kube-apiserver endpoints. + +`kubeadm reset phase` can be used to execute the separate phases of the above workflow. +To skip a list of phases you can use the `--skip-phases` flag, which works in a similar way to +the `kubeadm join` and `kubeadm init` phase runners. + +### External etcd clean up `kubeadm reset` will not delete any etcd data if external etcd is used. This means that if you run `kubeadm init` again using the same etcd endpoints, you will see state from previous clusters. diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade-phase.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade-phase.md new file mode 100644 index 0000000000000..6224a18e0e50e --- /dev/null +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade-phase.md @@ -0,0 +1,26 @@ +--- +title: kubeadm upgrade phase +weight: 90 +--- +In v1.15.0, kubeadm introduced preliminary support for `kubeadm upgrade node` phases. +Phases for other `kubeadm upgrade` sub-commands such as `apply`, could be added in the +following releases. + +## kubeadm upgrade node phase {#cmd-node-phase} + +Using this phase you can choose to execute the separate steps of the upgrade of +secondary control-plane or worker nodes. Please note that `kubeadm upgrade apply` still has to +be called on a primary control-plane node. + +{{< tabs name="tab-phase" >}} +{{< tab name="phase" include="generated/kubeadm_upgrade_node_phase.md" />}} +{{< tab name="control-plane" include="generated/kubeadm_upgrade_node_phase_control-plane.md" />}} +{{< tab name="kubelet-config" include="generated/kubeadm_upgrade_node_phase_kubelet-config.md" />}} +{{< /tabs >}} + +## What's next +* [kubeadm init](/docs/reference/setup-tools/kubeadm/kubeadm-init/) to bootstrap a Kubernetes control-plane node +* [kubeadm join](/docs/reference/setup-tools/kubeadm/kubeadm-join/) to connect a node to the cluster +* [kubeadm reset](/docs/reference/setup-tools/kubeadm/kubeadm-reset/) to revert any changes made to this host by `kubeadm init` or `kubeadm join` +* [kubeadm upgrade](/docs/reference/setup-tools/kubeadm/kubeadm-upgrade/) to upgrade a kubeadm node +* [kubeadm alpha](/docs/reference/setup-tools/kubeadm/kubeadm-alpha/) to try experimental functionality diff --git a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md index b525a62edc3e5..1de1da50b21d1 100644 --- a/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md +++ b/content/en/docs/reference/setup-tools/kubeadm/kubeadm-upgrade.md @@ -14,15 +14,16 @@ cluster if necessary. {{% /capture %}} {{% capture body %}} + ## kubeadm upgrade guidance Every upgrade process might be a bit different, so we've documented each minor upgrade process individually. For more version-specific upgrade guidance, see the following resources: - * [1.10 to 1.11 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-11/) - * [1.11 to 1.12 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12/) * [1.12 to 1.13 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-13/) - + * [1.13 to 1.14 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-14/) + * [1.14 to 1.15 upgrades](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/) + _For older versions, please refer to older documentation sets on the Kubernetes website._ In Kubernetes v1.11.0 and later, you can use `kubeadm upgrade diff` to see the changes that would be @@ -30,6 +31,11 @@ applied to static pod manifests. To use kube-dns with upgrades in Kubernetes v1.13.0 and later please follow [this guide](docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-addon). +In Kubernetes v1.15.0 and later, `kubeadm upgrade apply` and `kubeadm upgrade node` will also +automatically renew the kubeadm managed certificates on this node, including those stored in kubeconfig files. +To opt-out, it is possible to pass the flag `--certificate-renewal=false`. For more details about certificate +renewal see the [certificate management documentation](docs/tasks/administer-cluster/kubeadm/kubeadm-certs). + ## kubeadm upgrade plan {#cmd-upgrade-plan} {{< include "generated/kubeadm_upgrade_plan.md" >}} @@ -39,11 +45,8 @@ To use kube-dns with upgrades in Kubernetes v1.13.0 and later please follow [thi ## kubeadm upgrade diff {#cmd-upgrade-diff} {{< include "generated/kubeadm_upgrade_diff.md" >}} -## kubeadm upgrade node config {#cmd-upgrade-node-config} -{{< include "generated/kubeadm_upgrade_node_config.md" >}} - -## kubeadm upgrade node experimental-control-plane {#cmd-experimental-control-plane} -{{< include "generated/kubeadm_upgrade_node_experimental-control-plane.md" >}} +## kubeadm upgrade node {#cmd-upgrade-node} +{{< include "generated/kubeadm_upgrade_node.md" >}} {{% /capture %}} From d9b19701c3f0356a8b5553b69d35fcb39549ce53 Mon Sep 17 00:00:00 2001 From: Guy Templeton Date: Tue, 11 Jun 2019 03:04:17 +0100 Subject: [PATCH 27/31] Update HPA Algorithm Docs for v1.15 (#14728) Also changes a link to algorithm details from initial design proposal in Github. --- .../horizontal-pod-autoscale-walkthrough.md | 2 +- .../docs/tasks/run-application/horizontal-pod-autoscale.md | 7 +++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md b/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md index d6f95f0651332..fe280a47de21c 100644 --- a/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md +++ b/content/en/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough.md @@ -81,7 +81,7 @@ controlled by the php-apache deployment we created in the first step of these in Roughly speaking, HPA will increase and decrease the number of replicas (via the deployment) to maintain an average CPU utilization across all Pods of 50% (since each pod requests 200 milli-cores by [kubectl run](https://github.com/kubernetes/kubernetes/blob/{{< param "githubbranch" >}}/docs/user-guide/kubectl/kubectl_run.md), this means average CPU usage of 100 milli-cores). -See [here](https://git.k8s.io/community/contributors/design-proposals/autoscaling/horizontal-pod-autoscaler.md#autoscaling-algorithm) for more details on the algorithm. +See [here](/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details) for more details on the algorithm. ```shell kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10 diff --git a/content/en/docs/tasks/run-application/horizontal-pod-autoscale.md b/content/en/docs/tasks/run-application/horizontal-pod-autoscale.md index 37528b30521da..4fa90e7323389 100644 --- a/content/en/docs/tasks/run-application/horizontal-pod-autoscale.md +++ b/content/en/docs/tasks/run-application/horizontal-pod-autoscale.md @@ -155,9 +155,12 @@ used. If multiple metrics are specified in a HorizontalPodAutoscaler, this calculation is done for each metric, and then the largest of the desired -replica counts is chosen. If any of those metrics cannot be converted +replica counts is chosen. If any of these metrics cannot be converted into a desired replica count (e.g. due to an error fetching the metrics -from the metrics APIs), scaling is skipped. +from the metrics APIs) and a scale down is suggested by the metrics which +can be fetched, scaling is skipped. This means that the HPA is still capable +of scaling up if one or more metrics give a `desiredReplicas` greater than +the current value. Finally, just before HPA scales the target, the scale recommendation is recorded. The controller considers all recommendations within a configurable window choosing the From 57f6eeee687e1079fdbe11ebacd411e64a4a99ca Mon Sep 17 00:00:00 2001 From: "Dr. Stefan Schimanski" Date: Tue, 11 Jun 2019 07:54:15 +0200 Subject: [PATCH 28/31] concepts/extend-kubernetes/api-extension: add 1.15 features (#14583) --- .../api-extension/custom-resources.md | 4 +- .../custom-resource-definition-versioning.md | 12 +- .../custom-resource-definitions.md | 488 ++++++++++++++++-- 3 files changed, 455 insertions(+), 49 deletions(-) diff --git a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md index fc70413535713..33143c04a5b6a 100644 --- a/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md +++ b/content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md @@ -174,7 +174,7 @@ Aggregated APIs offer more advanced API features and customization of other feat | Feature | Description | CRDs | Aggregated API | | ------- | ----------- | ---- | -------------- | | Validation | Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time. | Yes. Most validation can be specified in the CRD using [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation). Any other validations supported by addition of a [Validating Webhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook-alpha-in-1-8-beta-in-1-9). | Yes, arbitrary validation checks | -| Defaulting | See above | Yes, via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook-beta-in-1-9); Planned, via CRD OpenAPI schema. | Yes | +| Defaulting | See above | Yes, either via [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) `default` keyword (alpha in 1.15), or via a [Mutating Webhook](/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook-beta-in-1-9) | Yes | | Multi-versioning | Allows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions. | [Yes](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning) | Yes | | Custom Storage | If you need storage with a different performance mode (for example, time-series database instead of key-value store) or isolation for security (for example, encryption secrets or different | No | Yes | | Custom Business Logic | Perform arbitrary checks or actions when creating, reading, updating or deleting an object | Yes, using [Webhooks](/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks). | Yes | @@ -183,7 +183,7 @@ Aggregated APIs offer more advanced API features and customization of other feat | Other Subresources | Add operations other than CRUD, such as "logs" or "exec". | No | Yes | | strategic-merge-patch | The new endpoints support PATCH with `Content-Type: application/strategic-merge-patch+json`. Useful for updating objects that may be modified both locally, and by the server. For more information, see ["Update API Objects in Place Using kubectl patch"](/docs/tasks/run-application/update-api-object-kubectl-patch/) | No | Yes | | Protocol Buffers | The new resource supports clients that want to use Protocol Buffers | No | Yes | -| OpenAPI Schema | Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an `int` in a `string` field?) | No, but planned | Yes | +| OpenAPI Schema | Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an `int` in a `string` field?) | Yes, based on the [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) schema (beta in 1.15) | Yes | ### Common Features diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md index 8bed722a7599f..34b079a862aeb 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md @@ -29,6 +29,8 @@ level of your CustomResourceDefinitions or advance your API to a new version wit ## Overview +{{< feature-state state="beta" for_kubernetes_version="1.15" >}} + The CustomResourceDefinition API supports a `versions` field that you can use to support multiple versions of custom resources that you have developed. Versions can have different schemas with a conversion webhook to convert custom resources between versions. @@ -147,9 +149,11 @@ the version. ## Webhook conversion +{{< feature-state state="beta" for_kubernetes_version="1.15" >}} + {{< note >}} -Webhook conversion is introduced in Kubernetes 1.13 as an alpha feature. To use it, the -`CustomResourceWebhookConversion` feature should be enabled. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. +Webhook conversion is available as beta since 1.15, and as alpha since Kubernetes 1.13. The +`CustomResourceWebhookConversion` feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. {{< /note >}} The above example has a None conversion between versions which only sets the `apiVersion` field @@ -182,6 +186,10 @@ mutual TLS or other ways to authenticate the clients, see how to [authenticate API servers](/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers). {{< /note >}} +#### Permissible mutations + +A conversion webhook must not mutate anything inside of `metadata` of the converted object other than `labels` and `annotations`. Attempted changes to `name`, `UID` and `namespace` are rejected and fail the request which caused the conversion. All other changes are just ignored. + ### Deploy the conversion webhook service Documentation for deploying the conversion webhook is the same as for the [admission webhook example service](/docs/reference/access-authn-authz/extensible-admission-controllers/#deploy_the_admission_webhook_service). diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md index 807aaf15b1270..466616488788f 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md @@ -65,6 +65,20 @@ spec: # shortNames allow shorter string to match your resource on the CLI shortNames: - ct + preserveUnknownFields: false + validation: + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + image: + type: string + replicas: + type: integer ``` And create it: @@ -178,6 +192,315 @@ Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontab If you later recreate the same CustomResourceDefinition, it will start out empty. +## Specifying a structural schema + +{{< feature-state state="beta" for_kubernetes_version="1.15" >}} + +CustomResources traditionally store arbitrary JSON (next to `apiVersion`, `kind` and `metadata`, which is validated by the API server implicitly). With [OpenAPI v3.0 validation](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) a schema can be specified, which is validated during creation and updates, compare below for details and limits of such a schema. + +With `apiextensions.k8s.io/v1` the definition of a structural schema will be mandatory for CustomResourceDefinitions, while in `v1beta1` this is still optional. + +A structural schema is an [OpenAPI v3.0 validation schema](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) which: + +1. specifies a non-empty type (via `type` in OpenAPI) for the root, for each specified field of an object node (via `properties` or `additionalProperties` in OpenAPI) and for each item in an array node (via `items` in OpenAPI), with the exception of: + * a node with `x-kubernetes-int-or-string: true` + * a node with `x-kubernetes-preserve-unknown-fields: true` +2. for each each field in an object and each item in an array which is specified within any of `allOf`, `anyOf`, `oneOf` or `not`, the schema also specifies the field/item outside of those logical junctors (compare example 1 and 2). +3. does not set `description`, `type`, `default`, `additionProperties`, `nullable` within an `allOf`, `anyOf`, `oneOf` or `not`, with the exception of the two pattern for `x-kubernetes-int-or-string: true` (see below). +4. if `metadata` is specified, then only restrictions on `metadata.name` and `metadata.generateName` are allowed. + + +Non-Structural Example 1: +```yaml +allOf: +- properties: + foo: + ... +``` +conflicts with rule 2. The following would be correct: +```yaml +properties: + foo: + ... +allOf: +- properties: + foo: + ... +``` + +Non-Structural Example 2: +```yaml +allOf: +- items: + properties: + foo: + ... +``` +conflicts with rule 2. The following would be correct: +```yaml +items: + properties: + foo: + ... +allOf: +- items: + properties: + foo: + ... +``` + +Non-Structural Example 3: +```yaml +properties: + foo: + pattern: "abc" + metadata: + type: object + properties: + name: + type: string + pattern: "^a" + finalizers: + type: array + items: + type: string + pattern: "my-finalizer" +anyOf: +- properties: + bar: + type: integer + minimum: 42 + required: ["bar"] + description: "foo bar object" +``` +is not a structural schema because of the following violations: + +* the type at the root is missing (rule 1). +* the type of `foo` is missing (rule 1). +* `bar` inside of `anyOf` is not specified outside (rule 2). +* `bar`'s `type` is within `anyOf` (rule 3). +* the description is set within `anyOf` (rule 3). +* `metadata.finalizer` might not be restricted (rule 4). + +In contrast, the following, corresponding schema is structural: +```yaml +type: object +description: "foo bar object" +properties: + foo: + type: string + pattern: "abc" + bar: + type: integer + metadata: + type: object + properties: + name: + type: string + pattern: "^a" +anyOf: +- properties: + bar: + minimum: 42 + required: ["bar"] +``` + +Violations of the structural schema rules are reported in the `NonStructural` condition in the CustomResourceDefinition. + +Not being structural disables the following features: + +* [Validation Schema Publishing](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#publish-validation-schema-in-openapi-v2) +* [Webhook Conversion](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/#webhook-conversion) +* [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting) +* [Pruning](#preserving-unknown-fields) + +and possibly more features in the future. + +### Pruning versus preserving unknown fields + +{{< feature-state state="beta" for_kubernetes_version="1.15" >}} + +CustomResourceDefinitions traditionally store any (possibly validated) JSON as is in etcd. This means that unspecified fields (if there is a [OpenAPI v3.0 validation schema](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#validation) at all) are persisted. This is in contrast to native Kubernetes resources like e.g. a pod where unknown fields are dropped before being persisted to etcd. We call this "pruning" of unknown fields. + +If a [structural OpenAPI v3 validation schema](#specifying-a-structural-schema) is defined (either in the global `spec.validation.openAPIV3Schema` or for each version) in a CustomResourceDefinition, pruning can be enabled by setting `spec.preserveUnknownFields` to `false`. Then unspecified fields on creation and on update are dropped. + +Compare the CustomResourceDefinition `crontabs.stable.example.com` above. It has pruning enabled. Hence, if you save the following YAML to `my-crontab.yaml`: + +```yaml +apiVersion: "stable.example.com/v1" +kind: CronTab +metadata: + name: my-new-cron-object +spec: + cronSpec: "* * * * */5" + image: my-awesome-cron-image + someRandomField: 42 +``` + +and create it: + +```shell +kubectl create --validate=false -f my-crontab.yaml -o yaml +``` + +you should get: + +```console +apiVersion: stable.example.com/v1 +kind: CronTab +metadata: + creationTimestamp: 2017-05-31T12:56:35Z + generation: 1 + name: my-new-cron-object + namespace: default + resourceVersion: "285" + selfLink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object + uid: 9423255b-4600-11e7-af6a-28d2447dc82b +spec: + cronSpec: '* * * * */5' + image: my-awesome-cron-image +``` + +The field `someRandomField` has been pruned. + +Note that the `kubectl create` call uses `--validate=false` to skip client-side validation. Because the [OpenAPI validation schemas are also published](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#publish-validation-schema-in-openapi-v2) to kubectl, it will also check for unknown fields and reject those objects long before they are sent to the API server. + +In `apiextensions.k8s.io/v1beta1`, pruning is disabled by default, i.e. `spec.preserveUnknownFields` defaults to `true`. In `apiextensions.k8s.io/v1` no new CustomResourceDefinitions with `spec.preserveUnknownFields: true` will be allowed to be created. + +### Controlling pruning + +With `spec.preserveUnknownField: false` in the CustomResourceDefinition, pruning is enabled for all custom resources of that type and in all versions. It is possible though to opt-out of that for JSON sub-trees via `x-kubernetes-preserve-unknown-fields: true` in the [structural OpenAPI v3 validation schema](#specifying-a-structural-schema): + +```yaml +type: object +properties: + json: + x-kubernetes-preserve-unknown-fields: true +``` + +The field `json` can store any JSON value, without anything being pruned. + +It is possible to partially specify the permitted JSON, e.g.: + +```yaml +type: object +properties: + json: + x-kubernetes-preserve-unknown-fields: true + type: object + description: this is arbitrary JSON +``` + +With this only object type values are allowed. + +Pruning is enabled again for each specified property (or `additionalProperties`): + +```yaml +type: object +properties: + json: + x-kubernetes-preserve-unknown-fields: true + type: object + properties: + spec: + type: object + properties: + foo: + type: string + bar: + type: string +``` + +With this, the value: + +```yaml +json: + spec: + foo: abc + bar: def + something: x + status: + something: x +``` + +is pruned to: + +```yaml +json: + spec: + foo: abc + bar: def + status: + something: x +``` + +This means that the `something` field in the specified `spec` object is pruned, but everything outside is not. + +### IntOrString + +Nodes in a schema with `x-kubernetes-int-or-string: true` are excluded from rule 1, such that the following is structural: + +```yaml +type: object +properties: + foo: + x-kubernetes-int-or-string: true +``` + +Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed (exactly those, without variations in order to additional fields): + +```yaml +x-kubernetes-int-or-string: true +anyOf: +- type: integer +- type: string +... +``` + +and + +```yaml +x-kubernetes-int-or-string: true +allOf: +- anyOf: + - type: integer + - type: string +- ... # zero or more +... +``` + +With one of those specification, both an integer and a string validate. + +In [Validation Schema Publishing](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#publish-validation-schema-in-openapi-v2), `x-kubernetes-int-or-string: true` is unfolded to one of the two patterns shown above. + +### RawExtension + +RawExtensions (as in `runtime.RawExtension` defined in [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery/blob/03ac7a9ade429d715a1a46ceaa3724c18ebae54f/pkg/runtime/types.go#L94)) holds complete Kubernetes objects, i.e. with `apiVersion` and `kind` fields. + +It is possible to specify those embedded objects (both completely without constraints or partially specified) by setting `x-kubernetes-embedded-resource: true`. For example: + +```yaml +type: object +properties: + foo: + x-kubernetes-embedded-resource: true + x-kubernetes-preserve-unknown-fields: true +``` + +Here, the field `foo` holds a complete object, e.g.: + +```yaml +foo: + apiVersion: v1 + kind: Pod + spec: + ... +``` + +Because `x-kubernetes-preserve-unknown-fields: true` is specified alongside, nothing is pruned. The use of `x-kubernetes-preserve-unknown-fields: true` is optional though. + +With `x-kubernetes-embedded-resource: true`, the `apiVersion`, `kind` and `metadata` are implicitly specified and validated. + ## Serving multiple versions of a CRD See [Custom resource definition versioning](/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/) @@ -231,17 +554,31 @@ Validation of custom objects is possible via [OpenAPI v3 schema](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schemaObject) or [validatingadmissionwebhook](/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook). Additionally, the following restrictions are applied to the schema: -- The fields `default`, `nullable`, `discriminator`, `readOnly`, `writeOnly`, `xml`, -`deprecated` and `$ref` cannot be set. +- These fields cannot be set: + - `definitions`, + - `dependencies`, + - `deprecated`, + - `discriminator`, + - `id`, + - `patternProperties`, + - `readOnly`, + - `writeOnly`, + - `xml`, + - `$ref`. - The field `uniqueItems` cannot be set to true. - The field `additionalProperties` cannot be set to false. +- The field `additionalProperties` is mutually exclusive with `properties`. -You can disable this feature using the `CustomResourceValidation` feature gate on -the [kube-apiserver](/docs/admin/kube-apiserver): +These fields can only be set with specific features enabled: -``` ---feature-gates=CustomResourceValidation=false -``` +- `default`: the `CustomResourceDefaulting` feature gate must be enabled, compare [Validation Schema Defaulting](/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/#defaulting). + +Note: compare with [structural schemas](#specifying-a-structural-schema) for further restriction required for certain CustomResourceDefinition features. + +{{< note >}} +OpenAPI v3 validation is available as beta. The +`CustomResourceValidation` feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. +{{< /note >}} The schema is defined in the CustomResourceDefinition. In the following example, the CustomResourceDefinition applies the following validations on the custom object: @@ -273,8 +610,10 @@ spec: validation: # openAPIV3Schema is the schema for validating custom objects. openAPIV3Schema: + type: object properties: spec: + type: object properties: cronSpec: type: string @@ -347,25 +686,106 @@ kubectl apply -f my-crontab.yaml crontab "my-new-cron-object" created ``` -### Publish Validation Schema in OpenAPI v2 +### Defaulting -{{< feature-state state="alpha" for_kubernetes_version="1.14" >}} +{{< feature-state state="alpha" for_kubernetes_version="1.15" >}} -Starting with Kubernetes 1.14, [custom resource validation schema](#validation) can be published as part -of [OpenAPI v2 spec](/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions) from -Kubernetes API server. +{{< note >}} +Defaulting is available as alpha since 1.15. It is disabled by default and can be enabled via the `CustomResourceDefaulting` feature gate. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. -[kubectl](/docs/reference/kubectl/overview) consumes the published schema to perform client-side validation -(`kubectl create` and `kubectl apply`), schema explanation (`kubectl explain`) on custom resources. -The published schema can be consumed for other purposes. The feature is Alpha in 1.14 and disabled by default. -You can enable the feature using the `CustomResourcePublishOpenAPI` feature gate on the -[kube-apiserver](/docs/admin/kube-apiserver): +Defaulting also requires a structural schema and pruning. +{{< /note >}} +Defaulting allows to specify default values in the [OpenAPI v3 validation schema](#validation): + +```yaml +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: crontabs.stable.example.com +spec: + group: stable.example.com + versions: + - name: v1 + served: true + storage: true + version: v1 + scope: Namespaced + names: + plural: crontabs + singular: crontab + kind: CronTab + shortNames: + - ct + preserveUnknownFields: false + validation: + # openAPIV3Schema is the schema for validating custom objects. + openAPIV3Schema: + type: object + properties: + spec: + type: object + properties: + cronSpec: + type: string + pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$' + default: "5 0 * * *" + image: + type: string + replicas: + type: integer + minimum: 1 + maximum: 10 + default: 1 ``` ---feature-gates=CustomResourcePublishOpenAPI=true + +With this both `cronSpec` and `replicas` are defaulted: + +```yaml +apiVersion: "stable.example.com/v1" +kind: CronTab +metadata: + name: my-new-cron-object +spec: + image: my-awesome-cron-image ``` -Custom resource validation schema will be converted to OpenAPI v2 schema, and +leads to + +```yaml +apiVersion: "stable.example.com/v1" +kind: CronTab +metadata: + name: my-new-cron-object +spec: + cronSpec: "5 0 * * *" + image: my-awesome-cron-image + replaces: 1 +``` + +Note that defaulting happens on the object + +* in the request to the API server using the request version defaults +* when reading from etcd using the storage version defaults +* after mutating admission plugins with non-empty patches using the admission webhook object version defaults. + +Note that defaults applied when reading data from etcd are not automatically written back to etcd. An update request via the API is required to persist those defaults back into etcd. + +### Publish Validation Schema in OpenAPI v2 + +{{< feature-state state="beta" for_kubernetes_version="1.15" >}} + +{{< note >}} +OpenAPI v2 Publishing is available as beta since 1.15, and as alpha since 1.14. The +`CustomResourcePublishOpenAPI` feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) documentation for more information. +{{< /note >}} + +With the OpenAPI v2 Publishing feature enabled, CustomResourceDefinition [OpenAPI v3 validation schemas](#validation) which are [structural](#specifying-a-structural-schema) are published as part +of the [OpenAPI v2 spec](/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions) from Kubernetes API server. + +[kubectl](/docs/reference/kubectl/overview) consumes the published schema to perform client-side validation (`kubectl create` and `kubectl apply`), schema explanation (`kubectl explain`) on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation. + +The OpenAPI v3 validation schema is converted to OpenAPI v2 schema, and show up in `definitions` and `paths` fields in the [OpenAPI v2 spec](/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions). The following modifications are applied during the conversion to keep backwards compatiblity with kubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejecting @@ -373,31 +793,8 @@ valid OpenAPI schemas that it doesn't understand. The conversion won't modify th and therefore won't affect [validation](#validation) in the API server. 1. The following fields are removed as they aren't supported by OpenAPI v2 (in future versions OpenAPI v3 will be used without these restrictions) - - The fields `oneOf`, `anyOf` and `not` are removed -2. The following fields are removed as they aren't allowed by kubectl in - previous 1.13 version - - For a schema with a `$ref` - - the fields `properties` and `type` are removed - - if the `$ref` is outside of the `definitions`, the field `$ref` is removed - - For a schema of a primitive data type (which means the field `type` has two elements: one type and one format) - - if any one of the two elements is `null`, the field `type` is removed - - otherwise, the fields `type` and `properties` are removed - - For a schema of more than two types - - the fields `type` and `properties` are removed - - For a schema of `null` type - - the field `type` is removed - - For a schema of `array` type - - if the schema doesn't have exactly one item, the fields `type` and `items` are - removed - - For a schema with no type specified - - the field `properties` is removed -3. The following fields are removed as they aren't supported by the OpenAPI protobuf implementation - - The fields `id`, `schema`, `definitions`, `additionalItems`, `dependencies`, - and `patternProperties` are removed - - For a schema with a `externalDocs` - - if the `externalDocs` has `url` defined, the field `externalDocs` is removed - - For a schema with `items` defined - - if the field `items` has multiple schemas, the field `items` is removed + - The fields `allOf`, `anyOf`, `oneOf` and `not` are removed +2. If `nullable: true` is set, we drop `type`, `nullable`, `items` and `properties` because OpenAPI v2 is not able to express nullable. To avoid kubectl to reject good objects, this is necessary. ### Additional printer columns @@ -569,9 +966,10 @@ the status replica value in the `/scale` subresource will default to 0. - It is an optional value. - It must be set to work with HPA. - - Only JSONPaths under `.status` and with the dot notation are allowed. + - Only JSONPaths under `.status` or `.spec` and with the dot notation are allowed. - If there is no value under the `LabelSelectorPath` in the custom resource, the status selector value in the `/scale` subresource will default to the empty string. + - The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form. In the following example, both status and scale subresources are enabled. From 71a7828152853432f9d44857dc4322c0aea978f5 Mon Sep 17 00:00:00 2001 From: "Lubomir I. Ivanov" Date: Tue, 11 Jun 2019 15:10:19 +0300 Subject: [PATCH 29/31] kubeadm-tasks: include v1.14->v1.15 upgrade document (#14595) Remove v1.11 upgrade documents. --- .../kubeadm/kubeadm-upgrade-1-12.md | 300 -------------- .../kubeadm/kubeadm-upgrade-1-15.md | 383 ++++++++++++++++++ .../kubeadm/kubeadm-upgrade-ha-1-12.md | 247 ----------- 3 files changed, 383 insertions(+), 547 deletions(-) delete mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12.md create mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15.md delete mode 100644 content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12.md deleted file mode 100644 index fa8831a92ca05..0000000000000 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -reviewers: -- sig-cluster-lifecycle -title: Upgrading kubeadm clusters from v1.11 to v1.12 -content_template: templates/task ---- - -{{% capture overview %}} - -This page explains how to upgrade a Kubernetes cluster created with `kubeadm` from version 1.11.x to version 1.12.x, and from version 1.12.x to 1.12.y, where `y > x`. - -{{% /capture %}} - -{{% capture prerequisites %}} - -- You need to have a `kubeadm` Kubernetes cluster running version 1.11.0 or later. - [Swap must be disabled][swap]. - The cluster should use a static control plane and etcd pods. -- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md) carefully. -- Make sure to back up any important components, such as app-level state stored in a database. - `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice. - - -[swap]: https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux -### Additional information - -- All containers are restarted after upgrade, because the container spec hash value is changed. -- You can upgrade only from one minor version to the next minor version. - That is, you cannot skip versions when you upgrade. - For example, you can upgrade only from 1.10 to 1.11, not from 1.9 to 1.11. - -{{% /capture %}} - -{{% capture steps %}} - -## Upgrade the control plane - -1. On your master node, upgrade kubeadm: - - {{< tabs name="k8s_install" >}} - {{% tab name="Ubuntu, Debian or HypriotOS" %}} - # replace "x" with the latest patch version - apt-mark unhold kubeadm && \ - apt-get update && apt-get upgrade -y kubeadm=1.12.x-00 && \ - apt-mark hold kubeadm - {{% /tab %}} - {{% tab name="CentOS, RHEL or Fedora" %}} - # replace "x" with the latest patch version - yum upgrade -y kubeadm-1.12.x --disableexcludes=kubernetes - {{% /tab %}} - {{< /tabs >}} - -1. Verify that the download works and has the expected version: - - ```shell - kubeadm version - ``` - -1. On the master node, run: - - ```shell - kubeadm upgrade plan - ``` - - You should see output similar to this: - - ```shell - [preflight] Running pre-flight checks. - [upgrade] Making sure the cluster is healthy: - [upgrade/config] Making sure the configuration is correct: - [upgrade/config] Reading configuration from the cluster... - [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' - [upgrade] Fetching available versions to upgrade to - [upgrade/versions] Cluster version: v1.11.3 - [upgrade/versions] kubeadm version: v1.12.0 - [upgrade/versions] Latest stable version: v1.11.3 - [upgrade/versions] Latest version in the v1.11 series: v1.11.3 - [upgrade/versions] Latest experimental version: v1.13.0-alpha.0 - - Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': - COMPONENT CURRENT AVAILABLE - Kubelet 2 x v1.11.1 v1.12.0 - 1 x v1.11.3 v1.12.0 - - Upgrade to the latest experimental version: - - COMPONENT CURRENT AVAILABLE - API Server v1.11.3 v1.12.0 - Controller Manager v1.11.3 v1.12.0 - Scheduler v1.11.3 v1.12.0 - Kube Proxy v1.11.3 v1.12.0 - CoreDNS 1.1.3 1.2.2 - Etcd 3.2.18 3.2.24 - - You can now apply the upgrade by executing the following command: - - kubeadm upgrade apply v1.12.0 - - _____________________________________________________________________ - - ``` - - This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to. - -1. Choose a version to upgrade to, and run the appropriate command. For example: - - ```shell - kubeadm upgrade apply v1.12.0 - ``` - - You should see output similar to this: - - - - ```shell - [preflight] Running pre-flight checks. - [upgrade] Making sure the cluster is healthy: - [upgrade/config] Making sure the configuration is correct: - [upgrade/config] Reading configuration from the cluster... - [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' - [upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file. - [upgrade/version] You have chosen to change the cluster version to "v1.12.0" - [upgrade/versions] Cluster version: v1.11.3 - [upgrade/versions] kubeadm version: v1.12.0 - [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y - [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd] - [upgrade/prepull] Prepulling image for component etcd. - [upgrade/prepull] Prepulling image for component kube-apiserver. - [upgrade/prepull] Prepulling image for component kube-controller-manager. - [upgrade/prepull] Prepulling image for component kube-scheduler. - [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd - [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver - [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler - [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager - [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd - [upgrade/prepull] Prepulled image for component kube-apiserver. - [upgrade/prepull] Prepulled image for component kube-controller-manager. - [upgrade/prepull] Prepulled image for component kube-scheduler. - [upgrade/prepull] Prepulled image for component etcd. - [upgrade/prepull] Successfully prepulled the images for all the control plane components - [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.0"... - Static pod: kube-apiserver-ip-172-31-80-76 hash: d9b7af93990d702b3ee9a2beca93384b - Static pod: kube-controller-manager-ip-172-31-80-76 hash: 44a081fb5d26e90773ceb98b4e16fe10 - Static pod: kube-scheduler-ip-172-31-80-76 hash: 009228e74aef4d7babd7968782118d5e - Static pod: etcd-ip-172-31-80-76 hash: 997fcf3d8d974c98abc14556cc02617e - [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests661777755/etcd.yaml" - [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-09-19-18-58-14/etcd.yaml" - [upgrade/staticpods] Waiting for the kubelet to restart the component - [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s - Static pod: etcd-ip-172-31-80-76 hash: 997fcf3d8d974c98abc14556cc02617e - - [apiclient] Found 1 Pods for label selector component=etcd - [upgrade/staticpods] Component "etcd" upgraded successfully! - [upgrade/etcd] Waiting for etcd to become available - [util/etcd] Waiting 0s for initial delay - [util/etcd] Attempting to see if all cluster endpoints are available 1/10 - [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests661777755" - [controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests661777755/kube-apiserver.yaml" - [controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests661777755/kube-controller-manager.yaml" - [controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests661777755/kube-scheduler.yaml" - [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-09-19-18-58-14/kube-apiserver.yaml" - [upgrade/staticpods] Waiting for the kubelet to restart the component - [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s - - Static pod: kube-apiserver-ip-172-31-80-76 hash: 854a5a8468f899093c6a967bb81dcfbc - [apiclient] Found 1 Pods for label selector component=kube-apiserver - [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! - [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-09-19-18-58-14/kube-controller-manager.yaml" - [upgrade/staticpods] Waiting for the kubelet to restart the component - [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s - Static pod: kube-controller-manager-ip-172-31-80-76 hash: 44a081fb5d26e90773ceb98b4e16fe10 - Static pod: kube-controller-manager-ip-172-31-80-76 hash: b651f83474ae70031d5fb2cab73bd366 - [apiclient] Found 1 Pods for label selector component=kube-controller-manager - [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! - [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2018-09-19-18-58-14/kube-scheduler.yaml" - [upgrade/staticpods] Waiting for the kubelet to restart the component - [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s - Static pod: kube-scheduler-ip-172-31-80-76 hash: 009228e74aef4d7babd7968782118d5e - Static pod: kube-scheduler-ip-172-31-80-76 hash: da406e5a49adfbbeb90fe2a0cf8fd8d1 - [apiclient] Found 1 Pods for label selector component=kube-scheduler - [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! - [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace - [kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster - [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.12" ConfigMap in the kube-system namespace - [kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" - [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "ip-172-31-80-76" as an annotation - [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials - [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token - [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster - [addons] Applied essential addon: CoreDNS - [addons] Applied essential addon: kube-proxy - - [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0". Enjoy! - - [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. - ``` - -1. Manually upgrade your Software Defined Network (SDN). - - Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. - Check the [addons](/docs/concepts/cluster-administration/addons/) page to - find your CNI provider and see whether additional upgrade steps are required. - -## Upgrade master and node packages - -1. Prepare each node for maintenance, marking it unschedulable and evicting the workloads: - - ```shell - kubectl drain $NODE --ignore-daemonsets - ``` - - On the master node, you must add `--ignore-daemonsets`: - - ```shell - kubectl drain ip-172-31-85-18 - node "ip-172-31-85-18" cordoned - error: unable to drain node "ip-172-31-85-18", aborting command... - - There are pending nodes to be drained: - ip-172-31-85-18 - error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-5798d, kube-proxy-thjp9 - ``` - - ``` - kubectl drain ip-172-31-85-18 --ignore-daemonsets - node "ip-172-31-85-18" already cordoned - WARNING: Ignoring DaemonSet-managed pods: calico-node-5798d, kube-proxy-thjp9 - node "ip-172-31-85-18" drained - ``` - -1. Upgrade the Kubernetes package version on each `$NODE` node by running the Linux package manager for your distribution: - - {{< tabs name="k8s_upgrade" >}} - {{% tab name="Ubuntu, Debian or HypriotOS" %}} - # replace "x" with the latest patch version - apt-get update - apt-get upgrade -y kubelet=1.12.x-00 kubeadm=1.12.x-00 - {{% /tab %}} - {{% tab name="CentOS, RHEL or Fedora" %}} - # replace "x" with the latest patch version - yum upgrade -y kubelet-1.12.x kubeadm-1.12.x --disableexcludes=kubernetes - {{% /tab %}} - {{< /tabs >}} - -## Upgrade kubelet on each node - -1. On each node except the master node, upgrade the kubelet config: - - ```shell - sudo kubeadm upgrade node config --kubelet-version $(kubelet --version | cut -d ' ' -f 2) - ``` - -1. Restart the kubelet process: - - ```shell - sudo systemctl restart kubelet - ``` - -1. Verify that the new version of the `kubelet` is running on the node: - - ```shell - systemctl status kubelet - ``` - -1. Bring the node back online by marking it schedulable: - - ```shell - kubectl uncordon $NODE - ``` - -1. After the kubelet is upgraded on all nodes, verify that all nodes are available again by running the following command from anywhere kubectl can access the cluster: - - ```shell - kubectl get nodes - ``` - - The `STATUS` column should show `Ready` for all your nodes, and the version number should be updated. - -{{% /capture %}} - -## Recovering from a failure state - -If `kubeadm upgrade` fails and does not roll back, for example because of an unexpected shutdown during execution, you can run `kubeadm upgrade` again. -This command is idempotent and eventually makes sure that the actual state is the desired state you declare. - -To recover from a bad state, you can also run `kubeadm upgrade --force` without changing the version that your cluster is running. - -## How it works - -`kubeadm upgrade apply` does the following: - -- Checks that your cluster is in an upgradeable state: - - The API server is reachable - - All nodes are in the `Ready` state - - The control plane is healthy -- Enforces the version skew policies. -- Makes sure the control plane images are available or available to pull to the machine. -- Upgrades the control plane components or rollbacks if any of them fails to come up. -- Applies the new `kube-dns` and `kube-proxy` manifests and enforces that all necessary RBAC rules are created. -- Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15.md new file mode 100644 index 0000000000000..22a6b8f633120 --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15.md @@ -0,0 +1,383 @@ +--- +reviewers: +- sig-cluster-lifecycle +title: Upgrading kubeadm clusters from v1.14 to v1.15 +content_template: templates/task +--- + +{{% capture overview %}} + +This page explains how to upgrade a Kubernetes cluster created with kubeadm from version +1.14.x to version 1.15.x, and from version 1.15.x to 1.15.y (where `y > x`). + +The upgrade workflow at high level is the following: + +1. Upgrade the primary control plane node. +1. Upgrade additional control plane nodes. +1. Upgrade worker nodes. + +{{% /capture %}} + +{{% capture prerequisites %}} + +- You need to have a kubeadm Kubernetes cluster running version 1.14.0 or later. +- [Swap must be disabled](https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux). +- The cluster should use a static control plane and etcd pods or external etcd. +- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md) carefully. +- Make sure to back up any important components, such as app-level state stored in a database. + `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice. + +### Additional information + +- All containers are restarted after upgrade, because the container spec hash value is changed. +- You only can upgrade from one MINOR version to the next MINOR version, + or between PATCH versions of the same MINOR. That is, you cannot skip MINOR versions when you upgrade. + For example, you can upgrade from 1.y to 1.y+1, but not from 1.y to 1.y+2. + +{{% /capture %}} + +{{% capture steps %}} + +## Determine which version to upgrade to + +1. Find the latest stable 1.15 version: + + {{< tabs name="k8s_install_versions" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + apt update + apt-cache policy kubeadm + # find the latest 1.15 version in the list + # it should look like 1.15.x-00, where x is the latest patch + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + yum list --showduplicates kubeadm --disableexcludes=kubernetes + # find the latest 1.15 version in the list + # it should look like 1.15.x-0, where x is the latest patch + {{% /tab %}} + {{< /tabs >}} + +## Upgrade the first control plane node + +1. On your first control plane node, upgrade kubeadm: + + {{< tabs name="k8s_install_kubeadm_first_cp" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + # replace x in 1.15.x-00 with the latest patch version + apt-mark unhold kubeadm && \ + apt-get update && apt-get install -y kubeadm=1.15.x-00 && \ + apt-mark hold kubeadm + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + # replace x in 1.15.x-0 with the latest patch version + yum install -y kubeadm-1.15.x-0 --disableexcludes=kubernetes + {{% /tab %}} + {{< /tabs >}} + +1. Verify that the download works and has the expected version: + + ```shell + kubeadm version + ``` + +1. On the control plane node, run: + + ```shell + sudo kubeadm upgrade plan + ``` + + You should see output similar to this: + + ```shell + [upgrade/config] Making sure the configuration is correct: + [upgrade/config] Reading configuration from the cluster... + [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' + [preflight] Running pre-flight checks. + [upgrade] Making sure the cluster is healthy: + [upgrade] Fetching available versions to upgrade to + [upgrade/versions] Cluster version: v1.14.2 + [upgrade/versions] kubeadm version: v1.15.0 + + Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': + COMPONENT CURRENT AVAILABLE + Kubelet 1 x v1.14.2 v1.15.0 + + Upgrade to the latest version in the v1.15 series: + + COMPONENT CURRENT AVAILABLE + API Server v1.14.2 v1.15.0 + Controller Manager v1.14.2 v1.15.0 + Scheduler v1.14.2 v1.15.0 + Kube Proxy v1.14.2 v1.15.0 + CoreDNS 1.3.1 1.3.1 + Etcd 3.3.10 3.3.10 + + You can now apply the upgrade by executing the following command: + + kubeadm upgrade apply v1.15.0 + + _____________________________________________________________________ + ``` + + This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to. + + {{< note >}} + With the release of Kubernetes v1.15, `kubeadm upgrade` also automatically renews + the certificates that it manages on this node. To opt-out of certificate renewal the flag `--certificate-renewal=false` can be used. + For more information see the [certificate management guide](/docs/tasks/administer-cluster/kubeadm/kubeadm-certs). + {{}} + +1. Choose a version to upgrade to, and run the appropriate command. For example: + + ```shell + sudo kubeadm upgrade apply v1.15.x + ``` + + - Replace `x` with the patch version you picked for this upgrade. + + You should see output similar to this: + + ```shell + [preflight] Running pre-flight checks. + [upgrade] Making sure the cluster is healthy: + [upgrade/config] Making sure the configuration is correct: + [upgrade/config] Reading configuration from the cluster... + [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' + [upgrade/version] You have chosen to change the cluster version to "v1.15.0" + [upgrade/versions] Cluster version: v1.14.2 + [upgrade/versions] kubeadm version: v1.15.0 + [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y + [upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd] + [upgrade/prepull] Prepulling image for component etcd. + [upgrade/prepull] Prepulling image for component kube-apiserver. + [upgrade/prepull] Prepulling image for component kube-controller-manager. + [upgrade/prepull] Prepulling image for component kube-scheduler. + [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager + [apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler + [apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd + [upgrade/prepull] Prepulled image for component etcd. + [upgrade/prepull] Prepulled image for component kube-controller-manager. + [upgrade/prepull] Prepulled image for component kube-apiserver. + [upgrade/prepull] Prepulled image for component kube-scheduler. + [upgrade/prepull] Successfully prepulled the images for all the control plane components + [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.15.0"... + Static pod: kube-apiserver-luboitvbox hash: 8d931c2296a38951e95684cbcbe3b923 + Static pod: kube-controller-manager-luboitvbox hash: 2480bf6982ad2103c05f6764e20f2787 + Static pod: kube-scheduler-luboitvbox hash: 9b290132363a92652555896288ca3f88 + [upgrade/etcd] Upgrading to TLS for etcd + [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests446257614" + [upgrade/staticpods] Preparing for "kube-apiserver" upgrade + [upgrade/staticpods] Renewing "apiserver-etcd-client" certificate + [upgrade/staticpods] Renewing "apiserver" certificate + [upgrade/staticpods] Renewing "apiserver-kubelet-client" certificate + [upgrade/staticpods] Renewing "front-proxy-client" certificate + [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-06-05-23-38-03/kube-apiserver.yaml" + [upgrade/staticpods] Waiting for the kubelet to restart the component + [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) + Static pod: kube-apiserver-luboitvbox hash: 8d931c2296a38951e95684cbcbe3b923 + Static pod: kube-apiserver-luboitvbox hash: 1b4e2b09a408c844f9d7b535e593ead9 + [apiclient] Found 1 Pods for label selector component=kube-apiserver + [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! + [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade + [upgrade/staticpods] Renewing certificate embedded in "controller-manager.conf" + [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-06-05-23-38-03/kube-controller-manager.yaml" + [upgrade/staticpods] Waiting for the kubelet to restart the component + [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) + Static pod: kube-controller-manager-luboitvbox hash: 2480bf6982ad2103c05f6764e20f2787 + Static pod: kube-controller-manager-luboitvbox hash: 6617d53423348aa619f1d6e568bb894a + [apiclient] Found 1 Pods for label selector component=kube-controller-manager + [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! + [upgrade/staticpods] Preparing for "kube-scheduler" upgrade + [upgrade/staticpods] Renewing certificate embedded in "scheduler.conf" + [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-06-05-23-38-03/kube-scheduler.yaml" + [upgrade/staticpods] Waiting for the kubelet to restart the component + [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) + Static pod: kube-scheduler-luboitvbox hash: 9b290132363a92652555896288ca3f88 + Static pod: kube-scheduler-luboitvbox hash: edf58ab819741a5d1eb9c33de756e3ca + [apiclient] Found 1 Pods for label selector component=kube-scheduler + [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! + [upgrade/staticpods] Renewing certificate embedded in "admin.conf" + [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace + [kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster + [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace + [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" + [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials + [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token + [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster + [addons] Applied essential addon: CoreDNS + [addons] Applied essential addon: kube-proxy + + [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.15.0". Enjoy! + + [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. + ``` + +1. Manually upgrade your CNI provider plugin. + + Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow. + Check the [addons](/docs/concepts/cluster-administration/addons/) page to + find your CNI provider and see whether additional upgrade steps are required. + +1. Upgrade the kubelet and kubectl on the control plane node: + + {{< tabs name="k8s_install_kubelet" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + # replace x in 1.15.x-00 with the latest patch version + apt-mark unhold kubelet && \ + apt-get update && apt-get install -y kubelet=1.15.x-00 kubectl=1.15.x-00 && \ + apt-mark hold kubelet + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + # replace x in 1.15.x-0 with the latest patch version + yum install -y kubelet-1.15.x-0 kubectl-1.15.x-0 --disableexcludes=kubernetes + {{% /tab %}} + {{< /tabs >}} + +1. Restart the kubelet + + ```shell + sudo systemctl restart kubelet + ``` + +## Upgrade additional control plane nodes + +1. Same as the first control plane node but use: + +``` +sudo kubeadm upgrade node +``` + +instead of: + +``` +sudo kubeadm upgrade apply +``` + +Also `sudo kubeadm upgrade plan` is not needed. + +## Upgrade worker nodes + +The upgrade procedure on worker nodes should be executed one node at a time or few nodes at a time, +without compromising the minimum required capacity for running your workloads. + +### Upgrade kubeadm + +1. Upgrade kubeadm on all worker nodes: + + {{< tabs name="k8s_install_kubeadm_worker_nodes" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + # replace x in 1.15.x-00 with the latest patch version + apt-mark unhold kubeadm && \ + apt-get update && apt-get install -y kubeadm=1.15.x-00 && \ + apt-mark hold kubeadm + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + # replace x in 1.15.x-0 with the latest patch version + yum install -y kubeadm-1.15.x-0 --disableexcludes=kubernetes + {{% /tab %}} + {{< /tabs >}} + +### Cordon the node + +1. Prepare the node for maintenance by marking it unschedulable and evicting the workloads. Run: + + ```shell + kubectl drain $NODE --ignore-daemonsets + ``` + + You should see output similar to this: + + ```shell + kubectl drain ip-172-31-85-18 + node "ip-172-31-85-18" cordoned + error: unable to drain node "ip-172-31-85-18", aborting command... + + There are pending nodes to be drained: + ip-172-31-85-18 + error: DaemonSet-managed pods (use --ignore-daemonsets to ignore): calico-node-5798d, kube-proxy-thjp9 + ``` + +### Upgrade the kubelet configuration + +1. Call the following command: + + ```shell + sudo kubeadm upgrade node + ``` + +### Upgrade kubelet and kubectl + +1. Upgrade the Kubernetes package version by running the Linux package manager for your distribution: + + {{< tabs name="k8s_kubelet_and_kubectl" >}} + {{% tab name="Ubuntu, Debian or HypriotOS" %}} + # replace x in 1.15.x-00 with the latest patch version + apt-get update + apt-get install -y kubelet=1.15.x-00 kubectl=1.15.x-00 + {{% /tab %}} + {{% tab name="CentOS, RHEL or Fedora" %}} + # replace x in 1.15.x-0 with the latest patch version + yum install -y kubelet-1.15.x-0 kubectl-1.15.x-0 --disableexcludes=kubernetes + {{% /tab %}} + {{< /tabs >}} + +1. Restart the kubelet + + ```shell + sudo systemctl restart kubelet + ``` + +### Uncordon the node + +1. Bring the node back online by marking it schedulable: + + ```shell + kubectl uncordon $NODE + ``` + +## Verify the status of the cluster + +After the kubelet is upgraded on all nodes verify that all nodes are available again by running the following command from anywhere kubectl can access the cluster: + +```shell +kubectl get nodes +``` + +The `STATUS` column should show `Ready` for all your nodes, and the version number should be updated. + +{{% /capture %}} + +## Recovering from a failure state + +If `kubeadm upgrade` fails and does not roll back, for example because of an unexpected shutdown during execution, you can run `kubeadm upgrade` again. +This command is idempotent and eventually makes sure that the actual state is the desired state you declare. + +To recover from a bad state, you can also run `kubeadm upgrade --force` without changing the version that your cluster is running. + +## How it works + +`kubeadm upgrade apply` does the following: + +- Checks that your cluster is in an upgradeable state: + - The API server is reachable + - All nodes are in the `Ready` state + - The control plane is healthy +- Enforces the version skew policies. +- Makes sure the control plane images are available or available to pull to the machine. +- Upgrades the control plane components or rollbacks if any of them fails to come up. +- Applies the new `kube-dns` and `kube-proxy` manifests and makes sure that all necessary RBAC rules are created. +- Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days. + +`kubeadm upgrade node` does the following on additional control plane nodes: + +- Fetches the kubeadm `ClusterConfiguration` from the cluster. +- Optionally backups the kube-apiserver certificate. +- Upgrades the static Pod manifests for the control plane components. +- Upgrades the kubelet configuration for this node. + +`kubeadm upgrade node` does the following on worker nodes: + +- Fetches the kubeadm `ClusterConfiguration` from the cluster. +- Upgrades the kubelet configuration for this node. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md b/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md deleted file mode 100644 index a8a9bbb43987e..0000000000000 --- a/content/en/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha-1-12.md +++ /dev/null @@ -1,247 +0,0 @@ ---- -reviewers: -- jamiehannaford -- luxas -- timothysc -- jbeda -title: Upgrading kubeadm HA clusters from v1.11 to v1.12 -content_template: templates/task ---- - -{{% capture overview %}} - -This page explains how to upgrade a highly available (HA) Kubernetes cluster created with `kubeadm` from version 1.11.x to version 1.12.x. In addition to upgrading, you must also follow the instructions in [Creating HA clusters with kubeadm](/docs/setup/independent/high-availability/). - -{{% /capture %}} - -{{% capture prerequisites %}} - -Before proceeding: - -- You need to have a `kubeadm` HA cluster running version 1.11 or higher. -- Make sure you read the [release notes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md) carefully. -- Make sure to back up any important components, such as app-level state stored in a database. `kubeadm upgrade` does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice. -- Check the prerequisites for [Upgrading/downgrading kubeadm clusters between v1.11 to v1.12](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-12/). - -{{< note >}} -All commands on any control plane or etcd node should be run as root. -{{< /note >}} - -{{% /capture %}} - -{{% capture steps %}} - -## Prepare for both methods - -Upgrade `kubeadm` to the version that matches the version of Kubernetes that you are upgrading to: - -```shell -apt-mark unhold kubeadm && \ -apt-get update && apt-get upgrade -y kubeadm && \ -apt-mark hold kubeadm -``` - -Check prerequisites and determine the upgrade versions: - -```shell -kubeadm upgrade plan -``` - -You should see something like the following: - - Upgrade to the latest stable version: - - COMPONENT CURRENT AVAILABLE - API Server v1.11.3 v1.12.0 - Controller Manager v1.11.3 v1.12.0 - Scheduler v1.11.3 v1.12.0 - Kube Proxy v1.11.3 v1.12.0 - CoreDNS 1.1.3 1.2.2 - Etcd 3.2.18 3.2.24 - -## Stacked control plane nodes - -### Upgrade the first control plane node - -Modify `configmap/kubeadm-config` for this control plane node: - -```shell -kubectl get configmap -n kube-system kubeadm-config -o yaml > kubeadm-config-cm.yaml -``` - -Open the file in an editor and replace the following values: - -- `api.advertiseAddress` - - This should be set to the local node's IP address. - -- `etcd.local.extraArgs.advertise-client-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.initial-advertise-peer-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.listen-client-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.listen-peer-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.initial-cluster` - - This should be updated to include the hostname and IP address pairs for each control plane node in the cluster. For example: - - "ip-172-31-92-42=https://172.31.92.42:2380,ip-172-31-89-186=https://172.31.89.186:2380,ip-172-31-90-42=https://172.31.90.42:2380" - -You must also pass an additional argument (`initial-cluster-state: existing`) to etcd.local.extraArgs. - -```shell -kubectl apply -f kubeadm-config-cm.yaml --force -``` - -Start the upgrade: - -```shell -kubeadm upgrade apply v -``` - -You should see something like the following: - - [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0". Enjoy! - -The `kubeadm-config` ConfigMap is now updated from `v1alpha2` version to `v1alpha3`. - -### Upgrading additional control plane nodes - -Each additional control plane node requires modifications that are different from the first control plane node. Run: - -```shell -kubectl get configmap -n kube-system kubeadm-config -o yaml > kubeadm-config-cm.yaml -``` - -Open the file in an editor and replace the following values for `ClusterConfiguration`: - -- `etcd.local.extraArgs.advertise-client-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.initial-advertise-peer-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.listen-client-urls` - - This should be updated to the local node's IP address. - -- `etcd.local.extraArgs.listen-peer-urls` - - This should be updated to the local node's IP address. - -You must also modify the `ClusterStatus` to add a mapping for the current host under apiEndpoints. - -Add an annotation for the cri-socket to the current node, for example to use Docker: - -```shell -kubectl annotate node kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock -``` - -Apply the modified kubeadm-config on the node: - -```shell -kubectl apply -f kubeadm-config-cm.yaml --force -``` - -Start the upgrade: - -```shell -kubeadm upgrade apply v -``` - -You should see something like the following: - - [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.0". Enjoy! - -## External etcd - -### Upgrade each control plane - -Get a copy of the kubeadm config used to create this cluster. The config should be the same for every node. The config must exist on every control plane node before the upgrade begins. - -``` -# on each control plane node -kubectl get configmap -n kube-system kubeadm-config -o jsonpath={.data.MasterConfiguration} > kubeadm-config.yaml -``` - -Open the file in an editor and set `api.advertiseAddress` to the local node's IP address. - -Now run the upgrade on each control plane node one at a time. - -``` -kubeadm upgrade apply v1.12.0 --config kubeadm-config.yaml -``` - -### Upgrade etcd - -Kubernetes v1.11 to v1.12 only changed the patch version of etcd from v3.2.18 to v3.2.24. This is a rolling upgrade with no downtime, because you can run both versions in the same cluster. - -On the first host, modify the etcd manifest: - -```shell -sed -i 's/3.2.18/3.2.24/' /etc/kubernetes/manifests/etcd.yaml -``` - -Wait for the etcd process to reconnect. There will be error warnings in the other etcd node logs. This is expected. - -Repeat this step on the other etcd hosts. - -## Next steps - -### Manually upgrade your CNI provider - -Your Container Network Interface (CNI) provider might have its own upgrade instructions to follow. Check the [addons](/docs/concepts/cluster-administration/addons/) page to find your CNI provider and see whether you need to take additional upgrade steps. - -### Update kubelet and kubectl packages - -Upgrade the kubelet and kubectl by running the following on each node: - -```shell -# use your distro's package manager, e.g. 'apt-get' on Debian-based systems -# for the versions stick to kubeadm's output (see above) -apt-mark unhold kubelet kubectl && \ -apt-get update && \ -apt-get install kubelet= kubectl= && \ -apt-mark hold kubelet kubectl && \ -systemctl restart kubelet -``` - -In this example a _deb_-based system is assumed and `apt-get` is used for installing the upgraded software. On rpm-based systems the command is `yum install =` for all packages. - -Verify that the new version of the kubelet is running: - -```shell -systemctl status kubelet -``` - -Verify that the upgraded node is available again by running the following command from wherever you run `kubectl`: - -```shell -kubectl get nodes -``` - -If the `STATUS` column shows `Ready` for the upgraded host, you can continue. You might need to repeat the command until the node shows `Ready`. - -## If something goes wrong - -If the upgrade fails, see whether one of the following scenarios applies: - -- If `kubeadm upgrade apply` failed to upgrade the cluster, it will try to perform a rollback. If this is the case on the first master, the cluster is probably still intact. - - You can run `kubeadm upgrade apply` again, because it is idempotent and should eventually make sure the actual state is the desired state you are declaring. You can run `kubeadm upgrade apply` to change a running cluster with `x.x.x --> x.x.x` with `--force` to recover from a bad state. - -- If `kubeadm upgrade apply` on one of the secondary masters failed, the cluster is upgraded and working, but the secondary masters are in an undefined state. You need to investigate further and join the secondaries manually. - -{{% /capture %}} From 21d3206ffd5a7d0ffe7586513db712e5f2a346ab Mon Sep 17 00:00:00 2001 From: Joe Betz Date: Tue, 11 Jun 2019 05:18:24 -0700 Subject: [PATCH 30/31] Document webhook and kube-aggerator port configuration (#14674) --- .../configure-aggregation-layer.md | 49 +++++++++++ .../custom-resource-definition-versioning.md | 82 +++++++++++++++++-- .../tasks/debug-application-cluster/audit.md | 70 ++++++++++++++++ 3 files changed, 195 insertions(+), 6 deletions(-) diff --git a/content/en/docs/tasks/access-kubernetes-api/configure-aggregation-layer.md b/content/en/docs/tasks/access-kubernetes-api/configure-aggregation-layer.md index e5a8f1932b70e..1199bd9f6af0d 100644 --- a/content/en/docs/tasks/access-kubernetes-api/configure-aggregation-layer.md +++ b/content/en/docs/tasks/access-kubernetes-api/configure-aggregation-layer.md @@ -224,6 +224,55 @@ If you are not running kube-proxy on a host running the API server, then you mus {{% /capture %}} +### Register APIService objects + +You can dynamically configure what client requests are proxied to extension +apiserver. The following is an example registration: + +```yaml + +apiVersion: apiregistration.k8s.io/v1 +kind: APIService +metadata: + name: +spec: + group: + version: + groupPriorityMinimum: + versionPriority: + service: + namespace: + name: + caBundle: +``` + +#### Contacting the extension apiserver + +Once the Kubernetes apiserver has determined a request should be sent to a extension apiserver, +it needs to know how to contact it. + +The `service` stanza is a reference to the service for a extension apiserver. +The service namespace and name are required. The port is optional and defaults to 443. +The path is optional and defaults to "/". + +Here is an example of an extension apiserver that is configured to be called on port "1234" +at the subpath "/my-path", and to verify the TLS connection against the ServerName +`my-service-name.my-service-namespace.svc` using a custom CA bundle. + +```yaml +apiVersion: apiregistration.k8s.io/v1 +kind: APIService +... +spec: + ... + service: + namespace: my-service-namespace + name: my-service-name + port: 1234 + caBundle: "Ci0tLS0tQk......tLS0K" +... +``` + {{% capture whatsnext %}} * [Setup an extension api-server](/docs/tasks/access-kubernetes-api/setup-extension-api-server/) to work with the aggregation layer. diff --git a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md index 34b079a862aeb..dfaf9bd5a3d16 100644 --- a/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md +++ b/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning.md @@ -250,7 +250,6 @@ spec: service: namespace: default name: example-conversion-webhook-server - # path is the url the API server will call. It should match what the webhook is serving at. The default is '/'. path: /crdconvert caBundle: # either Namespaced or Cluster @@ -267,11 +266,6 @@ spec: - ct ``` -{{< note >}} -When using `clientConfig.service`, the server cert must be valid for -`..svc`. -{{< /note >}} - You can save the CustomResourceDefinition in a YAML file, then use `kubectl apply` to apply it. @@ -281,6 +275,82 @@ kubectl apply -f my-versioned-crontab-with-conversion.yaml Make sure the conversion service is up and running before applying new changes. +### Contacting the webhook + +Once the API server has determined a request should be sent to a conversion webhook, +it needs to know how to contact the webhook. This is specified in the `webhookClientConfig` +stanza of the webhook configuration. + +Conversion webhooks can either be called via a URL or a service reference, +and can optionally include a custom CA bundle to use to verify the TLS connection. + +### URL + +`url` gives the location of the webhook, in standard URL form +(`scheme://host:port/path`). + +The `host` should not refer to a service running in the cluster; use +a service reference by specifying the `service` field instead. +The host might be resolved via external DNS in some apiservers +(i.e., `kube-apiserver` cannot resolve in-cluster DNS as that would +be a layering violation). `host` may also be an IP address. + +Please note that using `localhost` or `127.0.0.1` as a `host` is +risky unless you take great care to run this webhook on all hosts +which run an apiserver which might need to make calls to this +webhook. Such installs are likely to be non-portable, i.e., not easy +to turn up in a new cluster. + +The scheme must be "https"; the URL must begin with "https://". + +Attempting to use a user or basic auth e.g. "user:password@" is not allowed. +Fragments ("#...") and query parameters ("?...") are also not allowed. + +Here is an example of a conversion webhook configured to call a URL +(and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle): + +```yaml +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +... +spec: + ... + conversion: + strategy: Webhook + webhookClientConfig: + url: "https://my-webhook.example.com:9443/my-webhook-path" +... +``` + +### Service Reference + +The `service` stanza inside `webhookClientConfig` is a reference to the service for a conversion webhook. +If the webhook is running within the cluster, then you should use `service` instead of `url`. +The service namespace and name are required. The port is optional and defaults to 443. +The path is optional and defaults to "/". + +Here is an example of a webhook that is configured to call a service on port "1234" +at the subpath "/my-path", and to verify the TLS connection against the ServerName +`my-service-name.my-service-namespace.svc` using a custom CA bundle. + +```yaml +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +... +spec: + ... + conversion: + strategy: Webhook + webhookClientConfig: + service: + namespace: my-service-namespace + name: my-service-name + path: /my-path + port: 1234 + caBundle: "Ci0tLS0tQk......tLS0K" +... +``` + ## Writing, reading, and updating versioned CustomResourceDefinition objects When an object is written, it is persisted at the version designated as the diff --git a/content/en/docs/tasks/debug-application-cluster/audit.md b/content/en/docs/tasks/debug-application-cluster/audit.md index 87b7f5ef5edc0..64d55299e73ed 100644 --- a/content/en/docs/tasks/debug-application-cluster/audit.md +++ b/content/en/docs/tasks/debug-application-cluster/audit.md @@ -245,6 +245,76 @@ The AuditSink policy differs from the legacy audit runtime policy. This is becau The `level` field applies the given audit level to all requests. The `stages` field is now a whitelist of stages to record. +#### Contacting the webhook + +Once the API server has determined a request should be sent to a audit sink webhook, +it needs to know how to contact the webhook. This is specified in the `clientConfig` +stanza of the webhook configuration. + +Audit sink webhooks can either be called via a URL or a service reference, +and can optionally include a custom CA bundle to use to verify the TLS connection. + +##### URL + +`url` gives the location of the webhook, in standard URL form +(`scheme://host:port/path`). + +The `host` should not refer to a service running in the cluster; use +a service reference by specifying the `service` field instead. +The host might be resolved via external DNS in some apiservers +(i.e., `kube-apiserver` cannot resolve in-cluster DNS as that would +be a layering violation). `host` may also be an IP address. + +Please note that using `localhost` or `127.0.0.1` as a `host` is +risky unless you take great care to run this webhook on all hosts +which run an apiserver which might need to make calls to this +webhook. Such installs are likely to be non-portable, i.e., not easy +to turn up in a new cluster. + +The scheme must be "https"; the URL must begin with "https://". + +Attempting to use a user or basic auth e.g. "user:password@" is not allowed. +Fragments ("#...") and query parameters ("?...") are also not allowed. + +Here is an example of a webhook configured to call a URL +(and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle): + +```yaml +apiVersion: auditregistration.k8s.io/v1alpha1 +kind: AuditSink +... +spec: + webhook: + clientConfig: + url: "https://my-webhook.example.com:9443/my-webhook-path" +``` + +##### Service Reference + +The `service` stanza inside `clientConfig` is a reference to the service for a audit sink webhook. +If the webhook is running within the cluster, then you should use `service` instead of `url`. +The service namespace and name are required. The port is optional and defaults to 443. +The path is optional and defaults to "/". + +Here is an example of a webhook that is configured to call a service on port "1234" +at the subpath "/my-path", and to verify the TLS connection against the ServerName +`my-service-name.my-service-namespace.svc` using a custom CA bundle. + +```yaml +apiVersion: auditregistration.k8s.io/v1alpha1 +kind: AuditSink +... +spec: + webhook: + clientConfig: + service: + namespace: my-service-namespace + name: my-service-name + path: /my-path + port: 1234 + caBundle: "Ci0tLS0tQk......tLS0K" +``` + #### Security Administrators should be aware that allowing write access to this feature grants read access to all cluster data. Access should be treated as a `cluster-admin` level privilege. From b42f019aae053c8bc54e2bfb7f1dd948c5b3532b Mon Sep 17 00:00:00 2001 From: prameshj Date: Tue, 11 Jun 2019 07:24:33 -0700 Subject: [PATCH 31/31] Create nodelocaldns.md to describe NodeLocal DNSCache feature. (#14625) * Create nodelocaldns.md This contains info about NodeLocal DnsCache feature. * Update nodelocaldns.md * Address review comments --- .../tasks/administer-cluster/nodelocaldns.jpg | Bin 0 -> 37691 bytes .../tasks/administer-cluster/nodelocaldns.md | 60 ++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 content/en/docs/tasks/administer-cluster/nodelocaldns.jpg create mode 100644 content/en/docs/tasks/administer-cluster/nodelocaldns.md diff --git a/content/en/docs/tasks/administer-cluster/nodelocaldns.jpg b/content/en/docs/tasks/administer-cluster/nodelocaldns.jpg new file mode 100644 index 0000000000000000000000000000000000000000..e42b86ea0247a45bc322d698978265926e12a59e GIT binary patch literal 37691 zcmeFZ2Urx{mNwdC$%>#P0m(svXNpQ}W-Xz03a(f6yO6H?CpoV=-W1 zFoG~|U|`+AKz{+z1O3Fo_~Qfm*AE6J7B&tp-ZgvzLf{7Ybr2>778WKp77h+JHgIq0jL*Kt^sus>LUUxFS-`?Q7@y$QZAvO?7FITP4o*QK zVG&U=aoI<5@(PMd%39hwx_bHshL%>=&unb%>^(faynTHA{6j*+!XsY2h9)GwNlH$6 zo0^uBo0nfuSX5k6Rb2zGt*dWn{Pg)tM`u@ePw(j1_{8MY^jE~<((=mc+WN-k*1?a% zqvMm)vvcInabbY4{$*Ie|NkoKD)9VpGiP-pTfd@gy0sl;3CVx&!PP>vOhPl;Q!Xh{%v6YJ}v}^2nz#P zJggfaFz6J?p7Vm>e~-UE2mRyCO4+t!Wz+@mLjC^j7R$y)#kb{dag;Iqq|y_G#<*}a zh=l|VLbdBckiuIHmACz5j1E~Vjprv>`#v(GDm}!iqN3Yy>>L{ zNN59cc0U3Qf|e0*eL;it!JG5TXb@-z8GG7Bdd7+dU9ts{qCwdevS?6y6!=nyem~<3 zj0$f7F5>c`K?gc7<>vo1eJjT?_+I&=Ptnu?|{+#I%V3B-yW@t zLk=mUKiWG$gI>^UpiURjAnd)nXyFmF$9YTON?!2?Xz7b4iQ79 zIj7JdIc`siqlip$4LQ(1^Fi^^!R*JAJv3;C!Uhe>buMiy$1+5N5+WdI(9=@z6^)mv zvkPh+oU#@Z#e1N4Jj@w(QkS=xP^O2>Ps)pGN@A;5?772_AtfplRQbJa#eE!Tirx-r zipOEuoP2j){ZQ)J`~yI!t3N}I`0UKiXz0&IfCaS)2L$TrXxqLp_y7jwn9H&`1*Cu? z|6A98buW91^bpYD7q?P8=uu=t#K5Ar=R+18Anwmy~I&0Z;jxNA$ zD3GB^KjchB64(H;xJhWx%StpTntngo*98?$%!LMFS)xJI(>al2J19VCG3QaS6={fhVy@{yIp6zotYHyeclMzuB1w? ztfZlJl@e{r0T5>5UMz5_d$bB2&&Pbkvg6EDGV5- zM;3sm4?0<>=*9{WV0}HZ`VJN3Z=aPSHz;DA&>*JdLO_x))UT*KjDEjhc1L|NrEn`& zXEz#}9SGYLw<@JT6r1_Gsr+A%fmvmM1F9v%0SEM8QAM1BDHM zmfvE<1rC5-8~NqIt#OB+pzQB4{~3>ZVh5D+9W>}FxC~JAP7^-IzoYQL02=gd{m13V zNEnI})rey`1LMc?}ytIIcV#r8OVth zV9R{}U6E`^aO=}=i+7{QOw9Y%uiQaW;lvat5Qd%j5vdbMW_=L* ziihV|+gJV_<2xr-YlLNY?*IeX@0FytS%px`QXuuRVTfAH?2?vKk*p`moQXX&0T*K} zd&-lh;n;!OMInLGOTuHdo0Ua#3DZxW;bzwpxAyX0ou|d@rlV7)^ZOTODEv3MJkOA} z`eX277iXdFIcAMQW6CE{m!Be?>7fvNkhl5$6Kyaws5{e)NB^n{ zYWJ3GD3cG@p{qx_*V;RqrR^fP5uu5EAoY^O`j5EHDI5e9pCS?d`O8ibA`e8u@~7L4 z69v`6rGnSz?YYPveXt&#wUCIjdfCL?v!Fdm%~OudNUzhSX3fH>c1|i=CX@PX`9g$Y zT3|@icU$hmWrf4>Me@AF_j9~G-YY;|F-d%;qZ#q?KEOtXr#N_soJ=76666UjA&*f) z@g?D7%|gquq+QIh)`hR=#wBju&wDTYQT?e**JL9XF;_>LAR^w8tKsx5K}|I#akSUnmNE-r0;FVw!A%uI?OVI6OHZPOgB#qU$S_Zp{(O+uR!j zvAliRi3cLs;z}VOOQ>$m*gHE$#per-0@h1I^iRYGY{FmB!x*snz$=-)4uBulhU9Q~ zUokQSV$Xs8QyyF|CfioB=K-|m90JgORu5vlR=scr4>X5AnmDGxN6d16GdyQ8(a1<) zRGd%mp1p7Uc8lzb`KmHalgm~rjH*ag>8;mq#*Yd3k^yk7WOV^#eQpRjfUdVykwh;$ zDSz9#?K4xeEpo^cY94OYKuKrF-bJ;UsV;P=Q`IemQOK{R6u^rA8{{JfsCPkrGafG=x*t3F}%=yrlbaQLxNrx~#Vp z?;rtwi+6Mtb6XnQ;3=vNjrzMQhg8g=xZTc8`m~l)`3^(^>00V~fl~Nh_l_l?TthD5 zg}J^DNy(#~kxR##T(1}7+kbswfjKN^8wEmoUn2iZo*`EfJ_% zQKC6ar77S_yz@{$O0rni$LUH-C|3x-Sb*erI|Z=QLxGLEVLt`li&RJ5H$FX)8EJMZ zHp+Ood*Pk&3}rnW#n+$0gw-cUxhy5nv({=~t7k+k@5MR+8@-x}-;ljYkfyaip> zsBf+vcPS&D#@v#yc~brK>CHIhTG8Hx$z$?Bi&CHZ0MQe%{C$(8)~rIK={Zw`Y9XFc zz}v4XVIIl4INl#N$Gs`?b2s8PmR7$Uew#H>WUjAI5B_l-bO&G)`9KtfavXZa%;Q7 z+ZD0l`qcqIOmuRs>&m_IFIL!ttoZcZzE(%Yn#Qi7X>z7Z$W=j{ zT@}a>sY^e!4h^~n&%g^xW)}2 zMT;QHb7f|D<@2~3pHb;c;Ij=_i8cyn!!F9_%Q$2XlDcF@etfPT1v7`gDIT3LQ2fqu zxB49WvUJLm+mPGiQO$VvM}uV|Z3v;K`57&A`O;)@_HMJv`))@@-hk(yR^lN3!s}_@ zJ-_q!DZwwe!y1j+>ubMkCQdeNe6lZ_@uzFGe)U-$Bcf1^TZyea0|9|Ha*xPSOg=yR z%l`YH_ET5y^>jrjJo%1X)H)h;7j8cZj#Q!}nS6x?Wl=?-MD1<%tDkeW`0@&!M=r_{ zE?kk5o*#q)(jN@p=KcJDH_X#joRA%$C0lXqHAus7S>~GO-(3!>bhQ28JJUk7&7eUK zgeIq0mgdhXpL3u=X5Z1E_te|=cMtlI8!oTWAfN0FV8a&x)ULg2*uQ$Qzhm?N$@A-G zrZbzhw?(GhufbqBb_H?4-Z{bg3mzYF8|KgSgXmBv#Q>M?FoY7)eGd2_%!gmvP6!%- z{ikZ&0s`#7PoMWsPvPp9H%%c204;BR7I1+mMB!I}&nBJ%{y0EgeYljm`mP1=-BVeY z_OZXVsZag{z`M(PU{XkcHV}Rg0v3Tf7j=t1qOGL_vXFKN(5nD-jsYMMA?fG&xe4&T zsviKo04Et1vADNqpLNFC46FtaqWIs={Lh@3-UE>sPnCJXE$`azMdxEi;;dJkTS7gQ zt7&|oQ)YrvgVwufP<+q@sy+>xg>@wEs(h9PSu#reCaaQgox^!XDEZv*Z)^4%43def*a=fwJd**rT*2q zuPLcCe0WM=*@3>sw$DA6*ZCZX(q)n!U6#_yCgii-9P3$D*56VtpE84}cXnKn8rF<2 z>T@@L8t!cBF~*pIIZV=r3C{0a-4gctGKM<1dbt7SU2|~HxXq?2=4_oRALXVMO&TWM z7Q1BO0Qo^a0@+>x+(n@}{j~YZFhMiqIT%UoG(s8!Fsy(<+%$XyL23b%5M1=(Ut4eg z&EH9KKW6<ur zuOT}WUO+r9t-=3KlT)j0i>%{o$&hM0-Do0@K7yK*{-~KdaS7N8RGa~6K9|hmQ&$)} z;C%}~XFsd1**o@u90Mi{8kA3_(|c%dQ6lfe1oc|W8)X4pfd~Lo-%CZHhycGJ>Sv^u z1Tfnl1dv`*0x(EbR8)kVeHc{9Z+52WRe{8shAeadA=3f+{kUzjBRl}E+arP8gES~q zy6%bee|cNa1e|p&=yNR}8f4cL^KE6>*6|RjgD>P{>huiNrCtS(=O!}Gw1Xx#$a;b% z%H*C^WGlnvJ==bJ8hG*UWZMR*Ign;>8!#(`2937Gqeq%+RIXW$e>V_$ba<17$(@pq zZuY(o44!q}#MA`QRjBWmTvp2JOhaWJADc3oRR>!EJ3QI`ttlY#D+_}R(*b!1xV(S` zW1rM6BGHbw0pgty~d#Y>1t=akYvXZ!0`8$^|CN@D(`fjm)T7QyhE+8+z z^^F!W2XiC*%oADI>`YLVA$LxHx>j{;Vj;neyQ~GjZ(OF*;N7azfK^mlciVds3iEIy zx*LNL|7ra&??A{==v7IYEwCW?BsJhQQoz^zm+9yIb&X4v+=!m|{ZOTri#m_`$Rk}N z<3`;7P@gqMe@H$$1nUe~9BniGA|;;65_M*Eo`^z$M>BC5awx5BR6+jlGBB>KiJX;b z){#$evG_Bqo>lfZ1x0qp0rld9{_H*kC7|$Y=W=plVdJD_k0F=Lh;;t;lMK!g^90}Q zXEEm)DFG#z7_V$_BHT<=UH!j&nj6GVH3;>oE!V+S`f`wtv2B9sTc>~qC6B2(23!>& z6h8vIF2xLT>KR>ix(i$Jn0#)+X>>0qq0x-dM)QoyTW2QJB|jeh+v4rK>HjX>yY)8? z?B7Nk|9*nde~>&Sh6eqH#U=_dXb?b8r=Z&P%#a+VDSsz!8NRza*Ey;3t~i0xlFG*8 z1=qVQv+;@^YEMENXMbZ8Rx9(Z%7ePCF7VE{THezN*2e`o<%10ayd4n09K%Utg{V?Z(kv2agewXHOFd#4_RcK zB%K}E>AINO0l$$?ynq-1pz0^E_^r?QHGxRhm0^P2zibKO`!)14P z^~bom*MrwJYQlAz8fIw|OFr}u%F9THmjix|Ob)=@m;&1uQ|r=!s#r|KQjB$FHkz7C z0!2oA8x;)5C|MSeIl@fMdxZmFBn!B+Fu~;YYT$o*ON^2?iQaVXQtWvMWLlH-Dy`_V zO`zA(!1J$Lu~roCzuh7aoNuI`qt*S=I25t|b=}_3D$^)J^{~_X!-l*%-dyP9b1-d{ za$54+DwWv#lfLcGiHOn z=ipV}^4&R#BTb?y+TNC1aAPq>7rGX(#a)xE_y<&Iqs|W9QxQe zLk@ltKGIo9&pln{@#g|U?`slHl>(mfG4gk62jaS^*cG2Sf1>9cMT1m=CD$E2I^SoR z5_2fkOuT*PIb>M2pMw|9@mcTf8La!yUGXm)^->g)D>!WeC_~bQZ2)%+b4vg`FH28 zal>hp@*SF-D!at1X&;9!U~O>uho?d7g{+G;op=WAo-bdE2^xF!*q!u@YykzgAU4t* zm9|?6lSS*3e!Iez2Cg1@CtgQV6*X`!L3KYzBS#`GL~vZiUcU`n&*osSG{Z`YBK ze|_vu6V~lFle#;pY!33*R@PhO)1=bD2U`h-bo16pi&x|?O(JRNiZ=C=Chp8qbal9u zeNCU)dbo;rS=AgRva%g4UFc=MiDHH>Epqd|ci^g9q8L3UvPjR^9q|9 z;_(q*dfqJG`8qygY}8d`4C$Jz_CH5aDu2q|{D6C<`Vb+Np0%B9hX!r8d`W<8Ja))f z=72?+vfd#~Yl5G?Hl=C^?i94ml_H8xFh(wb@CpaE)k<`cs!18$nUgoLC=g3LSg|J& z?h@2Z#zG|lN{6&?wjnNnm@>Hjm${`#D&G>=tTYhBz)$1Y7>j0bL7(apu;E_g0Chr-4?jS&RyYIc>4w+Vg0K2n~=fbCbwg@0xq>Ft;%M2 zz*`W&*G4IKQPvqcgeyc3k7+x>7NiiR|f$^vMH zuG_~3u)rrmHDr942@)QBi7SkXn=lkx7#A{OR)7UIDy|$_syKMk8GdKjW4O%4qeQwX zy>~$9z7>JCM`FZ%!x?hJRPOy&l@gJLy}uhFBT*IYLMh1^ z!$hByTZN>Qhu;Ue04JK)*)9w5GNMQyq@Gj}*Q~86+r(vE49y7jr#s5Yk3i1t8zib` z2SPOn&zsw~Yz~+z3CR&WNq10v->uXImHO=j~Dx&7rqI;FqJxFL`$E`yp zb0;rsRWAM6E>BCDWG_v3yjJ|n5{azac5Rlkh5dCY32wwSeslc+H^xbg$2+8Ttli{n z%OdF!Yf{gC&;(Z^j1VmC&uGh4Y``yIRJ=|(igZdkx<1;kvX-8$5Ud`vuVp=%%KdRD zv9Tr01{N9mcwDp#kEGd#khD9vRx8%I3Y2)PzQ}t%7r~ZYxU=T~F(ge`mlbZyH<}fy zsjIwK(rzU7DeK`Z1~M(o>x}eumI90HJ(-v5EInY}1zZUA+Fp5f6?NfPA<7hiCY8MT zPq4KHF$q3*S??_|7*!$Dmuh*%+J>f4ou~4mS8Hagc;i|pN(?LVy$(q2l;8=M9D&=@ zc%s}_!?Ob4(>#%DsSdvP)|#Vv2urRFdekq|_Xl}(JlgMDj0_YvF$sOE7#*rxyk7Pq z2ETKA>Irqn-1AE*_{wGN@ zYdBO?+aa3vw-F~1wqAriWzH~5YqBod$FVhW{Do|p4s?q?XIfguNwZ}ided)QERHtH zEJn&Fi!?O0HR(@_cX_xEzKf8JIaW`QI0cAyz%hXXG41=W{yJcwE#kdDYuH2l5m%Tt zeVK0IgsD70#||D~yJ~A;;B2E6LVL{!Z#b*5?KOqr(wOE1_EPPUkbIE_FLtZ$0`X<3~R_y$O2CPpEAhBWB-dRk-uYx|Ho;CB#__K z#+1f#8xZJc`5EY+LP70UH~yL-z=|1HlQeQWsW=X&xM1DQG?fl)y=c5}Iw^L@9v7TZ z)tv8c4pn@ytCc`STJm94y}mqX?@{ZRxgYFUk)0Dt_$xW^-x9X0*n4_DZaKU=AZpAp+uTWHz6ExvQ!P?Eulg*}a=TVyB3dOVdx^`5ec+yMpYxQ! zwe;-7HG-q;3#xND zKaFWXD6K4nOxA8Om+i2Yb?&HMoRsgrs~nKP66<1xa$Hl*rTo~z{fA}KlO3~k3u>ho z9ZschMGXq1cy|?v{T%VlWQK0ect;!Z(TEGF5p6gYH$2}dabs2G{Av`kmsZMFeJ2GF zpJJP$cHaT&y%^7neP!XsD}}=P?$c5)ZlU_L5B~xbdJV0$)tuGJSZ(;!!Gr4~+U^jH zhpHPdxpuf$u75M=qv*OpAra$N*Xl7H({NHAD^s7j{Tb=iG5P%5Dc295vX25gtsOq% zB9yEImUB*+kGcQ6uUl9OIev zy@C?YwT<XmH7*M%?Jn%8dqZM{p}OCl z7r|0%TpQLw4?b~OW!g(A6P7yJ&*;Ky1CsO&=X`S+q(^0OjpInJZDt;!Uo}#YI$%?b$6@6Z#iBom@l1 z*tp%{M)H#FQk0j?Q@rg56mW_A9xiL0k9?kB?yJtKXCG?~$1J^bC=r^?6Hq9$S6yRl z>$71MqgqG;#rjv`M9;Ndz@bJ?LiNdhtmZWW-Ji{AeuG0Sv*;&b;GCDQWt0M zxz~>G3V4?9$F|4zLNM)D+`IVkUQdi4S&LCj#$98G3K^&F%ld4{{W&F&|Erah)oaN* zfV65aH+Y)z^%*HqqW~kcdG2Nv)^!v6>!c^nsX&&3)CARjO&_YlxD^emjsG;?gY#@( zYh#3zm?x@KP9LC<>@R9-=A*0rsELZ2JDr@To9xj#7SDW$tm!?{$zC)dMGp8p>B?c| zLW6303M)d(Vv*Y|@-DC$Hq^`7;-`fZ*=2%FsdfC#!h4JgYgJL9-)@6nFO}-spNPjX zBE0x{=U>-*A!FKNl{YN+3qq)HYPwmOvg>gz&s*&W1|xu!;j)`{Cu@SM!tXUBCKTJX zb~z^rRSh7zV;&^XO5U)-&~^2R0)Xhs)U#VJLefTDPEROQPQg&9EEi93i<4V0WPt}n ze>zfT9adC3-mA?TF`{F*>#N2i2vcShU;3JVW%-Shb+YCN4Kfl7MXJ&OByu)}exsIA zt_VI)7?MlU0U_Zz;RxBk3+_z!%HWV!D}096yp0l)tlHtxn{JV_$p}y!#4UW*HS$ao z!eG}|YS@FyGaf>NjJd560lr5|RqNeX%+Rcd#}pX11^tI5MhLGg$1-B zi=vV=^t*oexU)0w(B)xVSS0Gq7!y1d*hyld}U`Y!7Dz{3B`muZesJ-X1>Wu z1yb(t!o>mUYb&<|K9(_(t~ERM2gGT|__1Xf)(&qgoz@u8qZ*7KL5^4{w7ODec40f( z2h``W$C9MOLGbYnk$D6)OWDfmi4){B0`J`BAfHk*!|ve>4(#Y-d z6^xLVWK&lFe+f?NDy}X;WkoM7X1bNvLk1it zS%xlSZqXlqXFDt_*=f;R^La|K4&LuWgOr}ll4}^Ivywv1Uu_ayP?8!%JCS9)f1F%! za8K3s9h9TnRB`)ETb#8uEi>)Z$5f}5C%^8=sA~u1%c%w|(Xv>~t=e0egz&n?=DNDf zZb`lSeGFPRxoB7Dk_M9CiS7RP5HASLy_QHt&qF|8hFA%0D(SOfYFJ}VygPd%nS;B8 zNm}8yRwNilqD7q*1*v(V_QK)%-6WaryP9esCQ0`PaKZLfdSV@MZfZI&5VShq)OK2(dG?-EE>W`*>J_XqW0jiK zccZ3+vKC_|YF(5m*57WTW}^9mYMdKq(rqO*B`8pMU8sLkSC|)=8uIyXh%Uilb$od% zqMldUmkk>pn;TrQlgxOF%ce7i8z;J&@hpugcI98|)+Gj|gYu3)4uJdzCbO+Ao|w!T z&7_iCSK%yJ{+>9XG2+!RUh`UrF#b-Jf?#DO?~fNTddyk*7DjUBA~BK=g+l9muD74K zxnt1De{OQ#ovC})ppQvUd=|v@tPa)ZWV-Nd<0k%jR_}b2mCaNuJJy7gixY3oq2sQ3 zeXQs_(aX-Doa*D)>t8qObgvsX+fOC>`T1dnQZiQ#Ip04PipVi)mKI5ePo`DV++Cm1 zf3#J&Mq?7oRqrn48psKuWT`kQ3}0!~_YQ1x5PWY*XSA>MJq@%`?j=FUp~+RcZOaU) zPv|W%+Q46f>D|bc%Ny`o@}8U=*_AuWvGj<7wc@2r<$OoMOBdu*n<)xXqe6F_!qr+$ zPrd^5tJ8ZD=sy^i|7Q-z|GhvX7uh;Wfshn=l6XEj>v1Z=M}KR&K2s`Z?tzi-$F-`G zBh%7Di63WmVqXJOA)EI$*z>UYu&>Lk+)y!0$7O?pS8m_FdfovjF5OuR-&~Z88)T~A z3`ybe?a|nb3R&-db+!hx@3&B3{iC40;4dkC#}cKGhSB2`6hfnM9~B+wb@Hx-wb$>W zhnb*1C@;qTj*lE_DpvLNV-*V6jt@yp0}+&xqHEcBaqL)TS#Si%Dwud4K8Dy1vY4$` zyiM;x*?v@dnrN@(pxa7WHC_;zEZ>lY+ppm_q={5=%RoxC`?EN%qD>!3& zPjWcO!8gfuda*K!OlmvT41Y_{<3Y-@g5s#{YJDTTj!SApo(RhobLE8^{s~NiTfkrx z{tDYL9VnFI{E*>d@HnnkT2~ijMHfn!6q{pcZ@3CSUYTPahc!2SUDA&1n=UDTu$6ao z`krJ|!uD>mfkn7rf(5mn4yXYOlsF$FR-&>vd&4SCZuA~j|Fxvn!BkiOkS;Nx=n{7; z#lIlY-GxmhrM5Zwt8@Ir$S_U5Gg^<#xulUNI9vlvs^iZqYBx{nr1g+(;uL{dk7GDx z3-#ZL)CdD5B+uMcKCkk0GOr}mxA&}-jXa=m^z0+%5svxPJ2yeOnoRi#Gw@(CBfuPR zj91JI+dW8p9Mkx=vy7=LrGMg&XG6C4JF2bc6aZ>+x5FFL}8Hvi1|$9uqG`wwwW zn83FAXWR;$VVwI5s_eGENxk5iPF)^ec2?aF}3#e7L9%MlccYEg+%4*vm zk8mxY5{@yx5YKCrBLCT#sOXU}IXd~Eo%FRI9L-*+D!6*W^xw$g?uj_u|>8}l;KFQ`t6MHYX>l6-_D zQ{i1=$7(d%mao3Bn0?^ns-DhF!$(3oFa}ewEVXM)H8(FZoOQgId(?Y|T$sYjx6|`4 z{Op8{kf9iXH#$UAr`G0wdd+;UwE2zBcd5q7xu*D1seopd`c!6u>6t!haw$cpb_m^Z zxA{D+$LI?QgY|I4?KD44_BiWId4dwDo=t7b1Kdc#NLs_r%-t>w-Bo;WxC6K=>j&dD z{qZj3ZMl*5cugOSLoj<``)vhp&JU-|Ed=G0ZoQB;AH4d#2XrZNZq(r|Zi`1FmtfXH zYX`3rmVkHL{Z%jC^Y-$K+!x#O$I@tDE(TW%F)Q7x*n>R_zs6d}EGzbA5ub|r={l2C z{q_SOSl7pmYAf_jR2r*Tt7@!mJ@io_C#GRyyfqWt6}6lN^AZGVZ1vijBh|*rD+-7# zYG1FphA@`|t4W=`CnVS71p=HkkR$8~uqgQKJN^DUz>yWp3i>n`ghpKInNBG(`rF^3L9azt0!)0kc;BX+oNKE^zboxv0Vx=V&10i3po7r2L zRvLnqR|+2OfQ^niT{k|9!m%&rQOMmHGGiMwf03c&!RycMGF5oI<_*mz=Zs>f1bLIg z0b}Nq6a5v2koq7gNC>)D=kyf~l2V{Zd{_?Q4QeBk7sX*e47RvfG!N|csrn(GK zkY(xRRM62-zTRASKUMTx1Ir`*hdSKq%%i^|0n}<_?$2 zFRg-qdEnMWTl`hO6db%}<~K59*lz8ZeDa}-=tdN?Tak{&4vva(+b>gC1NrOOohXGc zUAzsjhxE4|7UeWR&Qt+6<~?iiIk0A3z_k60Ev zlwQTV<|-nc|j)!I>KAy^fa`AuqpJE=A3&@T! zwd|wb`MCL&NFE|On4_->UHSW8B)&MK&76Q-5KN(hywoqLfzqIZ-u=QaqWV1@orZqA zhf<2GLV1a*LEsh7H+mvhVGd{zSiBHL%mX=FrJvSqJ3It3>qpmtZKjO|mHC>jrhIGb zhkT{DsOY+kNr0{aQ3e=feI58?dp_Wn(&qm9DDb2t?~lG+vo-qvfnO<7eLDt+QO?Yo z#S4vfF%AR@mh`PCPjl`l=xB|>og#+#j<-7BSaP+g9}1Kv>*nK$30ZHW6m*Ph8hKxF zKbI-7r`Fj`k2e!^)Lgs$Ofx=oi_%|D%eXUlSzoXzk+?Gw_H^jPcg4uON{(R>d}%GE%Ffm;TWLKIUWuo|vnX^5;W@g!~CDL3BwSm1pwylE~&28zMcKUeY9(VrJ*Ir)E zwRo{8^{efF2jqlR`AZzf2_eGB~-U0=IuEhR$?{m{tMP+~USS79<8 znAvOYMk}l#ddLh4yVtJ<>RHgqQ0Z{BLp0e%7HsLI5R&zO8U-zCC*TH|Z|Vxh-uSwG z`=()@o^D+nyB&A=(*17oBS#;)r5DTxw~~=l5qqA~EBbRwQZJ{i2vp5vLkxz^R2oEy zUfheE6ruNus23s%Vyj9(&J8O_Z$}oxECO@wY{T`BH{0no^n|1g=anz$5w~Pdn)2YW zPw_~;cKU60fGm_2ugv<#;LI&Gfu@LWGiuh+QF_Z1>)xgHzkDvi#GKt?fs!w zB}>}LH(ltM;8qL;!+w%c>BmF%3EQ35WlCl@_cz7iE45LjT!u3dPYh&9eEg{!%$H2S z8XuY)BL;P=vA-GhQ7Kt4*x;ytScZc<4hw2pMfS(+r*1idM|H?1D2dLEEARPdsm&eF zOtEH))g({wzLTAtoU(b#qxzP!NMWwq!VS9l-8s9Z2D*iiVZO=WJL}IMA7_hc*nOaP z`5HNY>0M>-m{A)^=u%W zD8&%}Yh{Uh0F9uom0n}02YK@c5G6t`z=xkGc2NuqZ)~!(wlctprPS0wmRyAYp1JPx z?yyE@O04Y=rODt4)MNYf%+6Vg_}m1SV+axnyi?&;$=tM&y)=)$^U@)c5plc9jiJ=2 zgX6rdR_)Q*X>y94ufrqyn6rW=-kn_XUhrm;SbX>1vfR7j{{A||Bx_E>?uTLyG&8B! zfp;M#Yep~x5#gI{IOQ$T`={wLG4~?OogC0rh#uAMwI_MUH<1@u=Rf{Z3s_bG3%T{5Q01a zETjh%r3R){KwXR$U^P`xf{!o^;pfqirM?N24MOvbX(JfQUZ(ofsQ=}t-fLUA{~j_0 zlvQmC9%mbz&i|ks25+AO+-{ox5r4&LZ%`qt#MN9-gsgE}blZe2_=KhgFoUAKDAYAZ zzr*EfZ{K}YTd(r=q*mHfjOwIfgfxLq(&gpVi$h3P&em}1;V|htJL#E9r0#CDbcR!& zv1;k7=wO~_WCh1eOeG8J>26wP`iL!E&98G0tBrb_Oocb(EaVnoLe`<*jJHJMKDOu7 zHa6NZJ`}%GST1$=fa*5cR_+qZjO_=5UbR$`MbGb)ge{d(SbCaMPsVtTc`3H54E^?} zDJ_G?+~lH_f>fR+vmqMEN5AJzvP_VRAgsg8Et{fClxTxkGm19%w~|%uT+Mo(={_x_ zsr}$cZy6IYbst_m@bUy-MBj+;Nv2Ty#fv z;7z<$W{*ty+DDS0RiYcc3JlO}^F;|7?uQba&Mp<@=GYHU>$rK7GMi);=>0*(i}U9+ zmj-~Hcnf$79kZ@?ZumjPE8_a5Z>FjDhnq{c`}ScIo`^syeLnrpUuDD_m%2GsjdIqfqdiMHk30t5zF&`$-3JK_Po6pvVnBsk`a&27# zer|%neopClbC=2yQDm zmBaCwu^bQ2Ux&=t(vOzR&1~Iw=!4ie(QY_BPoOh-D^b1SU=JM`Rg)|}k1N>opqm>+ z?r8&WpL?R8eQWCzrKS5cce{nFgqw;@p7V&!hn6S02F#Wr>6l-FrI6H9ISIKGrBf)< zOTJlok7RM_V%(Gf!r9Q~fLLL%m!gh1Sl&`@Lz7tnBo(Cy8p24z08e+3q4m9wcw#!8 z9>?Xu(5%v~lq3Q) zX?p?42=x}!^{xX7I;}Qi-ERi0Fmi0@Bgcg`9bWss4y34w6JP1U?@A85#+LP|nSD4h zqNY*hu+DPKi!6nsko-U`Y%z)XiK{XnlXSXv(w9xd8-!U{e*KHPklLj4PI(fp?B?-; zx~ZcJ@@p3LB*yXGAG6{k)%uj<8FHpwTh z{KQC&NC(r_b{3YHD0|Z%Rq@_V|%iv;)ppA9t_| zYFmYLOINF#ZFg_li`s-QihSl&Z0KBhDi0xWdS8F5GP0VgzH88uo+q%{I6SsQIBC&4 zLG(*vpSR^}(uZG2;N%|$b5eLV^H1k5p&2^374THVi1DR=W}#(?A;P}YdDADJHqOe) zMK-IbC}+Y8rf6(pS84dp*ZO{$yUp{roqmIOh7hgHQ{BBl3B0 zOzrt?2Xr}Lxz!kb)$oEjS=V-C*SjH+S>*HqUKY&4-mt8(EyOy_T87d1RnOza%8>Xh zm65?g zkq|cW(I3f}<}B*5a0TTVSo(&RY>mX%i;7Vm515~2j;<}MWlU~#IFR2PAd9gwA7dM> z_EIvDBrNPK(^g%T|Kk5NYtuygLrI>+INW(I{ASq_qPBP4fcVR80z>@VH(0?MA(lT1 zt2sHIJuSgiVgu<35I2jPMKjb@%a98oE;m~E@z;;&VJb`Otb>Cy%nDMyOw090Xpr=q zKm+m0P}2lGRO{l!vuqBb5|Qjon$UToV$OA%_z=?3^xgPuA8Q@jWJ{o2p)yXpnV1GSeU!; z&6c*$Ju6Xh^ZYB`eU`{kmNIdt`S3!x3~geOfg(yWY!D(~La!OU(opNfo6uTdrthsb zdlMpJIrS$BMVgn3{vJL2A6@^m@oKRwBH(ok#eg?H6-AE(6&>I4RtbD$m*y2YKkpah z@IFpHp6!7_j9cbMOndcTZY`U=KRVUbcq9;)FsI-m)Vguq|B!UuEWJo>qLhQ{ z$LvgNaZueky;=V_Ln3^q4RavtRenHI@=f&7l@MQ#Zs@Y#a1wmSZPbYX z-XED%>l<+exeBOmAf80w^ zaO@c?k?QQHnXWq8``9ZNPpMntaFv#tN4dC+DoD<(d7j3&rhT5_O>2rxYSgG^tLtqJ zAo$*_jCrP_4+7yteFTD}#sfA+gp|sB>beL>U}T}8re>m{0r=C}+jy#8Jk15GB1tH< z{z#I^ghBliHGHHVo^)2B9g1#Pi1x0y^D zCe5J%p)F}_evR$aSHpQ%BX#Le7kr%C(l*uwh`AHQ1>`fnUgGmwz2mVT5^SE)DiWMU zTlxz+gm<(}w;GdPMY>F<3C%Z2oW6I?Xk4mqAr+q;ihvzgp1cBA=ch5JTR)3as=0-? zcY}^px&c-}fIxgb3;B|KqJN!vKTJoA;CZWd09Jg=tb~sCC;2ze^7fcp&B%;^GKtGD zf*(aoqg5{J@WJvbxOb{cjc%QsU&pqit|<0ZrsK+$fFV395J)SH#zM6L#X(Ai`AMbA z9LR8qD@0IOMUBJ!;UeGnck*NGoDGU`{ zdVv=U$OeX9`W5rDb4RmqDUQ9n|9ji0tphoe_1KfAI|H@4v?O0<{k##_)X_TOcj7St z5ay1<%X>U*$Y{ZqP+(mS>U?jQ#SfQqqO=}02ya=`8U=^|-(VN?BPq9Qdp4hb#z3Z* zxeV)xLTg9z1MTjYs~8?&YboSEBa3>|AH%BEysiBCa>^I_35l~8GF!K{%%R#cooZ%X zk)*|#HNhV2c=B07K59P4 zv8PHQg^yKd{Xwvl&%=!3}W=Bl|2ntN4e2#UKEtmgGgNO=sVlRI@3c0 zB-iB_X1*u(Ap;IjAhtUkU2ts1ED_(uNoy=Yxs4APY-tykzw<7$H$Ha-o0|GW=;$^2 zJ1LWWwj_xC;biee7BJWCodSnw8Fm4Tc1Lcoo|fQIb=jhAi7{lzXbL6!)RkW)S?0E) zV%PG++?!+K=Lfs~#3?#n=S`IKqooQJ#g@J%T4Lx9-W{c;>Q2qa&!}!iO}ZOYM)ZaZ z=Tr%Li>1r_wpJPepave>wF5VV{H z?rYOk70N!^rIKwi(A61#_T+p9A**D-!D_2CcT&0VGD{_;FN*N1d`Mn|$}r!^#=(_| zTGLs|T%L!dzRLN6L9g&yq~tnwCZv;Y zIkf6ZD=*EIasQ3#N9eZ1kA6gwi2HtgsEG}sVrL$es)+83m0O?>cI!coUWjgg7rI4> zXwDmbmmh5^lEj@pE7Qx(S`ADy#uKbyj9dEJ657UMOiWo=BCJDx9;HhN_A|+OCDq*w z#}%w5aC?GmbHSjX#i+)YpcjAYpCzjXcY|+v@#6wQzT{lKFr-B6473q>LiGa^UK>d_ zH;mS+)wbnnQ>wJC){;6$&o78K$R&U6^3{}0lc4-pzU!6YtG5w`J%#%< zjv((J>6TQhpBb3huxl@;v6JnaU*3apPfT{sm1fS5ik90LIDRBZ`_`*KoL|jx>Bvb( zf2;S)&v$sX_EJrN@LNvEy+#5z5xlr|hr!SKad7^mdbZ|afD53VQc3HCRpephJSu>0 zKm`q}nu@QNG2WE?7OOx}rtEe{tT?eL2CQA>I0ayo3_ zC|ZfTxyiB~`S5T_N|-OYSC@J2S)jrLHScUQb4Ycs+e_GBD;yD#6NJPRqH1!%YIdH7y*A0; zli0>jZfX?EXXWLrY09J3wKRLJB|E8KA>1Xp9$xqH-A9BGhf-cCSdEK0(eq~`+HnzB zY-xzHFMeK39ueekjaS+&2?>XR5nGNe>rD-0UqaGUwSULm|GfmC$=JcV)-?#Uz^oWa zQTvel8Ti(kvL$0(w~9=K%CO#ep)&Uu8%k;FB;y!>Wc7#mW`C4t_B)4>d~WtCw%tA{ zILG~+)Wf!T`F1ycs$VQPr15!AaV`b=lWFSs;mh!Nv80u7dS;>I+==c>@hQ)`girgk z6Hep03%Sr6NbGoK^&?07zGpG#tR6AUI5B&$2Ld~! zqzr(S^mloBHecFh=W=!ONV^>&JNKwWx_R*7@@ky!(4(T^gpQkdi%h!mdwGK%4Uv+D z>NYp9p`zs9XGM5p<*uonuVS0xrtKL<>Ccq<9eG7IWr?1>i8~%Sy{%{Gu>}n0_Bj@D zD=L-k&g490$o(G{;FU~02m>!EmY8ni9*UE4Bx2^8{b}S-UBj1fqxXcL7hQ<+X626- zl7_Pi=lSdQ_GC`Obb_J<^S z-m;bKjlF9y8nOk`L~yledSO3c7J8*;qK8a6jy~YBdd44N?jOzkG~WSGe&|MJ7s2pu zm^X!HpW0I^dlcPKPt$#&=cp(>i>VS9fcZIbYH$w7J)8_(wB*82`{t?Um7yL6vi?b8 z`re25cp&C86zBnSSvB9gU|3cKzGL0l{1apJf0ScKvkkg;)nzN&{Vn(3%KJa|et=?M z0@$d3ZH41+<#Ak7WI%J`G4rJ-Hx61(w*k9Lxk}@&8EuXej4&m#pyCd?Nf#bNO}Yn* zcwE8Z$4#Ig?jwl-$O$8}iSg*^Ne%u^ZLLX4znXkZ6J%7Pf%y(*Tw6&#^YRNa!HJ|Z ze)K0xpENqLl;U1*Yy?>AY9&3+)$_JMSqKGDRI!$2zoKVVc&G0sG&jW1L}l<|zR$G= zvXwF-vu<0voJghl#}4 z>?76@@ z>+QQLRDNhWlcFljV3_Wc%Lm*EdZ{J(A)T!y%5_&jCvQkM?Cepx(Yah>JhA8ju%Q{m zxtsdYW8JRlZd}y<1nwbZGhG7{9`=>pvn}8UAs)KI|D`HaV^qwZN1@KpHjFV1*?Nu= z7fspR*q_(|!@!X$5+g8uu?=xzpS?DLtC#vnF7pVm1c{RK6o~U3TvVqz+t$U|uM|ny z$-nj%e>f7~s8#rwe4LeAmcO>OJ$M(3@?)cI! zjB-iStd(c;+T7nDyH!HKzK8~|#fdNK*Hmjb?A@`C{>&;N#kr8#B_0tZO9tqmv@(Uw zTnsLVBZ{bRmSwBFW7sp(RH(nch|en5vqgKaa-@!DzjGdGz>{|T>X-YypKjDB{jA|Y z`7d6&ce6J>3^R;ATa$1zo+Ntl${G98hx0drCyBJ&{KCqNbbF)vExMzlOSXv!RDp@C z&Z@V5cwwz*^dOp zebTG+I$5!uyG5JaoPUFEG_wF~VqQ2LILs#xel+{=wbf6~^t}$Vx@!LxdI(gkqC?>D zn=1M7?N&|2eOCA5(Y|RWOkSEAd7_TW4DTH-^5x<1q@za_%}JM2)L6bsWQJAmtnQZ_ zMdtQzu4aRwp6ioKhQ1S~UhB-Oe!in*^BEeC54 zT(XkYkSJxqDBI-0XLf|d^8FQ>&(_^j&@XBm>V}1un(A#T>b#<;xv;KrZ*gDeD{!0Q zpa{BQOX!0wqIZbIcf;zYa*7E+6H=1ExNZ18McaKqv;k$n~gW3&`>0G7jMCudyMgqre*o_gF|F*6DfLgWpXp^@@J@b(8J`=CaFp}xylYqAMwVT8>!}l z0~GQ}$FrY7j*cF?fK3KExHDAfYqV+IUcNHTS5Y3m>5JQpiRikw$*(9I-Fq?$&6+F? z5h_9QdBCwJx!=kNKUr~}0((rb$00LYTB?i|Mh_-ku72s7pwIo*yE}!#79(Z$Fbn@-m zRU2HvnGJ8#h;@9u_d#>+nRHrz7vhdDnUZ5osIG7IgjfVbX4C0)pyhyTSFW{~x3-U~ z=UkCVa$_iG_bzL-1g-?$_OKkY82g+?9aPfko9sHVx?i+DXDVn*Rj$hj{YK0svb+jb zqI)8&KaCxCfUksUCmRc0n5EgiLb?|E#Ez2I)z*YK(#1ZWcWaFGImGpA+azgR0awD; zyw&fZ;Oe`I-&^Insyw^2YNV6a7+vJ7cHf{_+|%Nz4h7TW{n^JRd?s>F>eF4GJnPhr zjIXYlPHTBBP2+une`Lm5Lw`zF!Uxw96^}cLOs&)Yy3almPpRvOZO*HhG5HEF{n^1P zT*DS$%!(pPsA#HtZdy2RIroF*ZJ z!mNanl3<^7rpPUdT)!V7DMSoEkA#1qi-les)~wM&B-QG?RIiYD-}fWU_P?3w!&{x$ z{=kNl(qeCYY#R|fZR((752y0vz3fhwC>E1iyhz*$qZ+Nr8giQZQd_PJyEzvjDL9}> zxV)V=X|I1@LjMrW1Fa!=wD!aLlzGLcN7KL&Q5~U<2=q(jzoEXjH+CMLM!iE?m<^op zI^2##sHl_*zLo9;B&md58(PWsCLW~ni)d#yc{`VzS0z*t;OgqLf>9MaTX1uGNxRR( zky)}ZxAZ0t?8Bi^A2sq6o0d+J@N`DvVkpQUEuwoqC6L+2H^Ya?8B63ABi=?Ak&^nh z8g@fZi$&$0&c?_TyxREGoA%2AD|CI#Ds|Mw#nYU{3=#{BNde@-ZYFw>d?NArmQzf6 z(>bDKTYLcFI+ueVqha~HXg{h(F33%Jy*(~~4juH>+rQv*OLBt-v_ zxbq(rUt>juz!skac(wITstW>rq|J{NJrEY&MTZR^@5h*URbECUi8Kk z3?GDb!C%P%fNg+Ij+=mW(GxX6@LB-xJz3coaW0!e+5q;a!V?JoU4gV53jw5*=%I_` zQotPn)&Mp@#Af@qnMd5jf19wba%r>*D*zfZ&s09-t#tuZ34oJ#+>{z8*mD z|3+{L4cj?Y_#%>;FwKFj4cz)8)kkko+Swe19+q%4oJD-nQ~0IN`!L&eAYe_8fo+@K z0d^;eCu>?NqJlW?MZr9m4#Y@)`6fstQ|Ch_+c@8H@P2P&%;h&IpC|>*csJD3 z>dK1nnO-#sN*8M;@s}TqjNTJa#`+tGHVdU~O4Oq(ubf@Tsc}G59_ zPq|UAin^^VljEUaT@82lfn@+1m>dm5PB51U0L8n`a>BRIsP1xd1dxwt|GOQf{57xB1Z?wwVT5C7^~Wz=1GNKk zb%;+!mtv^?`hfac;er`j8_zhEjj*dA>jGm$cF{cYOr5a?b`3+dTv(<}mCVR}nDdYzH8oW&mBmk(l> z)J<9RW7qId__c!HKLCjBodE{Mbz$5BYi|wJtPC2$y?8@^nebZrG|J?3xSQ7X+n202 z%mvS~GuTAH9RIvBb&)o~7A{#nySgy8Vrt`v=V2s-UMD`^yK_2mkte6#RZX>EH*yt(^IijeuZJdywV;D)+m;J^;>fG{2zm`~9Qj&;3R~ zHEDn3opAqtnMu<(RYvNsFDS0pZQ@e46yn{jTq_Yeiy=!toWxd9Nh$}l+GbKK|Ek0$ zo9eGOm6X7KCp`IgN<@FJO7xHaCsX*H2E}^8?}aD#|8I9H5rzS1lJC08z}o;elZ?@+ zUu8Fqlok>W1~=P(k=lGZb6rWiE2PR0+Ip|sPE>`V@WngIKyk8nn}YeNSjF#ydxri<&qMxOxxoG=dIkfd2=nWtoT>1UsT zfzEp;+>>uLDzcjb?a?3E{r0vDyIp-VwdQ>`iaS$(LL6Vh%O-ul-S546aW|iJ){kOK z_?1lVj4>Fj2L0LP;O=bU6_&ZcaYu)IKDM{bPcz+Bu%jvOCS3TYfiF`ZBISG{Y%%~P zxsXd>g|%mvfQv;w1a(78tUb&X1_WG~hR;i5mF!BZB|F?kPx&&A?L3iGk`8yP+P`S> zSy+PZ%?e8re@Hh;+7ftmCwq0w)nsBq{@8u7IIbZ_YQ4duaEGJMP` z2A=rXXH+-f%L|ZKM9-(z1BDu@)1TlmL8GIj;Zr5&pLAy7uVBg^8eu1fw zTqC~R4_7EJ$I@Ew`o(JTo+S}1at&^XmvS6`a-bF)>Y~fzo^DmUJ^7J}*DCsH5(;CN z3Iz7aJNJ6#JZ0Xy4XE92ws@xXihyj+^qkoi@@fEMiR{OxkOtIRhdh7YqAH_!;> zIh+t}NweLDv@SYr%gHEfrZwEB!aeWp;Sj(&#oZB58`TbWUdO_F>kt$N}4M+ z_Q6-KK3qE;rH}H@u6KIeVbqYCGkat0lKuZ2;wI5iTa67Me?HlsiHVCL%z;9#=M;#i zy-Wprbt95F!Las)6zl1zIPkzfJOJN@Q*M@NsPtQG>0zRi}UE z>IrLfPrV<)l04;BR}?Nz-eI# z+&7X>{!e5x{(jvc?0{fRYa%=aUmO`LxnD)@!@Wj}mwO|vz6w^d(%+Izf&*rmJnQyh z&7sj!f|gGkc-*240IFTrza=T_ip`^E9t%AHpHwDGdS1D*8ri0ZInhzRL+nZtM3G`! zQH8v3UcWuv=70zKwUz z_!^wvwIkqtmc)_rVr2CqnND_z^jn?#Vd9RNzTUa^J5!Lu*v3bdKW$Pn+k72p-@Z+; z-V5(2-|ojuu6XOGaeTqNsH#=7v-u>8j?A&+^k5pe+>LV5Ppt}lU#admC9GaqopdKJ zRvkDr2!CXa+sh;=)6sgm9QA3mQ};O0?8=k@{szZnvobNfEhn1ULw@Q*SsvWq-LPFi8{WLBEExx*0WmK7} zGEMGc^-ePqdOw-9Y?&lTk@M~A2W%ZOren??O-fJ8D!wpyGvK&0AYPaF8_G4Q7zYAV z*V02hAcCGoTgi_w9-p5+yggU*9Adc4mpDv7MqpuE>fi_M7z^`|SqVC!w{^dBi^L_` z{_9phfsOJ!)X%qy<#r`l*FW#0vswf9%}d@wZ8f|xX&Hp;{c7}eb_EWB9L;A}5`~KG z(xhbDKEEb6@`_@STE}e}6jven77CC&NI+b!DMyxM7(F)>Dd2BG4hdvoP@V~KQ2 zu^V@x6Q)79ubYx$gc6bSk?IEY39hTv<&YyG+GPJ0QOB2lObqtKlf*F^?4V)iQsI!~ zya|&wtCxdng!L0{XPQ^|p`CJ~z8$tyR~@_!4sXau+T^cY*NioBfL8?56evhgEArN4 z`(X}H*Bp#<5@OV|47EHqXRJr8K0n+pJj;E}5@zo7Hk@FK@H-ia-6%Y(I&!Wm{0^^Z zJ*WQ1gBN7Cb_(tP(E{+#KKu7w9Qw`G6LkDl!WX{Zm4beC|K3GsenxYKe_NA&)H8bj zlMM(oa~`cxL}#y~Q;;>E28Z>{JnF9p-S}Ikp^Gp8-SJ=o)eYH0oxpg65thFZET5j` z;#H>T!Y+b=_|gNwQ0{{TbNn!X?X$L=cO7e^6n>t+UHbt$Id5Kt#ndRjfLWRX7B#>h z+T)m0JtP6nkSxP3TY$rrLTdTEC7kNdmG1tv(l-hhqzA%h#|qz~H!cHb+xvyEnN~cl zJJ3EYhd~<*1)P3g_nt=r{lrtBP?@?sM^%1>JyDorkHfz;F1`4T0635Txf4oj$1W#8 zI<~?&958a;`DvYAb!a*IQwF5!IZw{PXw|DTlZDfNnAshlE!~B`1UvYX!TH~b<@8-V z*T&tJy))oia9iLd02?hfy$P|_e^CFYc7fz1Y@c=nwhbJSk*Z<;Aw5ir}%N1v>W9@$36WH}bM)N;+yJb&erC94&#R9P1 zLUI9}(wotLpNMiu?>!Fv@(qR>1KtQ={Qo64^lWC0*N-I*ctZO0y(1VY8nErR9!_Iri zL^G_Laa4_Dj))Cn@(zO3CAxI(8p`|o>AIX-7OKt3_~Yc!Vk1{ifAl-v>LY0gFJGQq zAe%1ox!>tWnmRirjb3=h;SkQvOcI&8SG!HX<{(@)1zs@@7+Dk~!G?xeHr=glQ(@N* zmOm!m%eacu?XYRQ1ZW!KXkMrDBH~yW^Z@d#sIrXyh$qvzHv3yke`tPs#g&jWpT^EO z0mbJ!iVLhsOMzU5llE}q4~MRBjHm4flTK+RNl)$v8(%HX=4)Q5`DRrYJIU(hf2$@H zca#~YdoSSLq)t`q=e_h=Ld?iBr=h-l1bqk2iM+INr_7d2>XO^bw%sL!PqOwf(xuX$ zO~S___S8Ev2QGZ~HaR#?H6LknEH}@5urJLw(vJ#-U*gByljc2((wA_4JTX=(uJEB4 zP~U8!#7-i_L`}alg337#tPga^&#w|NB!#}0Rq)NLx;TiuK>oxUi>f+X#;EVxoJ6@! z>JMbj2XH*8ml#XPys0<6Qh+K80Ncx`j?wi@uN<0P-4TSBEhbjh87gkQ#DR>Y=1h&k zxU`hg(wd602Cym(rNC@?pZwT&6FWzT%XXDjRzVkkg?DLW=gF-QH#Cij^~(>iG$dDE zA5eUkA!PmiJC=d3c0c>+=>C*NPkjJ-U=X)QKten4@|r5y+b$;qs(fB)$~4Ad;j4Xq zCSGII0yQzgcV#~?slWM{@!<{bwJTVjuBl@9mFc`Af`*ywOo;c&8!qy%Uk=zWb`;qH z{vg2MUp;94HUGcQf%Cr%N>x`PlL>W~n_EK9Z%-H&wlYfcd<^IyP*e6j^KtmhUPvst&a#ZZ*WFH_l)r@oBsA~!MJ_e zBls*Bi_LED$sCyEbUiO1#=Iu5prio3##G5HYy%Vmkh?4%It=%-X>U*Di~Pasq_&t+ zwv4T+vl zH%?u~%Vm!lPnT-dFZ7qzTen*a&F;pcOibn1ep-w8zAp`G4-$#zxwVtOMq@w9VC?Q= zZVX@0@vp^fY|m8n^t?NtYqce?x+VMKwcO2rahd&vp!5gI{yQ$qtx)NW;|p;G&;=y~ z>TEr!Fv_G?_F*{#I8W?XRr(UaE6G<7Kh%oZ;P-y+s9r{BEr@t;J|NCD9g?TNyiDTP!4fO)1jK-Ll zrK|WH%2(}HF(@&@L2VBs^ z%!vTt26OI*yV6-v#a#sY0U;$d*yciuFSs+}N6oeyC&`2Iu2CW>-WgZ}$>9na#?J%6=z!qxS7E% zcyUFBf+7X175+`qGXAnNV0n#j{E{Lt%L439Nc1VGWBM^)1K^l`4*MlV=og-Sk*Bc2 zR05lYZvnz@+P|cpU0UOh{i2$_Yes=(iTmw6I$bkDpHj$e)LsfZcGv1$;l^c4T86_`fToqaQHWHXM z^D~akcG4a|5v}4MaHvF=P!=`(E)BzK_!wF&6=V|EZXo@$@jk;Hn3w=&*CIG@Rwrs1MdsBH9wgdS!$cXNG+QDy1s7AgEf_Hk&5bED9~XW23j#Z7!P#jt#UIg2 z=kAKE)j^sK?sLgdpMbpKRZK61iCe|@a;VPNi+P^bBDvyJyJ$V(aXW z-1-!(MDi<4JW+2~O*_t|o%0>ps+Qgqt5ZH=96O`gaO%>?QH?p)b;5Zd$ZrLWwbd6t z8zAa=tJKi;TIBm~-TZ5*-TlidR$oN< zB;Bzx;;{@dU?N)srDH0m7@pLfDJO)A`{YejDjW*V@yjuYY|8~Z{BD6IjG3Dy3zyX) zBgWkY{0{LY@bb(w?DYJ+j|UXOS8IX(GUU9wNJ#|qUJ=p#ec6fM3lO($+cb`*UE02= x>eHg!UacPLW)0VidG~~zbD#4N+4C=np8g|Z*?&F@|Ia%FRBiu%83(_O{~vs)xuyUB literal 0 HcmV?d00001 diff --git a/content/en/docs/tasks/administer-cluster/nodelocaldns.md b/content/en/docs/tasks/administer-cluster/nodelocaldns.md new file mode 100644 index 0000000000000..318a8d522055f --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/nodelocaldns.md @@ -0,0 +1,60 @@ +--- +reviewers: +- bowei +- zihongz +title: Using NodeLocal DNSCache in Kubernetes clusters +content_template: templates/task +--- + +{{% capture overview %}} +This page provides an overview of NodeLocal DNSCache feature in Kubernetes. +{{% /capture %}} + +{{% capture body %}} + +## Introduction + +NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a Daemonset. In today's architecture, pods in ClusterFirst DNS mode reach out to a kube-dns serviceIP for DNS queries. This is translated to a kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy. With this new architecture, pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking. The local caching agent will query kube-dns service for cache misses of cluster hostnames(cluster.local suffix by default). + + +## Motivation + +* With the current DNS architecture, it is possible that pods with the highest DNS QPS have to reach out to a different node, if there is no local kube-dns/CoreDNS instance. +Having a local cache will help improve the latency in such scenarios. + +* Skipping iptables DNAT and connection tracking will help reduce [conntrack races](https://github.com/kubernetes/kubernetes/issues/56903) and avoid UDP DNS entries filling up conntrack table. + +* Connections from local caching agent to kube-dns servie can be upgraded to TCP. TCP conntrack entries will be removed on connection close in contrast with UDP entries that have to timeout ([default](https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt) `nf_conntrack_udp_timeout` is 30 seconds) + +* Upgrading DNS queries from UDP to TCP would reduce tail latency attributed to dropped UDP packets and DNS timeouts usually up to 30s (3 retries + 10s timeout). Since the nodelocal cache listens for UDP DNS queries, applications don't need to be changed. + +* Metrics & visibility into dns requests at a node level. + +* Negative caching can be re-enabled, thereby reducing number of queries to kube-dns service. + +## Architecture Diagram + +This is the path followed by DNS Queries after NodeLocal DNSCache is enabled: + +![ ](nodelocaldns.jpg "NodeLocal DNSCache") + +## Configuration + +This feature can be enabled using the command: + +`KUBE_ENABLE_NODELOCAL_DNS=true go run hack/e2e.go -v --up` + +This works for e2e clusters created on GCE. On all other environments, the following steps will setup NodeLocal DNSCache: +* A yaml similar to [this](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml) can be applied using `kubectl create -f` command. +* --cluster-dns flag to kubelet needs to be modified to use the LOCAL_DNS IP that NodeLocal DNSCache is listening on (169.254.20.10 by default) + +Once enabled, node-local-dns pods will run in the kube-system namespace on each of the cluster nodes. This pod runs [CoreDNS](https://github.com/coredns/coredns) in cache mode, so all CoreDNS metrics exposed by the different plugins will be available on a per-node basis. + +### Feature availability + +The addon can be applied using the yaml specified above in any k8s version. The feature support is as described: + +| k8s version | Feature support | +| :---------: |:-----------:| +| 1.15 | Beta(Not enabled by default) | +| 1.13 | Alpha(Not enabled by default) |