Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

keps/sig-cluster-lifecycle/kubeadm: mark the kustomization KEP as replaced #1991

Merged
merged 1 commit into from
Sep 24, 2020
Merged

keps/sig-cluster-lifecycle/kubeadm: mark the kustomization KEP as replaced #1991

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 34 additions & 31 deletions ...-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ approvers:
- "@luxas"
editor: "@fabriziopandini"
creation-date: 2019-07-22
last-updated: 2019-09-02
status: implementable
last-updated: 2020-09-18
status: replaced
Copy link
Member Author

@neolit123 neolit123 Sep 17, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a relevant change, the rest is whitespace cleanup.
"replaced" is an official KEP status

[1]

---

# Advanced configurations with kubeadm (Kustomize)
Expand Down Expand Up @@ -64,7 +64,10 @@ status: implementable

## Summary

This KEP is aimed at defining a new kubeadm feature that will allow users to bootstrap
**Note:** this KEP was replaced by the KEP tracked in [#1739](https://github.com/kubernetes/enhancements/issues/1739)
which uses raw patches for customization.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[1]


This KEP is aimed at defining a new kubeadm feature that will allow users to bootstrap
a Kubernetes cluster with static pods customizations not supported by the Kubeadm component configuration.

## Motivation
Expand All @@ -75,7 +78,7 @@ Kubernetes cluster via the Kubeadm component configuration or the corresponding
More specifically the kubeadm component configuration provides an abstraction that allows to:

1. To define configurations settings at cluster level using the `ClusterConfiguration`
config object
config object
2. To define a limited set of configurations at the node level using the
`NodeRegistrationOptions` object or the `localAPIEndpoint` object

Expand All @@ -86,7 +89,7 @@ as of today. Some examples:
- It is not possible to add sidecars e.g. for authorization web-hooks serving components.
- It is not possible to set/alter timeouts for liveness probes in control plane components.

This KEP aims to introduce a new feature that will eneable users full control of static
This KEP aims to introduce a new feature that will eneable users full control of static
pod manifest generated by Kubeadm at node level - vs the kubeadm component configuration
that allows mostly cluster-wide configurations on control-plane/etcd args only -.

Expand All @@ -106,29 +109,29 @@ to some iterations. The goal of the current iteration is to:
during cluster lifecycle and more specifically for supporting the kubeadm upgrade workflow.
- To ensure the proper functioning of “advanced configurations” with kubeadm phases.
- To clarify what is in the scope of kubeadm and what instead should be the responsibility
of the users/of higher-level tools in the stack like e.g. cluster API
of the users/of higher-level tools in the stack like e.g. cluster API

### Non-Goals

- To provide any validation or guarantee about the security, conformance,
consistency, of “advanced configurations” for control-plane/etcd settings.
- To provide any validation or guarantee about the security, conformance,
consistency, of “advanced configurations” for control-plane/etcd settings.
As it is already for `extraArgs` fields in the kubeadm component configuration or in the
Kubelet/KubeProxy component config, the responsibility of proper usage of those
Kubelet/KubeProxy component config, the responsibility of proper usage of those
advanced configuration options belongs to higher-level tools/users.
- To deprecate the Kubeadm component configuration because:
- The component configuration provides an abstraction well suited for most common use
cases (that can be addressed with cluster-wide configurations on control-plane/etcd command line args only)
- The component configuration implicitly defines the main cluster variants the kubeadm team
- The component configuration implicitly defines the main cluster variants the kubeadm team
is committed to support and monitor in the Kubernetes test grid.
- To define how to manage “advanced configurations” when kubeadm and [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm)
project will integrate. This will be defined in following iterations of this KEP.
- To define how to manage “advanced configurations” for the addons
- To define how to manage “advanced configurations” for the addons
(this is postponed until kubeadm - [`addons`](https://github.com/kubernetes-sigs/cluster-addons)
project integration).

## Proposal

### User Stories
### User Stories

#### Story 1
As a cluster administrator, I want to add a sidecar to the kube-apiserver pod for running an
Expand All @@ -144,7 +147,7 @@ As a cluster administrator, I want to upgrade my cluster preserving all the

### Implementation Details

This proposal explores as a first option for implementing Kubeadm
This proposal explores as a first option for implementing Kubeadm
“advanced configurations“ the usage of Kustomize; please refer to
[Declarative application management in Kubernetes](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md)
and [Kustomize KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/0008-kustomize.md)
Expand All @@ -169,7 +172,7 @@ This has some implications:
3. Kubeadm is responsible for coordinating the execution of Kustomize within the
init/join/upgrade workflows

Additionally, in order to simplify the first implementation of this KEP, this
Additionally, in order to simplify the first implementation of this KEP, this
proposal is going to assume that Kustomize patches for kubeadm are always defined
specifically for the node where kubeadm is going to be executed.

Expand Down Expand Up @@ -203,20 +206,20 @@ EOF
```

In case higher-level tools/users are providing only strategic merge patches, like in the example above,
it is not requested to take care of defining a `kustomization.yaml` file. In case the `kustomization.yaml`
is missing kubeadm will create it on the fly using all the patches in the folder.
it is not requested to take care of defining a `kustomization.yaml` file. In case the `kustomization.yaml`
is missing kubeadm will create it on the fly using all the patches in the folder.

If instead higher-level tools/users are providing a `kustomization.yaml` file, kubeadm will use it.
This scenario allows e.g. usage of `patchesJson6902`/other kustomize features.

> In case a `kustomization.yaml` file exist, kubeadm ignores the `resource` value and replaces it with
> In case a `kustomization.yaml` file exist, kubeadm ignores the `resource` value and replaces it with
> the static pod file manifest that should be kustomized by each phase.

2 - Run kubeadm passing the folder where kustomization patches exist, e.g.

```
kubeadm init --experimental-kustomize kubeadm-patches/
or
or
kubeadm init phase control-plane ... --experimental-kustomize kubeadm-patches/
```
or
Expand All @@ -232,7 +235,7 @@ kubeadm upgrade apply --experimental-kustomize kubeadm-patches/
or
```
kubeadm upgrade node --experimental-kustomize kubeadm-patches/
or
or
kubeadm upgrade node phase control-plane --experimental-kustomize kubeadm-patches/
```

Expand All @@ -242,9 +245,9 @@ when beta is reached. `-k` abbreviation can be reserved or even fully connected.
#### Providing and storing Kustomize patches to kubeadm

Before kubeadm init, Kustomize patches should be eventually provided to kubeadm
by higher-level tools/users; patches should be defined in a custom location
by higher-level tools/users; patches should be defined in a custom location
on the machine file system and this location could be passed to
kubeadm init/join with a CLI flag.
kubeadm init/join with a CLI flag.

In order to simplify the first implementation of this KEP, this proposal is assuming
to use the same approach also for kubeadm join; this point could be reconsidered
Expand All @@ -258,7 +261,7 @@ for allowing kubeadm to preserve changes during the kubeadm upgrade workflow.

In order to simplify the first implementation of this KEP, this proposal is assuming
that Kustomize patches will remain stored in the custom location on the machine file system
for the necessary time, and that this location will be passed to kubeadm upgrade with a CLI
for the necessary time, and that this location will be passed to kubeadm upgrade with a CLI
flag; this point could be reconsidered in the future, by e.g. defining a method for
allowing higher-level tools/users to define Kustomize patches using a new CRD.

Expand All @@ -272,14 +275,14 @@ can create confusion in the users.
kubeadm maintainers should take care of making differences cristal clear in release notes
and feature announcement:

- The component configuration provides an abstraction well suited for most common use cases
- The component configuration provides an abstraction well suited for most common use cases
that can be addressed with cluster-wide configurations on control-plane/etcd command line args only,
while “advanced configurations”/kustomize allows users full control of static
while “advanced configurations”/kustomize allows users full control of static
pod manifest generated by Kubeadm at node level.
- The component configuration implicitly defines the main cluster variants the kubeadm team is
committed to support and monitor in the Kubernetes test grid, while instead higher-level
tools/user are responsible for the security, conformance, consistency, of
“advanced configurations” for control-plane/etcd static pod manifests.
tools/user are responsible for the security, conformance, consistency, of
“advanced configurations” for control-plane/etcd static pod manifests.

_Misleading expectations on the level of flexibility_
In order to provide guarantee about kubeadm respecting “advanced configurations” during
Expand All @@ -288,8 +291,8 @@ around _what_ can be customized and _how_.

Even if the proposed solution is based on the user feedback/issues, the kubeadm maintainers
want to be sure the implementation is providing the expected level of flexibility and, in
order to ensure that, we will wait for at least one K8s release cycle for the users to provide
feedback before moving forward in graduating the feature to beta.
order to ensure that, we will wait for at least one K8s release cycle for the users to provide
feedback before moving forward in graduating the feature to beta.

Similarly, the kubeadm maintainers should work with [`etcdadm`](https://github.com/kubernetes-sigs/etcdadm)
project and [`addons`](https://github.com/kubernetes-sigs/cluster-addons) project
Expand All @@ -300,7 +303,7 @@ Static pod manifests are managed by kubeadm, while “advanced configurations”
will be managed by users.

It might happen that a static pod manifest during upgrades is changed by kubeadm in a way that
some patches will not cleanly apply anymore.
some patches will not cleanly apply anymore.

The kubeadm maintainers will work on release notes to make potential breaking changes more visible;
additionally, upgrade instructions will be updated adding the recommendation to --dry-run and check
Expand All @@ -320,8 +323,8 @@ consistent state in case of errors.
Add at least one periodic E2E test job exercising “advanced configurations” during init,
join and upgrades.

Please note that, in accordance with the split of responsibilities defined in the previous paragraphs,
the new E2E test will focus _only_ on the mechanics of applying “advanced configurations”/kustomize
Please note that, in accordance with the split of responsibilities defined in the previous paragraphs,
the new E2E test will focus _only_ on the mechanics of applying “advanced configurations”/kustomize
patches, not on the possible combination of patches and nor on the security, conformance, consistency,
of the resulting Kubernetes cluster.

Expand Down