-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Advanced configurations with kubeadm (Kustomize) #1159
Advanced configurations with kubeadm (Kustomize) #1159
Conversation
oh hey this is definitely up my alley |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the writeup @fabriziopandini
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So I think we should limit the scope for the 1st batch to focus solely on control plane manifest customizations and follow the sematics of kubectl https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good, thank you @fabriziopandini, some comments on a first pass.
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
@neolit123 @ereslibre @timothysc @rosti Thanks for the highly valuable feedback! Most of the comments are already addressed; main changes (according to feedbacks):
There are still one/two points to be discussed in the kubeadm office hours. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @fabriziopandini !
Looks great and other than a few minor things, it's all good to go from my perspective.
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
keps/sig-cluster-lifecycle/kubeadm/20190722-Advanced-configurations-with-kubeadm-(Kustomize).md
Outdated
Show resolved
Hide resolved
@timothysc @neolit123 @rosti
|
name: kube-controller-manager | ||
namespace: kube-system | ||
{your kube-controller-manager patch here} | ||
EOF |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should there also be an example here for a net new static pod?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@detiber I'm not sure what do you mean with "net new static pod".
This feature targets only the 4 static pods manifests (3 in case of external etcd) created by kubeadm in the /etcd/kubernetes/manifests
folder
that Kustomize patches will remain stored in the custom location on the machine file system | ||
for the necessary time, and that this location will be passed to kubeadm upgrade with a CLI | ||
flag; this point could be reconsidered in the future, by e.g. defining a method for | ||
allowing higher-level tools/users to define Kustomize patches using a new CRD. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How will the kustomize patches be applied?
Generally kustomize operates on a single yaml file applying all defined patches. Would this proposal mean that the currently generated static manifests would be merged into a single file prior to applying the kustomize patches first?
Would they still be broken up before writing them to disk? If so, what are the plans for doing that splitting and for naming the resulting static manifest files? I'm assuming the existing static manifest file names would stay the same, but how would additional passed static pod files be named?
Are there plans to vendor kustomize to do the patching operations or to shell out to kubectl
? If the later, doesn't this introduce a new dependency on the kubectl
binary that didn't exist previously?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How will the kustomize patches be applied? ...
In the POC I'm exploring a different approach:
- read all the patches
- group patches by the target object (without using any naming conventions, but relying on TypeMeta/ObjetMeta embedded in patches)
- when rendering a static pod manifest, apply only relevant patches
how would additional passed static pod files be named?
ATM the KEP does not consider additional passed static pod, because of they are out of the kubeadm responsibility. But if you can provide more details about your use case I'm more than open to reconsidering this (TBD if in this or in future iterations of this KEP)
Are there plans to vendor kustomize
In the POC I'm exploring vendoring, because it allows me to do the kustomize process in memory and have full control on it; I also agree with you that the resulting solution is self-contained, but there are different opinion on this, see e.g. #1159 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @fabriziopandini !
I am in favor of merging this now. Looks like a good plan to start with and gather some user feedback from an implementation.
/lgtm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/approve
If we need to refine we can just update.
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: fabriziopandini, rosti, timothysc The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This PR adds a KEP for defining a new kubeadm feature that will allow users to bootstrap
a Kubernetes cluster with configuration options - control-plane or kubelet settings -
not supported by the Kubeadm config API.
@kubernetes/sig-cluster-lifecycle-pr-reviews
/cc @neolit123
/cc @rosti
/cc @ereslibre
/cc @detiber
/cc @vincepri
/assign @timothysc
/assign @luxas