design-proposal: add self-hosted kubernetes proposal #206
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
philips
force-pushed
the
self-hosted-proposal
branch
2 times, most recently
from
96e59d0
to
d450fab
Compare
|
|
||
| - [Health check endpoints for components don't work correctly](https://github.com/kubernetes-incubator/bootkube/issues/64#issuecomment-228144345) | ||
| - [kubeadm doesn't do self-hosted yet](https://github.com/kubernetes/kubernetes/pull/38407) | ||
| - Pivoting to working cluster networking isn't currently in scope |
There was a problem hiding this comment.
This statement can be dropped (we can deploy CNI plugins in this model).
|
|
||
| [Checkpointing](https://github.com/kubernetes/kubernetes/issues/489) is also required to make for a reliable system that can survive a number of normal operations like full down scenarios of the control plane. Today, we can sufficiently do checkpointing external of the Kubelet process, but checkpointing inside of the Kubelet would be ideal. | ||
|
|
||
| A simple updater can take care of helping users update from v1.3.0 to v1.3.1, etc overtime. |
|
|
||
| ### Conclusions | ||
|
|
||
| Kubernetes self-hosted is working today. Bootkube is an implementation of the "temporary control plane" and this entire process will be used by CoreOS Kubernetes and Tectonic users after the Kubernetes v1.3 release is finalized. We are excited to give users a simpler installation flow and sustainable cluster lifecycle upgrade/management. |
There was a problem hiding this comment.
v1.3 has been finalized for a while now. Maybe instead point to plans to integrate self-hosting into kubeadm: kubernetes/kubeadm#63
This had been a design doc on Google Docs but it is constantly referenced so move it to the Kubernetes repo. Old doc: https://docs.google.com/document/d/1VNp4CMjPPHevh2_JQGMl-hpz9JSLq3s7HlI87CTjl-8/edit#
philips
force-pushed
the
self-hosted-proposal
branch
from
d450fab
to
d5a8837
Compare
|
@aaronlevy addressed all feedback. |
|
lgtm |
|
cc @ethernetdan |
|
|
||
| 1) The self-hosted Kubelet is in a precarious position as there is no one around to restart the process if it crashes. The high level is that the system init system will watch for the Kubelet POSIX lock and start the system Kubelet if the lock is missing. Once the system Kubelet starts it will launch the self-hosted Kubelet. | ||
|
|
||
| 2) Recovering from reboots of single-master installations is a challenge as the Kubelet won't have an API server to talk to to restart the self-hosted components. We are solving this today with "[user space checkpointing](https://github.com/kubernetes-incubator/bootkube/tree/master/cmd/checkpoint#checkpoint)" container in the Kubelet pod that will periodically check the pod manifests and persist them to the static pod manifest directory. Longer term we would like for the kubelet to be able to checkpoint itself without external code. |
There was a problem hiding this comment.
This is a challenge even for HA systems if there is ever a "cool" boot event. Fun analog: in the public power utility world there is an idea of a "Black Start" (https://en.wikipedia.org/wiki/Black_start).
There was a problem hiding this comment.
Kubelet checkpointing of API resources is definitely needed for availability reasons, as well as for reliable self-hosting. kubernetes/kubernetes#489
|
|
||
| ## Long Term Goals | ||
|
|
||
| Ideally bootkube disappears over time and is replaced by a [Kubelet pod API](https://github.com/kubernetes/kubernetes/issues/28138). The write API would enable an external installation program to setup the control plane of a self-hosted Kubernetes cluster without requiring an existing API server. |
There was a problem hiding this comment.
I was pointed at this doc: https://docs.google.com/document/d/1tFTq37rSRNSacZeXVTIb4KGLGL8-tMw30fQ2Kkdm7JQ/edit#heading=h.fi1ijw76bqwg but it doesn't appear to be linked from that issue. I assume this will all end up in this repo eventually, yes?
There was a problem hiding this comment.
yes, the plan is to get into this repo. cc @ethernetdan
There was a problem hiding this comment.
The document is still under debate but should be in as a PR shortly.
|
|
||
| The kubelet could be upgraded in a very similar process to that outlined in the self-hosted proposal. | ||
|
|
||
| However, because of the challenges around the self-hosted Kubelet (see above) Tectonic has implemented an alternative scheme that side-steps the self-hosted Kubelet challenges. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html) then when an update is required a daemonset updates the kubelet-wrapper configuration based on version annotations and kills the kubelet PID triggering a restart. |
There was a problem hiding this comment.
This mechanism isn't really self hosting the kubelet then? This is a 1-4 system. The use of DaemonSet of one-off admin actions smells a bit bad to me, to be honest. Alternative options there would be to have the kubelet rewrite it's own config and kill itself. This could be based off of a ConfigMap. (Or this could be done by a KubeletConfigurer that is a true DaemonSet that watches and manages the kubelet).
The system as described here has the disadvantage that it just isn't declarative -- if a node happens to be down when an upgrade is run it'll miss out and will never be upgraded.
There was a problem hiding this comment.
Yes, this is a misunderstanding and I will improve the language. The daemonset is long running and not-edge triggered. It reconciles version information annotated on the node to the kubelet container that is running.
Like the userspace checkpointer we would like to get rid of this as well but we would need chroot kubelet proposal to be accepted and the ability to have a chroot supported by the CRI. Right now that seems unlikely.
| The advantages of a self-hosted Kubernetes cluster are: | ||
|
|
||
| 1. **Small Dependencies:** self-hosted should reduce the number of components required, on host, for a Kubernetes cluster to be deployed to a Kubelet (ideally running in a container). This should greatly simplify the perceived complexity of Kubernetes installation. | ||
| 2. **Deployment consistency:** self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments. |
There was a problem hiding this comment.
Another aspect of deployment consistency is to help reduce the amount of "distribution" drift. If we have a single packaged thing that can be used everywhere we reduce the need/opportunity for distributions to carry patches.
|  | ||
|
|
||
|
|
||
| ### Bootkube Challenges |
There was a problem hiding this comment.
One thing not mentioned here is the challenge of discovering the API server (and etcd). The way it is written now it is assumed that these components float across the cluster. However, most production components will want to pin them to nodes (or dedicated nodes) so that they are easily discovered. We can't use k8s services to find the api server as the only way to resolve them is via the api server!
There was a problem hiding this comment.
This is not the assumption. We do pin the components to a set of control nodes. I will make that explicit.
Address questions around the kubelet node agent and load balancing and control plane nodes.
|
Great stuff. Would also be great to add your thoughts around running HA master through self hosting. |
|
|
||
| #### API Server, Scheduler, and Controller Manager | ||
|
|
||
| Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors. |
There was a problem hiding this comment.
What about upgrades in multi-master environment? Are we going to upgrade each master node separately or just upgrade all nodes at the same time? If we're going with the second option - how can we solve a situation when upgrading API/schedules/controller leads to failure of the whole cluster and no API service is available - will the kubelet Pod API be used to rollback the upgrade and recover the old services?
There was a problem hiding this comment.
Upgrade each node separately. If you have multi-master you still have a load balancing tier or something in front, presumably.
There was a problem hiding this comment.
With deployments, one can enforce that only one (or a percentage of total) pod is to be updated at a time.
My concern here is with flags passed to each component and how that may affect the cluster, e.g. API server adds flag that requires kubelet to also specify a pairing flag or run a newly needed feature. But this could be resolved with a simple updater, as mentioned in Long Term Goals section.
|
|
||
| ## Motivations | ||
|
|
||
| > Running in our components in pods would solve many problems, which we'll otherwise need to implement other, less portable, more brittle solutions to, and doesn't require much that we don't need to do for other reasons. Full self-hosting is the eventual goal. |
There was a problem hiding this comment.
"Running in our components"?
|
|
||
| > Running in our components in pods would solve many problems, which we'll otherwise need to implement other, less portable, more brittle solutions to, and doesn't require much that we don't need to do for other reasons. Full self-hosting is the eventual goal. | ||
|
|
||
| - Brian Grant ([ref](https://github.com/kubernetes/kubernetes/issues/4090#issuecomment-74890508)) |
There was a problem hiding this comment.
add ">" to this and above line. Otherwise this won't show as quote format.
address some feedback requesting HA notes
|
Please take a look everyone. I would like to get this merged this week. |
|
@philips What's the goal with this PR? Is it to upstream bootkube? |
| # Proposal: Self-hosted Control Plane | ||
|
|
||
| Author: Brandon Philips <[email protected]> | ||
| Last Updated: 2016-12-20 |
There was a problem hiding this comment.
I don't think date is needed as we have VCS in place, but in case you want to keep it, please update it before merge.
| 2. **Deployment consistency:** self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments. | ||
| 3. **Introspection:** internal components can be debugged and inspected by users using existing Kubernetes APIs like `kubectl logs` | ||
| 4. **Cluster Upgrades:** Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet's are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc. (An example script is in progress.) | ||
|
|
There was a problem hiding this comment.
Please, point to the script or remove the reference?
|  | ||
|
|
||
| For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. | ||
|
|
There was a problem hiding this comment.
Another advantage is how easier it becomes to do master components HA.
There was a problem hiding this comment.
I just realized you added a section for HA below but adding small reference here still makes sense to me.
| ### Bootkube | ||
|
|
||
| Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). A kubelet connects to the temporary Kubernetes API server provided by bootkube and is told to deploy the required Kubernetes components, as pods. This diagram shows all of the moving parts: | ||
|
|
There was a problem hiding this comment.
For someone that doesn't know bootkube, it may add value to briefly refer to the temporary control-plane, what it is and why it needs to be in place.
| There are a few things to note. First, generally, the control components like the API server, etc will be pinned to a set of dedicated control nodes. For security policy, service discovery, and scaling reasons it is easiest to assume that control nodes will always exist on N nodes. | ||
|
|
||
| Another challenge is load balancing the API server. Bedrock for the API server will be DNS, TLS, and a load balancer that live off cluster and that load balancer will want to only healthcheck a handful of servers for the API server port liveness probe. | ||
|
|
There was a problem hiding this comment.
Should a reference to kubeconfig v2 be added?
|
|
||
| ## Long Term Goals | ||
|
|
||
| Ideally bootkube disappears over time and is replaced by a [Kubelet pod API](https://github.com/kubernetes/kubernetes/issues/28138). The write API would enable an external installation program to setup the control plane of a self-hosted Kubernetes cluster without requiring an existing API server. |
There was a problem hiding this comment.
The document is still under debate but should be in as a PR shortly.
| The kubelet could be upgraded in a very similar process to that outlined in the self-hosted proposal. | ||
|
|
||
| However, because of the challenges around the self-hosted Kubelet (see above) Tectonic currently has a 1-4 self-hosted cluster with an alternative Kubelet update scheme which side-steps the self-hosted Kubelet issues. First, a kubelet system service is launched that uses the [chrooted kubelet](https://github.com/kubernetes/community/pull/131) implemented by the [kubelet-wrapper](https://coreos.com/kubernetes/docs/latest/kubelet-wrapper.html) then when an update is required a node annotation is made which is read by a long-running daemonset that updates the kubelet-wrapper configuration. This makes Kubelet versions updateable from the cluster API. | ||
|
|
There was a problem hiding this comment.
nit: ...implemented by the kubelet-wrapper. Then, when an update is required, a node annotation...
|
|
||
| #### API Server, Scheduler, and Controller Manager | ||
|
|
||
| Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors. |
There was a problem hiding this comment.
With deployments, one can enforce that only one (or a percentage of total) pod is to be updated at a time.
My concern here is with flags passed to each component and how that may affect the cluster, e.g. API server adds flag that requires kubelet to also specify a pairing flag or run a newly needed feature. But this could be resolved with a simple updater, as mentioned in Long Term Goals section.
| ### Highly-available Clusters | ||
|
|
||
| Self-hosted will make operating highly-available clusters even easier. For internal critical components like the scheduler and controller manager, which already know how to leader elect themselves, creating HA instances will be a simple matter of `kubectl scale` for most administrators. For the data store, etcd, the etcd Operator will ease much of the scaling concern. | ||
|
|
There was a problem hiding this comment.
Should it be made clear that said components rely on the API server for leader election?
| Self-hosted will make operating highly-available clusters even easier. For internal critical components like the scheduler and controller manager, which already know how to leader elect themselves, creating HA instances will be a simple matter of `kubectl scale` for most administrators. For the data store, etcd, the etcd Operator will ease much of the scaling concern. | ||
|
|
||
| However, the API server will be a slightly trickier matter for most deployments as the API server relies on either external load balancing or external DNS in most common HA configurations. But, with the addition of Kubernetes label metadata on the [Node API](https://github.com/kubernetes/kubernetes/pull/39112) self-hosted may make it easier for systems administrators to create glue code that finds the appropriate Node IPs and adds them to these external systems. | ||
|
|
There was a problem hiding this comment.
Again, should a reference to kubeconfig v2 be added?
There was a problem hiding this comment.
I don't know about adding a reference to something without a concrete proposal.
Good feedback on HA setup notes and some grammar fixups
|
@vishh the goal of the PR is to document the current state of self-hosted and where things are going. I don't know what you mean by "upstream bootkube"; we are trying to "upstream bootkube" by replacing it by the Kubelet API which is a separate proposal. But, without this proposal the Kubelet API doesn't have good context. |
|
|
||
| ## Practical Implementation Overview | ||
|
|
||
| This document outlines the current implementation of "self-hosted Kubernetes" installation and upgrade of Kubernetes clusters based on the work that the teams at CoreOS and Google have been doing. The work is motivated by the upstream "Self-hosted Proposal". |
There was a problem hiding this comment.
The google doc links to the upstream proposal. Why take that link out in the md version?
|
|
||
| The entire system is working today and is used by Bootkube, a Kubernetes Incubator project, and all Tectonic clusters created since July 2016. This document outlines the implementation, not the experience. The experience goal is that users not know all of these details and instead get a working Kubernetes cluster out the other end that can be upgraded using the Kubernetes APIs. | ||
|
|
||
| The target audience of this document are others, like [kubeadm](https://github.com/kubernetes/kubernetes/pull/38407), thinking about and building the way forward for install and upgrade of Kubernetes. If you want a higher level demonstration of "Self-Hosted" and the value see this [video and blog](https://coreos.com/blog/self-hosted-kubernetes.html). |
There was a problem hiding this comment.
target audience == cluster lifecycle sig?
|
|
||
| For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. | ||
|
|
||
| ## Practical Implementation Overview |
There was a problem hiding this comment.
Should we clarify that this implementation is a "2-4 self hosted cluster" as per the above definition?
There was a problem hiding this comment.
Actually, from below it looks like there's a self hosted kubelet, but not etcd. Which means given this taxonomy you can't describe the proposed implementation.
There was a problem hiding this comment.
kubelet technically isn't self-hosted because it isn't running as a daemonset because of limitations of the docker engine.
|
|
||
| ### Bootkube | ||
|
|
||
| Today, the first component of the installation of a self-hosted cluster is [`bootkube`](https://github.com/kubernetes-incubator/bootkube). Bootkube provides a temporary Kubernetes control plane that kicks tells a kubelet to execute all of the components necessary to run a full blown Kubernetes control plane. When the kubelet connects to this temporary API server it will deploy the required Kubernetes components, as pods. This diagram shows all of the moving parts: |
There was a problem hiding this comment.
"... that kicks tells a kubelet ..." has too many verbs.
|
|
||
| Upgrading these components is fairly straightforward. They are stateless, easily run in containers, and can be modeled as pods and services. Upgrades are simply a matter of deploying new versions, health checking them, and changing the service label selectors. | ||
|
|
||
| In HA configurations the API servers should be able to be upgraded in-place one-by-one and rely on external load balancing or client retries to recover from the temporary downtime. This relies on Kubernetes [versioning policy](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/versioning.md). |
There was a problem hiding this comment.
The versioning policy doesn't actually describe any supported version skew between master components. It was written under the assumption that all master components would be upgraded together and be run from the same github snapshot.
There was a problem hiding this comment.
"upgraded together" doesn't that mean that there will be a skew in a distributed multi-component system?
There was a problem hiding this comment.
It was written without thought for HA or a deployment with replicated master components:
"Different components are expected to be compatible across different amounts of skew, all relative to the master version."
"the master version" --> all master components are at a single version.
The skew mentioned in the doc is between "the master version" and node versions and also "the master version" and kubectl versions. If you look at the gce upgrade script, "upgraded together" means "we replace the VM where the components are running with a new one with new components" so they are upgraded in lock-step.
We also don't (yet) have any e2e or conformance testing for clusters with any skew in the master component versions, so while it may happen to work, we have few assurances that it will continue to work as k8s evolves.
There was a problem hiding this comment.
@roberthbailey OK, I will add this whole situation to the "Issues to be resolved section".
|
|
||
| Self-hosted Kubernetes runs all required and optional components of a Kubernetes cluster on top of Kubernetes itself. | ||
|
|
||
| The advantages of a self-hosted Kubernetes cluster are: |
There was a problem hiding this comment.
There are tradeoffs that should be highlighted too.
There was a problem hiding this comment.
what tradeoffs would you like highlighted?
| 2. **Deployment consistency:** self-hosted reduces the number of files that are written to disk or managed via configuration management or manual installation via SSH. Our hope is to reduce the number of moving parts relying on the host OS to make deployments consistent in all environments. | ||
| 3. **Introspection:** internal components can be debugged and inspected by users using existing Kubernetes APIs like `kubectl logs` | ||
| 4. **Cluster Upgrades:** Related to introspection the components of a Kubernetes cluster are now subject to control via Kubernetes APIs. Upgrades of Kubelet's are possible via new daemon sets, API servers can be upgraded using daemon sets and potentially deployments in the future, and flags of add-ons can be changed by updating deployments, etc. | ||
| 5. **Easier Highly-Available Configurations:** Using Kubernetes APIs will make it easier to scale up and monitor an HA environment without complex external tooling. Because of the complexity of these configurations tools that create them without self-hosted often implement significant complex logic. |
There was a problem hiding this comment.
There will always be environments that will require some of these external tools due to strict tolerations and regulations. We should strive to make it easier, but also allow the existing mechanisms.
There was a problem hiding this comment.
I don't understand your objection. Nothing about this proposal is removing anyone's ability to deploy Kubernetes however they wish. In this line I am highlighting a motivation for self-hosted from Brian Grants comment from 2014:
"Leveraging the systems/APIs we build for our users avoids needing to build and maintain 2 systems to do the same thing, and helps us to eat our own dogfood."
kubernetes/kubernetes#246 (comment)
You might not agree with that, and that is OK, but I don't think anything about this proposal creates impedances to using configuration tools. In fact I think the dogfooding will improve Kubernetes operational design and likely make deploying Kubernetes via config tools easier from lessons learned.
There was a problem hiding this comment.
@philips
I think @timothysc didn't say he object it :)
The words seem to compare self hosted and ambiguous "external tooling", which might not be the same level nor comparable.
It would be great to rephrase and point out that it doesn't impede other config tools in the future.
There was a problem hiding this comment.
@hongchaodeng OK, I will add that to an FAQ section in a second.
There was a problem hiding this comment.
Thanks for the feedback @timothysc. I added a paragraph after this section in d086034, PTAL.
- fixup some grammar mistakes - add skew of control plane to known issues - add links dropped from original proposal
|
@roberthbailey addressed your feedback. |
| @@ -24,20 +24,19 @@ However, there is a spectrum of ways that a cluster can be self-hosted. To do th | |||
|
|
|||
|  | |||
|
|
|||
| For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. | |||
| For example, a 0-4 self-hosted cluster means that the kubelet is a daemon set, the API server runs as a pod and is exposed as a service, and so on. While a 1-4 self-hosted cluster would have a system installed Kubelet. And a 02-4 system would have everything except etcd self-hosted. | |||
|
|
|||
There was a problem hiding this comment.
Just want to make sure you really wanted to have 02-4 here.
There was a problem hiding this comment.
yes, it could be 02-4 or 0,2-4. I just went with 02-4
…ethods Thanks to @timothysc for pointing out that I didn't make a note about the intentions of this proposal with existing install/config methods.
|
@pires addressed your question. |
|
@timothysc Could you give me an LGTM on this? I think everyone else has given me a 👍 at this point. |
|
@timothysc any final comments? I am going to merge this tomorrow otherwise. |
|
Cool, got a slack LGTM from @timothysc. Merging. |
design-proposal: add self-hosted kubernetes proposal
Prepare draft for 2017-03-16
design-proposal: add self-hosted kubernetes proposal
This had been a design doc on Google Docs but it is constantly
referenced so move it to the Kubernetes repo.
Old doc: https://docs.google.com/document/d/1VNp4CMjPPHevh2_JQGMl-hpz9JSLq3s7HlI87CTjl-8/edit#