Merge pull request #9349 from ari-becker/networking-docs

Cleanup networking docs

docs/networking.md

@@ -1,10 +1,18 @@
 # Kubernetes Networking Options
 
+## Introduction
+
+Kubernetes has a networking model in which Pods and Services have their own IP
+addresses. As Pods and Services run on servers with their own IP addresses and
+networking, the Kubernetes networking model is an abstraction that sits
+separately from the underlying servers and networks. A number of options,
+listed below, are available which implement and manage this abstraction.
+
 ## Supported networking options
 
-The following table provides the support status for various networking providers with regards to Kops version: 
+The following table provides the support status for various networking providers with regards to Kops version:
 
-| Network provider | Experimental | Stable | Deprecated | Removed | 
+| Network provider | Experimental | Stable | Deprecated | Removed |
 | ------------ | -----------: | -----: | ---------: | ------: |
 | AWS VPC | 1.9 | - | - | - |
 | Calico | 1.6 | 1.11 | - | - |
@@ -19,14 +27,11 @@ The following table provides the support status for various networking providers
 | Romana | 1.8 | - | 1.18 | 1.19 |
 | Weave | 1.5 | - | - | - |
 
+### Which networking provider should you use?
 
-The networking options determines how the pod and service networking is implemented and managed.
-
-Kubernetes Operations (kops) currently supports 3 networking modes:
+Kops maintainers have no bias over the CNI provider that you run, we only aim to be flexible and provide a working setup of the CNIs.
 
-* `kubenet` Kubernetes native networking via a CNI plugin.  This is the default.
-* `cni` Container Network Interface(CNI) style networking, often installed via a Daemonset.
-* `external` networking is done via a Daemonset. This is used in some custom implementations.
+We do recommended something other than `kubenet` for production clusters due to `kubenet`'s limitations, as explained [below][#kubenet-default].
 
 ### Specifying network option for cluster creation
 
@@ -51,6 +56,12 @@ Users running `--topology private` will not be able to choose `kubenet` networki
 requires a single routing table. These advanced users are usually running in multiple availability zones
 and as NAT gateways are single AZ, multiple route tables are needed to use each NAT gateway.
 
+Kubenet is the default networking option because of its simplicity, however, it should not be used in
+production clusters which expect a gradual increase in traffic and/or workload over time. Such clusters
+will eventually "out-grow" the `kubenet` networking provider.
+
+For more on the `kubenet` networking provider, please see the [`kubenet` section of the Kubernetes documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#kubenet).
+
 ### CNI
 
 [Container Network Interface](https://github.com/containernetworking/cni) provides a specification
@@ -69,20 +80,32 @@ Several CNI providers are currently built into kops:
 * [Romana](networking/romana.md)
 * [Weave](networking/weave.md)
 
-The manifests for the providers are included with kops, and you simply use `--networking <provider-name>`.
-Replace the provider name with the names listed above with you `kops cluster create`.  For instance
-to install `calico` execute the following:
+Kops makes it easy for cluster operators to choose one of these options. The manifests for the providers
+are included with kops, and you simply use `--networking <provider-name>`. Replace the provider name
+with the name listed in the provider's documentation (from the list above) when you run
+`kops cluster create`.  For instance, for a default Calico installation, execute the following:
 
 ```console
 kops create cluster --networking calico
 ```
 
-### External CNI
+Later, when you run `kops get cluster -oyaml`, you will see the option you chose configured under `spec.networking`.
+
+### Advanced
+
+Kops makes a best-effort attempt to expose as many configuration options as possible for the upstream CNI options that it supports within the Kops cluster spec. However, as upstream CNI options are always changing, not all options may be available, or you may wish to use a CNI option which Kops doesn't support. There may also be edge-cases to operating a given CNI that were not considered by the Kops maintainers. Allowing Kops to manage the CNI installation is sufficient for the vast majority of production clusters; however, if this is not true in your case, then Kops provides an escape-hatch that allows you to take greater control over the CNI installation.
+
+When using the flag `--networking cni` on `kops create cluster`  or `spec.networking: cni {}`, Kops will not install any CNI at all, but expect that you install it.
 
-When using the flag `--networking cni` on `kops create cluster`  or `spec.networking: cni {}` Kops will not install any CNI at all, but expect that you install it.
+If you try to create a new cluster in this mode, the master nodes will come up in `not ready` state. You will then be able to deploy any CNI DaemonSet by following [the vanilla kubernetes install instructions](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network). Once the CNI DaemonSet has been deployed, the master nodes should enter `ready` state and the remaining nodes should join the cluster shortly thereafter.
 
-When launching a cluster in this mode, the master nodes will come up in `not ready` state. You will then be able to deploy any CNI daemonset by following vanilla kubernetes install instructions. Once the CNI daemonset has been deployed, the master nodes should enter `ready` state and the remaining nodes should join the cluster shortly thereafter.
+#### Important Caveats
 
+For some of the CNI implementations, Kops does more than just launch a DaemonSet with the relevant CNI pod. For example, when installing Calico, Kops installs client certificates for Calico to enable mTLS for connections to etcd. If you were to simply replace `spec.networking`'s Calico options with `spec.networking: cni {}`, you would cause an outage.
+
+If you do decide to take manual responsibility for maintaining the CNI, you should familiarize yourself with the parts of the Kops codebase which install your CNI ([example](https://github.com/kubernetes/kops/tree/master/nodeup/pkg/model/networking)) to ensure that you are replicating any additional actions which Kops was applying for your CNI option. You should closely follow your upstream CNI's releases and Kops's releases, to ensure that you can apply any updates or fixes issued by your upstream CNI or by the Kops maintainers.
+
+Additionally, you should bear in mind that the Kops maintainers run e2e testing over the variety of supported CNI options that a Kops update must pass in order to be released. If you take over maintaining the CNI for your cluster, you should test potential Kops, Kubernetes, and CNI updates in a test cluster before updating.
 
 ## Validating CNI Installation
 
@@ -98,14 +121,12 @@ Here are some steps items that will confirm a good CNI install:
 The sig-networking and sig-cluster-lifecycle channels on K8s slack are always good starting places
 for Kubernetes specific CNI challenges.
 
-## How to pick the correct network provider
-
-Kops maintainers have no bias over the CNI provider that you run, we only aim to be flexible and provide a working setup of the CNIs.
-
-We do recommended something other than `kubenet` for production clusters due to `kubenet`'s limitations.
-
 ## Switching between networking providers
 
 Switching from `kubenet` providers to a CNI provider is considered safe. Just update the config and roll the cluster.
 
 It is also possible to switch between CNI providers, but this usually is a disruptive change. Kops will also not clean up any resources left behind by the previous CNI, _including_ the CNI daemonset.
+
+## Additional Reading
+
+* [Kubernetes Cluster Networking Documentation](https://kubernetes.io/docs/concepts/cluster-administration/networking/)