Configurable installation for versions and OS image matrix

[krousey]

Currently in the GCE actuator, we compile in a script template that installs Kubernetes on Ubuntu. If we have a bug, or need to change the startup logic slight, we have to recompile and redeploy the controller.

The GCE actuator (and perhaps others but right now let's just limit it to GCE), should have a structured config that can be read from file (and configmap) that can allow us to configure a mapping of supported Kubernetes versions and OS image tuples to startup scripts on how to install and set them up. This would allow us to

• Support multiple target OSes without having to add support directly to the controller binary
• Use preloaded images. No install steps, just configure.
• Allow admins to have a list of vetted setup combinations.
• Allow admins to bring their own OS and installation method without having to worry about building their own controller.

At the very minimum, we should be able to map a Kubernetes version, CRI version, OS identifier to an image and startup script. I imagine this will affect the GCE actuator and libraries and the GCE prodiver config.

[krousey]

cc @kcoronado

[kcoronado]

I came up with an example yaml file for the config. I'm not sure what to name the struct yet, though (InstallationConfig, SetupConfig?).

- os: ubuntu-1710
  role: master
  kubeletVersions:
  - 1.8.2
  - 1.8.3
  controlPlaneVersions:
  - 1.8.2
  - 1.8.3
  containerRuntimes:
  - name: docker
    version: 1.12.0
  - name: docker
    version: 1.12.1
  image: projects/ubuntu-os-cloud/global/images/family/ubuntu-1710
  startupScript: |
    #!/bin/bash
    ...

Using lists of versions in one instance will prevent the same startupScript from being copied and pasted multiple times in the file.

[jessicaochen]

I am for making things more configurable.

I do have some thoughts on the suggested yaml. A situation to consider: You can have the node version behind the master version but you cannot have a master version behind a node version.

Looking at the suggested configuration, it seems to imply that all the listed versions are cross compatible. Eg. we could have a 1.8.3 kubelet with a 1.8.2 control plane. Do we have plans on how to disallow/prevent bad combinations?

Versioning is interesting too from the perspective that how you pull the code affects the version. Are they released package versions or version labels on the git repo? Would you ever want to pull a unofficial version of the kubelet for test/dev purposes? (the answer is yes if we want to rebase e2e tests on top of this)

[kcoronado]

That's a good point. I wrote that with the assumption that the versions would be both cross compatible and able to use the same startup script.

Since they're not necessarily cross compatible, I could reuse the MachineVersionInfo struct here and make a list of valid combinations. That way each entry can still use the same startup script so it won't get duplicated in the file. Plus, that makes comparing the given machine versions to the available config versions easier 😄

Take two:

- os: ubuntu-1710
  role: master
  versions:
  - kubeletVersion: 1.8.2
    controlPlaneVersion: 1.8.2
    containerRuntime:
      name: docker
      version: 1.12.0
  - kubeletVersion: 1.8.3
    controlPlaneVersion: 1.8.3
    containerRuntime:
      name: docker
      version: 1.12.1
  image: projects/ubuntu-os-cloud/global/images/family/ubuntu-1710
  startupScript: |
    #!/bin/bash
    ...

[rsdcastro]

This issue was moved to kubernetes-sigs/cluster-api#85