[Roadmap] Improve kubeadm support for pluggable addons / operator driven addons

[fabriziopandini]

Kubeadm addons were subject to several discussion in the past year, but as of today there is no clear path forward (we are only supporting skip phase)

This issue is about discussing some possible alternatives for:

• supporting pluggable addons during cluster setup
• provide support for the lifecycle of operator driven addon (where e.g. the upgrade addon operation should not be managed by kubeadm)

[fabriziopandini]

I would like to point out that also some features of addon management are relevant for declarative approaches as well.
e.g. the ability to opt-out from installing/upgrading default addons (because the user wants to take control of the addon lifecycle).

[fabriziopandini]

WRT to addons, kubeadm as of today offers an integrated UX based on an detailed knowledge of the internals of the addons in scope. Let's consider CoreDNS as example:

• kubeadm knows in advance the images required by a specific version of CoreDNS, so we can pre-pull required images
• kubeadm knows in deep how the CoreDNS configuration works, so we are managing a seamless flow of kubeadm configurations to the CoreDNS configurations e.g. ClusterConfiguration.Networking.ServiceSubnet is used to determine the IP for the CoreDNS service.
• kubeadm knows in advance upgrade steps for CoreDNS, including e.g. CoreDNS config migration

While the above integrated UX is nice and desirable, maintaining such a deep knowledge of addons internals is an approach that can't scale, and in fact it is the main blocker to support pluggable addons.

So, I would like to gather opinions on a possible alternative approach:

kubeadm should

• Define an interface with all the expected behaviors a provider should support, e.g GetImages, SetConfiguration etc.;
• Trigger those behaviors for all the selected addons during init, upgrade / whenever necessary

Each addon

• should provide an implementation of the above interface

Some of those implementations will be embedded in the kubeadm (e.g. CoreDNS), thus continuing to address the most common use cases right out of the box; intestead, other implementations could be provided to kubeadm as a plugin, thus allowing any addon authors, or even the users, the chance to provide the same integrated UX for any addon/addon variants, but without requiring changes on the kubeadm codebase.

[neolit123]

could be provided to kubeadm as a plugin

if by "plugin" you mean -buildmode=plugin then this does not work on Windows, so we cannot support it.
kubetest2 and kubectl use "exacutable plugins" (not shared library ones). which means that the interface must be CLI (i'm excluding the option for an addon service on the host).

Some of those implementations will be embedded in the kubeadm (e.g. CoreDNS), thus continuing to address the most common use cases right out of the box; intestead, other implementations could be provided to kubeadm as a plugin, thus allowing any addon authors, or even the users, the chance to provide the same integrated UX for any addon/addon variants, but without requiring changes on the kubeadm codebase.

some questions here:

• how would we embed these addons in kubeadm and how would they be bundled next to the kubeadm binary?
• where would the code for the addons reside? if they are not in the kubeadm codebase this means they have to be built and released with kubeadm and included in deb/rpm packages.

what are your main objections against using operators for addons?

problems with operators that i'm aware of include:

• chicken and egg. we could ask the operator to give us an image for an addon, but this means the operator has to be running and currently kubeadm config images works without a cluster running.
• operators needs lifecycle management and present complications around CRD instance migration and upconverting user YAML on disk.

[fabriziopandini]

if by "plugin" you mean -buildmode=plugin then this does not work on Windows

I don't have a specific solution in mind for this implementation detail, but there are prior art we can explore for this like eg. kubectl plugins and hashicorp plugins.

how would we embed these addons in kubeadm and how would they be bundled next to the kubeadm binary?
where would the code for the addons reside?

TBD; the fallback solution is to continue as we are doing today by having this code in kubeadm, but some other implementation approach can be explored as well (this is somehow related to the target plugin model)

chicken and egg.

The plugin runs before the addon operator is installed, so there should be no chicken and egg problem; you can see the plugin as the addon operator installer for kubeadm.

WRT to the operator lifecycle management, the plugins should have all the required knowledge in order to provide an integrated UX with kubeadm; the nice point here is that this logic is pluggable and not anymore entangled with the kubeadm codebase

[neolit123]

i must say that i like the plugin binaries idea.

addon plugins would not be able to manage their own lifecycle (if they don't run as services), but the kubeadm operator can look for addon binaries on the nodes and call certain CLI (interface) methods such as "upgrade", "refresh" etc.

if users don't want to install the kubeadm operator they will also have the option to execute functionality on the addon binary on demand.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fabriziopandini]

/remove-lifecycle stale

[k8s-triage-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fabriziopandini]

/lifecycle frozen

[neolit123]

discussion with @kfox1111 on slack about using operators and OLM with kubeadm and the future of kube-proxy/coredns as kubeadm addons:
https://kubernetes.slack.com/archives/C13J86Z63/p1650578534276089