Optimise Pod Handling in Machine Roll-Out

[vlerenc]

When we do a rolling update, we don't give Kubernetes any hints to schedule the drained pods on the new machines rather than the olds ones, right? This in turn means, a rolling update may hit a pod more than once. Is there a chance to "guide" the scheduler in scheduling the displaced pods rather on the new than the old machines, maybe with some temporary node labels but ideally without modifying/patching the deployment/pod specs, which would be pretty messy? Maybe, the easiest solution could be to actually cordon the old machines the moment the first batch of new machines joined the cluster (I am wary to do it earlier because that could mean that pods can't be scheduled anymore until new machines have joined the cluster). What do you think?

[hardikdr]

At the moment we do not ensure that pods evicted due to Rolling-update do not get back to old machines.
As you already suggested, It would be nice to cordon all the old-versioned machines once rolling-update starts.

[bergerx]

cordon the old machines the moment the first batch of new machines joined the cluster

This could be a dangerous assumption that all users will have maxSurge>0. But would still make sense after gardener/gardener#274 is merged, if i get it right.

Another alternative could be applying PreferNoSchedule taint to the old machines rather than cordon them all at once. Seems like a safer alternative that could also cover the maxSurge=0 case, but i'm not really sure.

Also i thought this should be an already solved problem, and maybe we could get benefit from the work of others. But seems like this specific case is not covered by most of the major kubernetes providers:

• https://docs.openshift.org/latest/upgrading/index.html#install-config-upgrading-strategy-inplace
• https://console.bluemix.net/docs/containers/cs_cluster_update.html#worker_node
• https://cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster#about
• https://docs.microsoft.com/en-us/azure/aks/upgrade-cluster
• https://kubernetes.io/docs/getting-started-guides/ubuntu/upgrades/#patch-kubernetes-upgrades-for-example-1-9-0-1-9-1

Also worth checking in related kubernetes SIGs.

[vlerenc]

Yes, #274 was necessary in case the cluster is minimal and all resources are used up, which also means, that when it's merged, maxSure must be at least 1, but that it is since the beginning.

We actually hope to find some time later to make the update behaviour more configurable - through the end user that knows the work load (best). At the latest, when we see very large clusters, that will become a necessity. We used to work with Bosh 2 years ago and know that you if you have thousands of machines, you need to work in large waves (significant percentages of the machines) and need hints by the workload owners, otherwise this can't work. At the moment we run with a low maxSurge>0, simply because we respect PDBs, but need to terminate uncooperative pods/nodes after a certain grace period, which means that a low number is safe. But theoretically your workload could be slow to start and that could mean, that hard limits cause too many pods to become unavailable at the same time.

Yes, the others have no great solutions either and yes, PreferNoSchedule is the safer/better way, thank you!

[vlerenc]

@amshuman-kr @prashanth26 @hardikdr Can we implement something here to improve the shoot control plane availability (PreferNoSchedule)?

[hardikdr]

Thanks, PreferNoSchedule sounds good to me, we could basically taint all the old-nodes while rolling-update, this should try not to schedule the pods on old-nodes unless new-ones are not available.

• On a side note, we will then enable mcm to sync taints to/from machine-objects from/to node-objects - which would anyway be a useful function.

[prashanth26]

I have opened up this issue, that would help us in allowing us to specify taints on nodes which would help us in adding taints and tolerations to a node.

After which we could add this taint of PreferNoSchedule while doing a rolling update as mentioned by @hardikdr above.

Thanks & Regards,
Prashanth

[amshuman-kr]

PreferNoSchedule looks good to try and attempt.

[prashanth26]

Hi @ggaurav10 & @hardikdr ,

Could you review this PR when you get time? I have updated the testcases.