Delay upgrade of cluster nodes

[seimic]

Hi,

is there a way to delay the upgrade of nodes in a cluster?
I mean to upgrade one by one but with a delay of lets say 5 minutes to make sure all pods
on the upgraded server are in ready state.
There are some deployments (cilium and mongodb for example) which need more time to
come up and if the upgrade goes too fast all mongodb cluster nodes are in unready state.

Example: Upgrade of 3 nodes RKE2 cluster (3 master nodes only).

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: rke2-server-1.21.8-plan
  namespace: system-upgrade
  labels:
    rke2-upgrade: server
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
       - {key: rke2-upgrade, operator: Exists}
       - {key: rke2-upgrade, operator: NotIn, values: ["disabled", "false"]}
       - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/rke2-upgrade
  version: v1.21.8+rke2r1

The upgrade finishes faster than the initialization of the mongodb nodes in this cluster.
This results in an unnecessary downtime of the deployment.

Kind regards,
Michael

[Martin-Weiss]

I thought "drain" should do the trick to ensure the workload is moved to an other node before the current one is upgraded..

[seimic]

Those apply only if you modify the replicaset (rescale, upgrade etc.). In case of a cluster upgrade the cluster node being upgraded goes nuts for some time. Until the upgrade is finished, nothing is accessible. After this mongo initializes but slower than the whole upgrade of all nodes. This results in all instances in init-state. After about 1 minute everything is fine again... at least at mongo site. Anyhow, I upgrade them now one by one.

[Martin-Weiss]

I thought with the PDB you can ensure that if you have i.e. replica=2 and "delete one pod due to a drain" - you can only drain the second one after we are back on 2 healthy ones.. - https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ ..

[seimic]

Sure, but the cluster upgrade doesn't care for single deployments. It just kicks old rke2 pods and installs new ones. the whole cni restarts too. And I don't want to drain nodes. Can't move all storage and other services from a to b and back again only because of restart of a single node. This makes things unnecessary complicated.

[Martin-Weiss]

In case you can/do not want to do the drain to allow k8s to do the work - you can also adjust the image, create your own script and put tests / delays into it.. similar as here https://github.com/rancher/system-upgrade-controller/blob/master/examples/suse/sles.yaml... problem with a "sleep" is that you might not know the right timing as this also might be related to load/performance..

[seimic]

OK, thanks. I wanted to avoid doing anything like this... the upgrade-controller replaces our old scripts (ssh-provisioning via terraform to upgrade nodes). I'll see... next upgrade maybe in 3 months. Now I have enough other issues to solve :)