Interactively Upgrade EKS Worker Node

[mrichman]

Upgrading EKS worker nodes to the latest version (or to a specific version) should be as easy as clicking a button in the management console, or a single AWS CLI command.

For comparison, ECS offers the ability to upgrade the agent using both the management console and CLI.

[jaredeis]

Draining nodes before removing is important for us for this as well. We are working on improving our automation to do this but would be nice if this was part of the worker upgrade process.

[nukepuppy]

Ditto.. this is real pain point as it's just so ad-hoc/hacky really shouldn't involve interactions with cloudformation directly (that's AWS problem not mine IMO)

[christopherhein]

@mrichman @jaredeis @nukepuppy thanks for submitting this, would you want to this to update the kubelet running on the worker nodes in your account in-place or how would you like to see this type of operation happen?

[jaredeis]

I don't have enough K8s experience to know if just updating kubelet in-place is best practice, or if draining the node off first would be better. I envision some workflow where for every node that's in the ASG, it replaces them one by one while draining off the obsolete nodes. That's how we are trying to do it right now, but our current solution using SSM and Lambda doesn't seem to work as scale (it times out and the process fails).

[mrichman]

I second @jaredeis comments. Whatever the best practice is. My assumption is a rolling update to preserve capacity in the ASG.

[vincentheet]

Would be great if the nodes are drained with respect to PodDistrubtionBudgets: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#how-disruption-budgets-work
Adding a new (updated) node, then draining an old one until all old nodes are updated is a great thing to see happening. The applications keeps available (when more than 1 replica is requested of course) and all the nodes get updated. Then there is no need to worry about downtime anymore.

I agree the current process is way to complex and would involve (a lot of) scripting to make this happening in a sub-optimal way.

[christopherhein]

Thanks for that @mrichman and @jaredeis really helpful. If you wouldn't mind as you do go through your updates this was something that was discussed yesterday at the SIG-AWS meeting at KubeCon and brought up by @spiffxp, overall the sig-cluster-lifecycle needs help in discussing the ways people are upgrading their clusters to really get to a prescriptive approach for doing so.

This means your experiences are incredibly useful, if you could document and make notes and share about what you are doing, and what is working well vs not you will not only help this proposed feature but also will help the Kubernetes community as a whole.

[christopherhein]

Would be great if the nodes are drained with respect to PodDistrubtionBudgets: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#how-disruption-budgets-work
Adding a new (updated) node, then draining an old one until all old nodes are updated is a great thing to see happening…

Is this not what you are seeing when you kubectl drain <node> as far as I understand it drain is supposed to respect the PodDistrubtionBudgets? And as long as you cordon your old nodes it should be replacing them in place on the new instances.

[jaredeis]

As I understand it, drain does respect PDB so that shouldn't be an issue as long as you have proper PDBs.

@christopherhein, where would you like me to send what we are trying now (that's not really working)? I would be happy to do so.

[mrichman]

@jaredeis Would love to review your strategy as well. Could be too noisy to post here, but perhaps a link to a Google Doc or similar?

[christopherhein]

If you and your org is okay you could post that here, or maybe a gist and a link?

Whatever medium it would be nice to make sure we can also get this into the hands of sig-cluster-lifecycle too.

[vincentheet]

@christopherhein @jaredeis yes, you are correct. The drain command respects the PodDistruptionBudgets according to the docs. Would be great if that is supported with an interactive upgrade. To clarify, cordon itself doesn't replace the nodes, it marks them as unavailable for "new" pods.

[jaredeis]

I will work up something tomorrow and post the link here. Thanks for listening to customers!

[christopherhein]

Of course, thank you for being a part of this!

[patrickleet]

with GKE you can just set node pool to autoupgrade to master version, which would be nice

[mrichman]

Do you know what strategy GKE uses to perform the upgrade?

[patrickleet]

I'm not sure if it has a specific name, but it's a rolling update of the worker nodes, one at a time, within the pool

[Graham-M]

We do this at the moment with a combination of:

# drop the pods from a node
kubectl drain <node-name>

and then:

# remove a node gracefully from the ASG, including 
# allowing it to be removed and drained from the ELB
aws autoscaling \
  terminate-instance-in-auto-scaling-group \
  --no-should-decrement-desired-capacity \
  <instance-id>

....and then waiting for all nodes to be in the Ready state before we go and trash the next one.

[jaredeis]

Maybe this will be enough info, but still short enough to fit here. Our requirements from the architect were that any change to the ASG (userdata, AMI, etc) would trigger a node drain before the existing node is removed. So one of our enginners came up with this solution:

1. The ASG has a lifecycle hook that sends a message to SNS that a node is terminating after a new node has been stood up to replace it.
2. SNS triggers a lambda that triggers drain directly on the node itself using the ssm-agent.
3. The result of the drain is returned to lambda (either success, timeout, or failure)
4. Lambda output sent to cloudwatch for troublehshooting.
5. If the drain is a success, the node is terminated and the next node is drained.
6. If it fails, an alert is sent to our alerting platform via SNS and the stack update is rolled back, deleting the new nodes and creating as many of the old nodes as needed to return to the original desired count.

This worked fine in testing, but now that we have devs using it and there are more active pods on the nodes, the drain times out since the lambda can only run for 15 min. We have a pause time in our ASG between each node of 15 min, but the actual drain function that the lambda runs has an 8 min time out so the whole lambda has time to run. So we can only do 4 nodes per hour, which isn't good.

Obviously this is complex, and it's not working at scale anyway as some nodes are taking more than 8m to drain. I know there are probably some improvements to this design we can do, but I know there has to be a better way than this.

[nukepuppy]

I think reviewing just what a kops rolling update does https://github.com/kubernetes/kops/blob/master/docs/cli/kops_rolling-update.md

In theory a button to do the rolling update for KOPS and it would mostly just work (even though there be dragons sometimes ya know)... behind the scenes of what the "button" would do in the context of EKS sounds like its already understood.. but just not wired together - so end users need to wire it up.. but probably not the best experience in end

UX wise.. what seems to be desired.. some command or process issued and the cluster will be in the desired state at the end of it...

however it's a bit fuzzy.. how much do we want end users to control the nodes? if a lot (for user_data scripts to do monitoring/asset tracking / system users or other processes) it becomes a double edged sword in a way.

[jlongtine]

@jaredeis Could you use a Step Function to watch the result of the drain, instead of relying strictly on the Lambda? That way you won't have the issue with the Lambda timeout.

[jaredeis]

@jlongtine If we decide to improve the functionality as is, then yes I was going to work on some way to make this more event based. However, I think we are going to have to rethink the whole thing and stand up another ASG and migrate to it. Just have to figure out how to automate that and make our existing pipeline to deploy EKS (and a few other components) still work idempotently.

[tabern]

This is partially fulfilled by #139
Keeping open until upgrades supported in AWS management console.

[harshal-shah]

We have implemented a kops like rolling update functionality here which works fine for us.

[mikestef9]

Support worker node upgrades is now available through the EKS API and Management Console

[inductor]

@mikestef9 Hey, why is this on "coming soon" status? :)

[tabern]

@inductor our automation missed it! Moving now.