🔧 Update Kubernetes Version on all Airflow Clusters

[jhpyke]

User Story

Our Kubernetes version for the existing Airflow setup is 1.24, which is going into extended support on Jan 31st 2024 (end of this month). At this point, we will incur higher costs with keeping these clusters running. Given we don't rely on anything that is specific to lower versions, we should be seeking to upgrade the existing clusters to a supported version. We cannot afford to wait for migration, as this is unlikely to occur prior to the cutoff date.

Value / Purpose

This will prevent us from incurring unnecessary additional costs, and allow us to bring our systems up to date.

Useful Contacts

Jacob Hamblin-Pyke

User Types

Airflow Users

Hypothesis

If we... [do a thing]
Then... [this will happen]

Proposal

Upgrade in Sandpit to test, then do a standard deployment via Dev to Prod.

Steps for Upgrade:

• Update base version of Kubernetes Cluster from 1.24 to 1.25
• Once the above has succeeded, create add-on releases for kube-proxy, VPC-CNI, and CoreDNS. Versions should match those deployed in Dev
• Once all add-ons are deployed, upgrade the AMI releases in the individual node groups to one appropriate for 1.25
• Test this has all succeeded using monitoring DAGs.

Additional Information

Supported Kubernetes Versions in AWS

Definition of Done

• All airflow EKS clusters are on kubernetes >1.24
• Any issues found during testing are remediated before rollout.
• Any helm charts that need updating are updated accordingly.

[jacobwoffenden]

sandpit and dev are done

sandpit was clickopsed

dev had an issue with kube-proxy, @jhpyke please add resolution notes

prod to be done on Tuesday 16/01, comms to go out on Monday 15/01

prod rescheduled to thurs 18/01 comms on weds

[Emterry]

Addons for vpc-cni, kube-proxy, and coredns in code successfully deployed in dev

prod to be done Tuesday 23/01, Comms to go out Monday 22/01

[jhpyke]

Following comms on Monday and Today, will be running the upgrade in Prod. Updating ticket with order of operations for upgrade.