🪵 Add Karpenter to Airflow

[jacobwoffenden]

User Story

As an Analytical Platform engineer
I want to offer various Karpenter profiles to Airflow customers
So that they have flexibility in what instance type and size they use

Value / Purpose

The current Airflow compute types are:

• general (t3.xlarge 4 vCPU, 16 GiB)
• high-memory (r6i.8xlarge 32 vCPU, 256 GiB)

general is an EKS MNG with a desired size of 3 but can scale to 10

high-memory is an EKS MNG with a desired size of 0 but can scale to 1, if multiple high-memory jobs are scheduled they will share this instance, and if non can fit, presumably they will fail

---

In Analytical Platform Compute, we have Karpenter deployed and can create node pools that can provide different types of compute

Useful Contacts

@jacobwoffenden

User Types

No response

Hypothesis

If we create imports that create resources (requests and limits, tolerations and affinity
Then users can use them to utilise our compute platform better

Proposal

We already have general-on-demand, general-spot and gpu-on-demand, so we should enhance this offering, and have a bigger mix. Also introduce resource limits and requests profiles so Karpenter can find an appropriate instance

Additional Information

No response

Definition of Done

• Karpenter node pools diversified
• Import for resource limits and requests created
• User docs have been updated
• Another team member has reviewed
• Tests are green