🚚 Convert Kube2IAM-based Airflow pod to using IRSA, and capture process

[jhpyke]

Describe the bug.

The current Kube2IAM deployment does not have support for IMDSv2, which means any new nodes created cause Kube2IAM to fail, due to the new default being IMDSv2 mandated. Instead, we should be converting our pods to use IRSA, so they can successfully run under the new standard.

To Reproduce

1. Launch a new node instance in Airflow without IMDSv2 optional
2. Attempt to schedule a pod
3. Watch Kube2IAM fail in pod logs to supply role

Expected Behaviour

User pods should recieve roles correctly with IMDSv2 enabled.

Additional context

AWS's guide on setting up IRSA for an EKS Cluster can be found here. We already have an OIDC provider for each cluster, so instead we should be looking to create a service account that is assumable by one of our pods, and prove that a DAG can be written that successfully picks it up.

Acceptance Criteria

• A service account should be created that can assume the role used by our 'example' DAGs in dev.
• examples.use_high_memory and examples.daily_test should be able to get roles using IRSA alone.
• Any modifications required to the current role policies are known.
• A writeup on what would be required to extend this process to all DAGs should be produced.

Out of Scope

• Prod (for this ticket)
• The work on the airflow repo required to change how the roles are created.

[jhpyke]

Progress as of Sign-off:

• PR Created with correct chart for release in dev
• Plan shows changes to node groups that need to be backported into code before deployment to avoid disruption to other environments.
• Updated chart adds support for IMDSv2, but due to unexpected changes in plan have not been able to test deployment.

Next Steps:

• Backport changes into terraform until plan shows only changes to helm chart
• Import existing release to ensure settings are retained correctly (note - no import block for helm charts so will temporarily block stack)
• Make backup of values in existing chart (aws-vault into data-prod, run aws eks --region eu-west-1 update-kubeconfig --name airflow-<env> subbing for correct environment to auth with cluster, then helm -n kube2iam-system get all kube2iam to get the current config)
• Deploy updated chart
• Run examples.use_high_memory_node. If it fails the first time, this may be due to it scheduling faster than Kube2IAM. As long as it passes on a second try, then kube2IAM is behaving normally. If pods fail to schedule, then use kubectl logs -n kube2iam-system <name of kube2iam pod> to see the logs and diagnose what's not working.

[AntFMoJ]

airflow-monitoring DAG successfully migrated from Kube2IAM to IRSA in airflow dev. Next step is to make changes to migrate the rest of the DAGs in dev.

[jacobwoffenden]

APC OIDC added to APDP

[AntFMoJ]

IRSA has been successfully tested in multiple DAGs in Airflow Development, including a DAG owned by the CJS dashboard team. A test will be performed by COP 28/06 on a data engineering DAG in Airflow Development.

Update to mojap-airflow-tools required to allow IRSA to work in Airflow Prodiction.

Beginning to write up instructions, which will be added to the Airflow user guidance once it has been reviewed by Airflow users.

[jhpyke]

Todo: Update validation script to ensure that env variable matches environment folder used in airflow repo