⛰ Move Airflow Compute to APC cluster

[jhpyke]

Brief summary of what this Epic is

This Epic captures the work required to move Airflow jobs from the existing Airflow Cluster(s) to APC. This work was triggered by the failure of Kube2IAM, but represents a long term plan to improve the stability of the cluster

Other information

Overall plan is:

• Create a new release of mojap-airflow-tools to be available within dev. This will have conditional logic that triggers the new cluster/IRSA on our basic pods ONLY.
• Release it, and update requirements file for airflow-dev to reflect the new release. This should take effect within 20 mins - if not check scheduler logs (requirements-ip-install-<date/time>.log)
• Run a test pod using the BasicPodOperator to ensure it can accept the new service-account info correctly.
• Assuming testing is smooth, release comms to users on how to update their images to be accessible from new cluster (new image release pipeline already made by @jacobwoffenden)
• Work with users with specific networking requirements to ensure these are met by new cluster
• Transition new user workloads to APC.

#4472

Airflow Migration (Phase 1) - google doc
Roadmap item: Upgrade Airflow (Due date Jan '25)

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.