📖 Update Airflow Validation Script to enforce exemption list

[jhpyke]

User Story

As a maintainer of Airflow
I need to be able to enforce that all non-exempt DAGs run on APC
So that we can smoothly move towards shutting down the existing EKS clusters

Value / Purpose

By enforcing this via the validation script, we will be able to ensure that no new workloads are able to be scheduled on the old EKS clusters, and that only known exempt ones are able to continue to update without migrating.

Useful Contacts

@jhpyke

User Types

No response

Hypothesis

If we... prevent users writing bad DAGs that use the old cluster
Then... we will largely be ready to shut said cluster down.

Proposal

Update the validation script currently used in the repo - right now it is configured to allow use of compute_context="aws" OR enforce the compute context to match the environment for the new clusters. From September 9th, we want it to instead only allow compute_context="aws" if you are explicitly exempted to do so.

Suggested approach:

• Encode the exempted DAG list as a file (json? something structured) in the repo
• Read in this file during validation, and compare the name of the DAG being validated to the exemption list
• If file is exempt, allow compute_context="aws" and associated arguements
• Else throw an error when the user attempts to supply this compute_context

Additional Information

No response

Definition of Done

• Validation successfully blocks non-exempt DAGs from merging while containing the old cluster_context
• Validation successfully allows exempt DAGs to do the same
• This has been tested in a PR with the updated validation to prove it works as intended.

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.

[github-actions]

This issue is being closed because it has been open for a further 7 days with no activity. If this is still a valid issue, please reopen it, Thank you!