Move the DAGs folder to a shared Persistent Volume

[LucaCinquini]

Description: Instead of "baking" the DAGs folder into each SPS Docker image, move it to a shared location on a Persistent Volume, where updates can be immediately propagated to all Airflow components.

See https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-an-externally-populated-pvc for instructions.

Acceptance Criteria:
o Demonstrated execution of an existing DAG that is written to the shared PV.
o All nightly tests keep working when executed versus the updated SPS with shared DAGs folder.

[drewm-jpl]

Notes for implementation:
Using an S3 Mountpoint-backed persistent volume:
apache/airflow#24049 (reply in thread)

[drewm-jpl]

I attempted to use S3 MountPoint as a source for the PV, but ran into an issue which seems to be related to SELinux and the golden AMI. More info here: awslabs/mountpoint-s3-csi-driver#169

I plan on opening an AWS Enterprise Support ticket to investigate this since we will likely want to use MountPoint at some point in the future. We can proceed with using EFS as the PV source for now.

[LucaCinquini]

In the meantime, Drew's changes seem to work.

I was able to deploy the system using this branch, and I was able to copy a modified DAG to one of the scheduler pod's /opt/airflow/dags folder and see the changes reflected in the UI. Note that copying the modified DAG to one of the webserver pods does not seem to work - the UI never showed the updated DAG.

Closing the ticket anyway.