Put DAGs in a S3 Bucket

[davvar38]

Hello Airflow community,

We set up Airflow on an Amazon EKS cluster, with an Amazon EFS for DAG files storage but it created several issues of performance, and I was wondering if storing these DAG files into a S3 bucket was possible and, if not, which part of code I should modify because I cannot find where the DAG files reading is actually made.

Thank you in advance,

[pingzh]

i think it might be even slower to store dag files to s3 bucket.

which airflow version are you using? and do you use KubernetesExecutor or CeleryExecutor?

[davvar38]

The airflow version is 2.2.4 with KubernetesExecutor

[potiuk]

Yes. the EFS has performance issues that others complained about - they were usually solved by reserving IOPS for them (basically adding money to get better performance).

I personally recommend going git-sync route, for me this is the most stable and easiest way to make good performance of file syncing works (and you usually anyhow keep your DAG files in Git more often than not). You can use S3 if you make some S3 -> local file syncing but you are basically on your own to do such solution (though MWAA is apparently using it so it is possible).

Basically you have to make sure that all the components of airlfow see the DAG folder in specific location and files are synced there. There is no "place" in Airflow to do that - airlflow simply reads the files from that folder and how the files are synced to the folder is out-of-the-realm of Airflow as application. This is generally a deployment issue how you sync the files.

In our official chart you can see some of the options how syncing can be done https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html - and take it as a "base". Or (actually I'd recommend that) you could use our chart to deploy Airflow - it's rather versatile and solves a lot of deployment issues like that.

[davvar38]

Yes I've read this doc and the git syncing solution paragraph but my issue is that we must specify a git branch, however we sometimes need to modify DAGs and test them feature deployment with a feature branch, so how can we handle this problem?

[potiuk]

Watch this video from the summit https://www.youtube.com/watch?v=uA-8Lj1RNgw

[cdmikechen]

@davvar38 You can try https://github.com/cdmikechen/airflow-dag-operator
This is a CRD based definition of DAG, which I think is a good solution on the cloud.

[ismailsimsek]

Is it possible to run mount-s3 with dockers entry-point? that should mount s3-bucket with startup which then could be pointed as dags_folder. seems like except editing a file, reading and writing is supported by it.

[potiuk]

Possible - yes. Whether it work well - you will need to test it. it might also be that there will be some problems that I am not able to guess.

[persinac]

FWIW -> I PoC'd a PV with mountpoint on a dev box and it works fine. I even used AWS S3 Express to help with any latency, works as expected, DAGs load and run without a hitch. I'd be interested to hear how this would perform in a large production environment.