Suggesting to add documentation on how to run Airflow on Windows using WSL

[TJaniF]

What do you see as an issue?

Hi! :)

There currently isn't a page discussing how Airflow can be run on Windows with WSL (other than one mention of WSL on "Installation from PyPI").
We (@paolaperaza, @kentdanas, @jwitz and I) discovered this when discussing what to do with the old and outdated "Running Airflow on Windows 10 & WSL" guide on the Astronomer website. This guide will be deprecated soon due to website restructuring, but is continuously receiving a lot of traffic indicating there is an interest of users for instructions on the topic.

Solving the problem

We wanted to suggest adding information about the possibility of using WSL to run Airflow on Windows. Potentially as a sub page to Home/Installation.

Anything else

I'm happy to PR a draft of an updated version of the old guide.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

I think running Airlfow in WSL2 in production is not something we should recommend, There are multiple implications and problems (ranging from filesystem, networking issuess down to running Airflow in unattended way on Windows server.

IMHO making Airlfow to run in "Production" as opposed for "Development of Airflow" should not happen at all (and adding this info to Installation" page implies running Airlfow in production in WSL2 is possible and recommended).

I woudl be totally against any entry that suggest you can install Airlfow in WSL2 that would suggest that this is a setup people can "expect to work" and "get support from the community" about. If you have experience about runing Airlfow in production on WSL2 you could write Blog post about it and make a link from the "Ecosystem" page - but that would be maximum that we should go for.

We already have documentation about running Airflow in WSL2 for "Airflow Development" - i.e. contributing to Airlfow in our Github pages - to run Dockerized Breeze development environment in WSL2 https://github.com/apache/airflow/blob/main/BREEZE.rst#docker-in-wsl-2 . One thing that is likely missing (and where you could contribute to is to add chapter about it in local virtuelenv case: https://github.com/apache/airflow/blob/main/LOCAL_VIRTUALENV.rst - there we have a generic venv case but likely adding WSL2 there and some specific things like filesystem use etc. might be useful.

However - what we should really focus on is to make Airlfow run on Windows natively. We are not far from that - with the completion of https://github.com/apache/airflow/projects/13 (Rewrite Breeze to Python), merging #26105 (git clone on Windows) and successufl completion of #12874 and #10388 - we will be able to make first - development natively on Windows and later possibly even running Airflow in Production on Windows natively.

This is far more important direction to follow and Rather than documenting how you can run Airlfow in WSL2 I'd rather see steps to implement Native Windows compatibility for Airflow.

[TJaniF]

Thank you for the detailed response! I'm sorry I wasn't aware of the docker in WSL content in the breeze documentation and how close native Airflow on Windows is getting, that is very helpful to know! I'll report back to the team and sorry again for bothering 🙂

[potiuk]

No problem whatsoever :). And yeah- we are getting very close

[uranusjr]

I’m not sure there’s much to add, really. After you spin up WSL2, you’re essentially in Linux, and can simply follow the “normal” deployment guides.

[potiuk]

I’m not sure there’s much to add, really. After you spin up WSL2, you’re essentially in Linux, and can simply follow the “normal” deployment guides.

Actually - there are quite a few caveats that have been nicely captured in the BREEZE docs (but due to us getting much closer to running airflow on Windows natively (and because we are - I think - never going to support Airflow for production, we should not make it "official" way of running airflow other than for local development :

• WSL 2 Filesystem Performance :
  Accessing the host Windows filesystem incurs a performance penalty, it is therefore recommended to do development on the Linux filesystem. E.g. Run cd ~ and create a development folder in your Linux distro home and git pull the Airflow repo there.
• WSL 2 Docker mount errors:
  Another reason to use Linux filesystem, is that sometimes - depending on the length of your path, you might get strange errors when you try start Breeze, such as caused: mount through procfd: not a directory: unknown:. Therefore checking out Airflow in Windows-mounted Filesystem is strongly discouraged.
• WSL 2 Docker volume remount errors:
  If you're experiencing errors such as ERROR: for docker-compose_airflow_run Cannot create container for service airflow: not a directory when starting Breeze after the first time or an error like docker: Error response from daemon: not a directory. See 'docker run --help'. when running the pre-commit tests, you may need to consider installing Docker directly in WSL 2 instead of using Docker Desktop for Windows.
• WSL 2 Memory Usage :
  WSL 2 can consume a lot of memory under the process name "Vmmem". To reclaim the memory after development you can:
  On the Linux distro clear cached memory: sudo sysctl -w vm.drop_caches=3
  If no longer using Docker you can quit Docker Desktop (right click system try icon and select "Quit Docker Desktop")
  If no longer using WSL you can shut it down on the Windows Host with the following command: wsl --shutdown
• Developing in WSL 2:
  You can use all the standard Linux command line utilities to develop on WSL 2. Further VS Code supports developing in Windows but remotely executing in WSL. If VS Code is installed on the Windows host system then in the WSL Linux Distro you can run code . in the root directory of you Airflow repo to launch VS Code.

[ErandH]

Hi I am new to Airflow. @potiuk , any updates on when Airflow will be supported natively on Windows? You mention that running airflow through wsl2 is not appropriate for production. What if i am running Airflow on a seperate linux server (scheduler and webserver), postgres on another server, and i use CeleryExecutor to connect to workers on remote windows machines that run in ubuntu wsl2? So in this case the scheduler would not run on wsl2 but the workers would. The reason for this is that i need to run tasks on my windows environments ; with this setup i can run cmd.exe easily from tasks in ubuntu. Any better alternatives to this that avoid having to use wsl2 for workers? My constraint is i need to access my windows environment. Any advice would be greatly appreciated.

[potiuk]

No other advice at the moment. There are no "plans" to support Airflow for Windows natively. If someone would like to take on that task - it is up for grabs.

WSL2 is not recommended for production - in the sense that if you have problems with it, and they are coming from the setup you have - we have no CI and tests run on such environment and your problems might be difficult to reproduce and it will be hard for you to get help. But if you are willinng to try it out and report back your experiences from such a setup, you are of course free to do it if you have good reasons for it, but them rather expect you will be solving some problems on your own.

There is a WinRM support for Windows as well if you wish to use it https://airflow.apache.org/docs/apache-airflow-providers-microsoft-winrm/stable/index.html - and there are some people using it, maybe that will be enough for you.

[ErandH]

Thank you so much for the reply. I will probably try it out and report back my experience with using it. Will look into WinRM as well, thanks for the suggestion.