Rewrite Breeze in Python

[potiuk]

This is follow up and discussion space for the discussion started in https://lists.apache.org/thread.html/ra7e0629922436607de72fa3ea124aa4336df033163f4af0ebcee7b98%40%3Cdev.airflow.apache.org%3E

While I was opposing this while we were focusing on Airflow 2.0, and there are some good reasons why initially I started Breeze in Bash, I think with the current state of Airflow 2.0 betas, with Airflow 2.0 fully based on Python 3.6 and with some "stability" and "good set of features" we have in Breeze and a good level of modularisation we achieved - it's the right time to think about a rewrite.

I did not raise this subject to add a distraction on top of what is already a lot of work for 2.0, but I think having Breeze rewritten in Python could be the "one more thing" that we could do - as a community to make 2.0 experience even better, and one that can make the community even closer.

I was thinking that Breeze is perfect to be split into separate smaller pieces, describe some assumptions that we will have for its use, and turn it into a true community effort where a lot of people will contribute and where we will be able to simplify some of the stuff, and - most importantly - make more people from the community know about how our CI and development environment works and be able to solve any problems there.

Breeze (and underlying bash libraries) are crucial, to get our CI working and I am mostly the single point of contact (and failure!) when it comes to that - I would love to not be one :) and I think with most of the core committers busy with 2.0, this is also an opportunity for more of the contributors to take their part in it (and eventually earn their rank to become committers!). For the core committers, this is an extra opportunity to learn how the system works, influence its design, and possibly simplify some parts of it - even if they will be mostly focused on 2.0.

I would like to do it well - write some assumptions in a design doc, plan the work and split it into separate issues, and lead the effort - but I would love if most of the work is done by others, who would then become familiar with the whole of it.

---

THIS IS NOT YET A DESIGN PHASE. I am merely trying to see whether there is an interest in taking part in it and helping in the project. Design discussions will follow, and proper project plan will be developed for that, I am looking for people who would like to help with that and become part of the "Breeze 2" team to design and develop it.

Anyone interested - please comment here.

---

[potiuk]

And just another comment here. I would love to have a good mixture of people participating:

• core commiters
• "fresh" commiters
• frequent contributors
• casual contributors
• users who would like to contribute

[davido912]

Raising my hands up here. Would be interested to hear and help. ✋

[phraniiac]

Hi! I would like to help with this task. ✋

[Zedmor]

I would like that! sign me up

[caddac]

I'm interested.

[yangrong688]

I would like to contribute.

[mik-laj]

I'm interested too.

[potiuk]

Cool! I think the critical mass has been reached :), I am writing a short design doc to start with :)

[judoole]

I would like to contribute if I have something to offer. I do most of the grunt work of keeping Composer and local Airflow in our company running, so I hope finally my experience will come to good open source use.

[TobKed]

I'm interested as well.

[turbaszek]

[potiuk]

OK. Seems like we are close to get 9 companions very soon !

[potiuk]

Wait It is already 9!

[potiuk]

I will prepare the initial doc over weekend :)

[leahecole]

Happy to be a tester and to help with communication + devrel-y docs things once we're closer to rollout!

[potiuk]

cool! I am pushing the providers first for beta 3 and will send the docs shortly after :)

[potiuk]

still pushing :) (but getting closer) stay tuned.

[brighton1101]

Late, but interested as well.

[potiuk]

Cool. Swamped with 2.0 .. but ... it's coming !

[baolsen]

Okay sure, I'll bite :)

[coopergillan]

Also very late, but I am pretty curious about this one 🙂

[kishvanchee]

Late, but interested :) I hope I didn't miss out on this yet

[potiuk]

Naaa 2.0 took all the time I had :). But RC1 is out and we are cleaning up so ... soon :)

[RNHTTR]

I’m interested :)

[mrbazzan]

I am interested.

[potiuk]

FINALLY FINALLY IT'S THERE.

Yeah. Finally the time had come to start work on the rewrite:

First, founding PR is here: #19867 - those who are interested will find there some ADR (Architecture Decision Records) which we will keep throughout the whole project to capture all important decisions we've made and once we merge it, more stuff will come.

We have interns working on it - and help of anyone here to comment, test any anything (or even do some stuff !) is MORE than welcome!

[potiuk]

Hey everyone. We are progressing with Breeze2 implementation with Edith and Bowrna and I think we are getting close to start thinking about switching for some users, however I just proposed (based on earlier discussions and proposals with @uranusjr ) a change in the approach of breeze command line that I wanted to get opinion of the current users.

You might lack some context, but I am happy to answer questions here and discuss it here as well or in slack - I described the change proposla in #20921 and I would love some comments of everyone who uses Breeze currently.

Also we are quickly approaching the time when all important decisions are made, some simplifications (#20664 especially) are made in the current Breeze to make it "far less complex" and some basic "working" scaffolding is usable, and I think we might to reach out to some of the people who were happy to help to help us with some tasks.

I think it would be great to engage more people to help @Bowrna and @edithturn to complete their tasks

[potiuk]

Just react with hands up if you want to help and I will post here some more details on how you can join the effort in couple of days.

[edithturn]

✋

[bhavaniravi]

✋

[potiuk]

Hey @bhavaniravi -> if you want to help you can take a look at https://github.com/apache/airflow/projects/13 - there some issues in progress but there are some "ready to pick" and you can take a look at the history of those that we've already implemented and possibly pick one of the tasks there :). Happy to explain more if you might have questions.

[potiuk]

Hello here!

Feel the (new) breeze - written in Python.

I have just merged a PR (#22880) that marks a completion of the most important stage of the work that @Bowrna and @edithturn (our interns from Outreachy) have completed - with myself, @eladkal and @xurror mentoring them. Big claps to them!! That was quite an internship :).

It also marks the beginning of a new era of a Bash-less (or Bash-low) dev environment for Airflow. If you’ve used Breeze before, you will get mostly the same things you are used to - but it is written in Python rather than in Bash.

There are many reasons we moved, I will not get into details (see below if you want to know all the context). Vast majority of “user” functionality is ported to new Breeze (with small differences but you will likely find it easy to switch). The UI is soooo much nicer (see screenshots in https://github.com/apache/airflow/blob/main/BREEZE.rst). Also we kept the ./breeze-legacy script and for now it continues to work the same as old Breeze - so you can use it for a while if things do not work (./breeze-legacy and breeze work in parallel).

We will gradually remove Bash code as we get confirmation that Python works for you.

There are some prerequisites that you need to follow (install pipx and setup breeze + autocomplete). But there are generally less than the old breeze, really.

• https://github.com/apache/airflow/blob/main/BREEZE.rst#the-pipx-tool
• https://github.com/apache/airflow/blob/main/BREEZE.rst#installation

Important - after you install the new breeze completely, you no longer need to run ./breeze. Just breeze is enough. The ./breeze script should continue to work for a while, but we will eventually remove it too.
Please let us know at the #airflow-breeze channel in slack if you have any problems. Any questions/comments are welcome. There will likely be some teething problems.... But remember ./breeze-legacy is there for you just in case something does not work (and it should work seamlessly alongside the new breeze).

Also there are quite a few tasks left in the project https://github.com/apache/airflow/projects/13 that we will be completing - feel free to comment and take any of the tasks left. We will be happy to help to jump-start it.
For those who are interested - the reasoning and all design decisions are kept here: https://github.com/apache/airflow/tree/main/dev/breeze/doc/adr in the form of Architecture Decision Records.
Once again - great job @Bowrna and @edithturn !

[potiuk]

Some status with so-far bash remoal for Breeze:

Before:

python:      288754 (96.35%)
sh:           10950 (3.65%)

now:

python:      348937 (97.73%)
sh:            8095 (2.27%)

We have not yet removed some of the legacy code so the numbers are a little misleading, but I hope soon we will be able to remove all of them, we are getting closer to it

https://github.com/apache/airflow/projects/13

[potiuk]

And today:

python:      371161 (97.77%)
sh:            5128 (1.35%)
javascript:      3204 (0.84%)
xml:            132 (0.03%)

[potiuk]

After those two PRs: #26612 and #26828 we can officially close this task with Shell code getting to less than

We go from here:

sh:           10945 (3.51%)

to here (less than 1%):

sh:            3536 (0.92%)

@Bowrna @edithturn @eladkal

[potiuk]

Done. Closed. Complete. @Bowrna @edithturn

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

[edithturn]

Wow Jarek!
Congratulations 👏🏽👏🏽

[Bowrna]

Congrats Jarek :) I am happy to be part of this