-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial commit for new Breeze project #19867
Conversation
a25afa1
to
47d5087
Compare
b1de041
to
ff2944b
Compare
It's not exactly what you proposed @uranusjr , but I think with this setup we will be able to modify (in the future) the entrypoint script, to be able to create venv and run equivalent of |
I am not sure if we want to release the whole I am not sure yet how to approach the "small bootstrap" package - I like how the install would work now with editable installed Breeze in local virtualenv (where I think we want to have the whole of Breeze installed rather than the "bootstrap" script only). Maybe when we install it as "editable" locally it should install the whole of breeze (all packages) but then when we distribute it in PyPI we only install the "bootstrap" script there. I guess that would be possible? Happy to brainstorm on this one @uranusjr |
Using Flit + PEP 621 would eliminate much of the guess work how to get things right 😉 |
ff2944b
to
444f8aa
Compare
I just added the I think that might be nice and portable and "self-running" without extra deps like flit/PEP 621. For me this is really the case of having a runnable "devel" environment. |
Sounds simple enough:
|
(and we'd need to add some protection against old python version) |
Note that You mention pipx earlier, and I feel it might be a better setup for contributors? Just |
This actually gives me a thought, since you feel it’s important to make the virtual environment (for Breeze) invisibal to the user. Maybe we should build Breeze entirely upon an existing task runner solution, instead of rolling our own infrastructure. Nox has pretty much have virtual environment management and task invocation figured out—we can make Breeze a wrapper to |
I have also added current stats of the languages used - I beleive sh should be less than 300 lines of code (< 0.1% when we finish).
I am pretty happy with what we have now as the first PR:
I don't think we need to have another Breeze package in PyPI with this setup, it's simply not needed as the I already nicely integrated it as module in my IntelliJ without extra "magic" using the bootstrapped virtualenv - should nicely work also on Windows. @edithturn @Bowrna - maybe you could check it out and check it on your windows machines to see how easy it is to bootstrap (also maybe you could try to generate ./Breeze2.exe using |
46c639a
to
d919fc3
Compare
4cd2cf9
to
6ca6f02
Compare
It includes: * proposal for initial ADRs (Architecture Decision records) where we will keep decision records about both - Breeze2 and CI * scaffolding for the new breeze command including command line, pre-commit checks, automated tests in CI and requirements
6ca6f02
to
d2c23ed
Compare
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Any more thought @uranusjr ? |
I have more thoughts on environment provision (still don’t like automatically setting up virtual environments much, it can go wrong way to often), but let’s merge this first since those don’t really affect the “actual Breeze” functionalities. |
Yep. happy to improve it later on :) |
BTW. Following @mik-laj's advice I reserved PyPI package: the https://pypi.org/project/apache-airflow-breeze/ as well. Maybe we can figure out other /parallell workflows with installig the package from PyPI. |
One more comment: It actually might well be that when |
import os | ||
import sys | ||
|
||
# Python <3.4 does not have pathlib |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment doesn't line up with code :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh yeah. That was a mentl shortcut. The comment should be:
We do the check here, before importing pathlib becasue Python <3.4 does not have pathlib and import would fail with cryptic "missing pathlib module" error.
- [Decision](#decision) | ||
- [Consequences](#consequences) | ||
|
||
# 1. Record architecture decisions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep. I thin you were the one who motivated me to do it :)
It includes: * proposal for initial ADRs (Architecture Decision records) where we will keep decision records about both - Breeze2 and CI * scaffolding for the new breeze command including command line, pre-commit checks, automated tests in CI and requirements
It includes:
where we will keep decision records about both - Breeze2 and CI
pre-commit checks, automated tests in CI and requirements
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.