Skip to content

Commit

Permalink
Replace old Breeze with Python based implementation (#22880)
Browse files Browse the repository at this point in the history
Over the last few months together with Outreachy interns
we rewrote the most important functionality of the old Bash-based
Breeze with Python Based implementation.

We approached it in systematic way with capturing all our decisions
in the ADR format (dev/breeze/docs) and implementing the parts
that are used on a daily basis by the users. Breeze2 as it was
called is ready for Prime-Time with the users so we are swapping
out the old breeze wiht the new one.

The old `breeze` has been moved to `breeze-legacy` and we will
gradually parts of it that are already migrated to Python and proven
while continue rewriting the parts that are missing (mostly the
maintainer tools) and replacing the remaining CI shell scripts with
the new `breeze` commands.

We also need to make sure that there is no accidental top-level
import for extra packages added in the future, because people
who installed breeze before will not have it - so we have
a pre-commit that checks if breeze.py can be parsed and
--help executed with just rich and click installed.

This PR:

* moves `breeze` to `breeze-legacy`
* moves `Breeze2` to `breeze`
* updates documentation and screenshots where applicable
* explains old vs. new breeze in documentation
* adds protection so that no accidental top-level import is
  added to breeze.py and files imported from there

Fixes: #22827
  • Loading branch information
potiuk authored Apr 10, 2022
1 parent 7be57eb commit 1007828
Show file tree
Hide file tree
Showing 63 changed files with 6,718 additions and 6,059 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build-images.yml
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ jobs:
- name: "Free space"
run: airflow-freespace
- name: "Build CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
run: Breeze2 build-image
run: breeze build-image
env:
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
GITHUB_REGISTRY_PUSH_IMAGE_TAG: ${{ env.IMAGE_TAG_FOR_THE_BUILD }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -329,7 +329,7 @@ jobs:
run: airflow-freespace
if: needs.build-info.outputs.inWorkflowBuild == 'true'
- name: "Build CI image ${{ matrix.python-version }}:${{ env.IMAGE_TAG_FOR_THE_BUILD }}"
run: Breeze2 build-image
run: breeze build-image
if: needs.build-info.outputs.inWorkflowBuild == 'true'
env:
GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
Expand Down Expand Up @@ -417,7 +417,7 @@ jobs:

run-new-breeze-tests:
timeout-minutes: 10
name: Breeze2 tests
name: breeze tests
runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }}
needs: [build-info]
steps:
Expand Down
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -222,5 +222,5 @@ pip-wheel-metadata
# Generated UI licenses
licenses/LICENSES-ui.txt

# Packaged Breeze2 on Windows
/Breeze2.exe
# Packaged breeze on Windows
/breeze.exe
2 changes: 1 addition & 1 deletion .gitpod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
# Docs: https://www.gitpod.io/docs/config-gitpod-file/

tasks:
- init: ./breeze -y
- init: ./breeze-legacy -y
- name: Install pre-commit
openMode: split-right
command: |
Expand Down
23 changes: 16 additions & 7 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ repos:
- id: insert-license
name: Add license for all shell files
exclude: ^\.github/.*$|^airflow/_vendor/|^dev/breeze/autocomplete/.*$
files: ^breeze$|^breeze-complete$|\.bash$|\.sh$
files: ^breeze-legacy$|^breeze-complete$|\.bash$|\.sh$
args:
- --comment-style
- "|#|"
Expand Down Expand Up @@ -266,7 +266,7 @@ repos:
The word(s) should be in lowercase." && exec codespell "$@"' --
language: python
types: [text]
exclude: ^airflow/_vendor/|^CHANGELOG\.txt$|^airflow/www/static/css/material-icons\.css$
exclude: ^airflow/_vendor/|^CHANGELOG\.txt$|^airflow/www/static/css/material-icons\.css$|^images/.*$
args:
- --ignore-words=docs/spelling_wordlist.txt
- --skip=docs/*/commits.rst,airflow/providers/*/*.rst,*.lock,INTHEWILD.md,*.min.js,docs/apache-airflow/pipeline_example.csv
Expand Down Expand Up @@ -320,11 +320,12 @@ repos:
files: ^airflow/providers/.*\.py$
additional_dependencies: ['rich']
- id: update-breeze-file
name: Update output of breeze command in BREEZE.rst
entry: ./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.sh
language: system
files: ^BREEZE\.rst$|^breeze$|^breeze-complete$|^Dockerfile$
name: Update output of breeze commands in BREEZE.rst
entry: ./scripts/ci/pre_commit/pre_commit_breeze_cmd_line.py
language: python
files: ^BREEZE\.rst$|^dev/breeze/.*$
pass_filenames: false
additional_dependencies: ['rich']
- id: update-local-yml-file
name: Update mounts in the local yml file
entry: ./scripts/ci/pre_commit/pre_commit_local_yml_mounts.sh
Expand Down Expand Up @@ -525,7 +526,7 @@ repos:
name: Check Shell scripts syntax correctness
language: docker_image
entry: koalaman/shellcheck:v0.7.2 -x -a
files: ^breeze$|^breeze-complete$|\.sh$|^hooks/build$|^hooks/push$|\.bash$
files: ^breeze-legacy$|^breeze-complete$|\.sh$|^hooks/build$|^hooks/push$|\.bash$
exclude: ^dev/breeze/autocomplete/.*$
- id: stylelint
name: stylelint
Expand Down Expand Up @@ -591,6 +592,14 @@ repos:
files: ^dev/breeze/setup.*$|^dev/breeze/pyproject.toml$|^dev/breeze/README.md$
pass_filenames: false
require_serial: true
- id: limit-breeze-dependencies
name: Breeze should have small number of top-level dependencies
language: python
entry: ./scripts/tools/check_if_limited_dependencies.py
files: ^dev/breeze/.*$
pass_filenames: false
require_serial: true
additional_dependencies: ['click', 'rich']
- id: check-system-tests
name: Check if system tests have required segments of code
entry: ./scripts/ci/pre_commit/pre_commit_check_system_tests.py
Expand Down
Loading

0 comments on commit 1007828

Please sign in to comment.