Switch tests in ci to use Python Breeze #26612

potiuk · 2022-09-22T22:27:48Z

This is the last big change in "Rewrite Breeze to Python".

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

potiuk · 2022-09-22T22:28:55Z

cc: @Bowrna - not tested yet, but roughly complete.

potiuk · 2022-09-23T01:12:41Z

Last "BIG" change to get breeze replaced completely wiht Python !

scripts/ci/docker-compose/base-ports.yml

dev/breeze/src/airflow_breeze/params/shell_params.py

potiuk · 2022-09-23T08:12:20Z

Two example screenshot of showing progress of parallel tests run (much nicer and more informative now):

I want to run a few more tests with public runners and such, but it should be ready to review/merge

potiuk · 2022-09-23T08:13:29Z

Did I mention -800 lines of bash code ?

potiuk · 2022-10-01T00:56:41Z

Looking for approval :)

potiuk · 2022-10-01T00:57:30Z

Very thoroughly tested for Public/Self-hosted runners.

potiuk · 2022-10-01T00:59:32Z

CI.rst


-The images produced during the ``Build Images`` workflow of CI jobs are stored in the
-`GitHub Container Registry <https://github.com/orgs/apache/packages?repo_name=airflow>`_
+Debugging CI Jobs in Github Actions


Added chapter on how to debug CI jobs - all the possibilities of how you can test different parts are now nicely separated out.

potiuk · 2022-10-01T01:00:40Z

dev/breeze/src/airflow_breeze/commands/ci_commands.py

@@ -92,6 +92,11 @@ def free_space(verbose: bool, dry_run: bool, answer: str):
        )
        run_command(["df", "-h"], verbose=verbose, dry_run=dry_run)
        run_command(["docker", "logout", "ghcr.io"], verbose=verbose, dry_run=dry_run, check=False)
+        run_command(
+            ["sudo", "rm", "-f", os.fspath(Path.home() / MSSQL_TMP_DIR_NAME)],
+            verbose=verbose,


We cleanup MSQL local folder that is used on Self-Hosted runner here.

potiuk · 2022-10-01T01:01:18Z

dev/breeze/src/airflow_breeze/commands/ci_commands.py

@@ -111,7 +116,7 @@ def resource_check(verbose: bool, dry_run: bool):
    HOME_DIR / ".azure",
    HOME_DIR / ".config/gcloud",
    HOME_DIR / ".docker",
-    AIRFLOW_SOURCES_ROOT,
+    HOME_DIR / MSSQL_TMP_DIR_NAME,


AIRFLOW_SOURCES_ROOT was duplicated.

potiuk · 2022-10-01T01:02:56Z

dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py

@@ -147,8 +148,7 @@ def kubernetes_group():
 K8S_CONFIGURE_CLUSTER_PROGRESS_REGEXP = r'.*airflow-python-[0-9.]+-v[0-9.].*'
 K8S_DEPLOY_PROGRESS_REGEXP = r'.*airflow-python-[0-9.]+-v[0-9.].*'
 K8S_TEST_PROGRESS_REGEXP = r'.*airflow-python-[0-9.]+-v[0-9.].*|^kubernetes_tests/.*'
-PERCENT_K8S_TEST_PROGRESS_REGEXP = r'^kubernetes_tests/.*\[[ \d*%]*\].*'
-K8S_SKIP_TRUNCATION_REGEXP = r'^kubernetes_tests/.*'


I found better ways to truncate the output (I simply check the length of the output after stripping or ANSI Colours (standard textwrap does not truncate ANSI colors and treats ANSI colors as characters)

This is the last big change in "Rewrite Breeze to Python". Fixes: apache#23538

potiuk · 2022-10-03T12:45:41Z

Re-running it one more time without resource debugging to see that the output is right.

ashb · 2022-10-03T13:36:15Z

dev/breeze/src/airflow_breeze/params/shell_params.py

+
+    @property
+    def mssql_data_volume(self) -> str:
+        docker_filesystem = get_filesystem_type("/var/lib/docker")


This is run from the host isn't it? If so this call might not get the right info in cases of Docker Desktop etc. Does that matter?

It's actually only really used when you have tmpfs in '/var/lib/docker' (which is the case of self-hosted runners). We could likely do a better job in finding the right folder, but I think it is 'good enough' to handle the nasal on tmpfs case

ashb

LGTM - just one small question about mssql tmpfs volume.

potiuk · 2022-10-03T14:47:23Z

LGTM - just one small question about mssql tmpfs volume.

Thanks! It looks big but it is not as huge as the numbers on the PR says :).

This is the last big change in "Rewrite Breeze to Python". Fixes: #23538 (cherry picked from commit 2921796)

boring-cyborg bot added the area:dev-tools label Sep 22, 2022

potiuk force-pushed the switch-to-breeze-python-tests branch 14 times, most recently from 3a0b200 to 65bb711 Compare September 23, 2022 01:11

potiuk marked this pull request as ready for review September 23, 2022 01:12

potiuk requested review from ashb, mik-laj, jedcunningham and kaxil as code owners September 23, 2022 01:12

potiuk requested a review from eladkal September 23, 2022 08:03

potiuk commented Sep 23, 2022

View reviewed changes

scripts/ci/docker-compose/base-ports.yml Show resolved Hide resolved

potiuk commented Sep 23, 2022

View reviewed changes

dev/breeze/src/airflow_breeze/params/shell_params.py Show resolved Hide resolved

potiuk added the use public runners Makes sure that Public runners are used even if commiters creates the PR (useful for testing) label Sep 23, 2022

potiuk force-pushed the switch-to-breeze-python-tests branch 2 times, most recently from d75eb46 to e7ec1db Compare September 23, 2022 08:15

potiuk requested a review from uranusjr September 30, 2022 04:05

potiuk added use public runners Makes sure that Public runners are used even if commiters creates the PR (useful for testing) debug ci resources Set it on PR if you want to debug resource usage for it labels Sep 30, 2022

potiuk force-pushed the switch-to-breeze-python-tests branch 2 times, most recently from ce14db5 to a0ff5e2 Compare October 1, 2022 00:55

potiuk removed the use public runners Makes sure that Public runners are used even if commiters creates the PR (useful for testing) label Oct 1, 2022

potiuk force-pushed the switch-to-breeze-python-tests branch from a0ff5e2 to 3e2e02b Compare October 1, 2022 00:56

potiuk commented Oct 1, 2022

View reviewed changes

potiuk force-pushed the switch-to-breeze-python-tests branch 2 times, most recently from a0a162b to a7c9f8d Compare October 2, 2022 13:57

This was referenced Oct 2, 2022

Move downgrade/upgrade tests to run new Python breeze #26828

Merged

Rewrite Breeze in Python #12282

Closed

potiuk removed the debug ci resources Set it on PR if you want to debug resource usage for it label Oct 3, 2022

Switch tests in ci to use Python Breeze

9bf2d71

This is the last big change in "Rewrite Breeze to Python". Fixes: apache#23538

potiuk force-pushed the switch-to-breeze-python-tests branch from a7c9f8d to 9bf2d71 Compare October 3, 2022 12:45

ashb reviewed Oct 3, 2022

View reviewed changes

ashb approved these changes Oct 3, 2022

View reviewed changes

potiuk merged commit 2921796 into apache:main Oct 3, 2022

potiuk deleted the switch-to-breeze-python-tests branch October 3, 2022 14:47

ephraimbuddy pushed a commit that referenced this pull request Nov 10, 2022

Switch tests in ci to use Python Breeze (#26612)

a2309d9

This is the last big change in "Rewrite Breeze to Python". Fixes: #23538 (cherry picked from commit 2921796)

ephraimbuddy pushed a commit that referenced this pull request Nov 10, 2022

Switch tests in ci to use Python Breeze (#26612)

4e454db

This is the last big change in "Rewrite Breeze to Python". Fixes: #23538 (cherry picked from commit 2921796)

ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Nov 10, 2022

ephraimbuddy added this to the Airflow 2.4.3 milestone Nov 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch tests in ci to use Python Breeze #26612

Switch tests in ci to use Python Breeze #26612

potiuk commented Sep 22, 2022

potiuk commented Sep 22, 2022

potiuk commented Sep 23, 2022

potiuk commented Sep 23, 2022

potiuk commented Sep 23, 2022

potiuk commented Oct 1, 2022

potiuk commented Oct 1, 2022

potiuk Oct 1, 2022

potiuk Oct 1, 2022

potiuk Oct 1, 2022

potiuk Oct 1, 2022

potiuk commented Oct 3, 2022

ashb Oct 3, 2022

potiuk Oct 3, 2022

ashb left a comment

potiuk commented Oct 3, 2022

Switch tests in ci to use Python Breeze #26612

Switch tests in ci to use Python Breeze #26612

Conversation

potiuk commented Sep 22, 2022

potiuk commented Sep 22, 2022

potiuk commented Sep 23, 2022

potiuk commented Sep 23, 2022

potiuk commented Sep 23, 2022

potiuk commented Oct 1, 2022

potiuk commented Oct 1, 2022

potiuk Oct 1, 2022

Choose a reason for hiding this comment

potiuk Oct 1, 2022

Choose a reason for hiding this comment

potiuk Oct 1, 2022

Choose a reason for hiding this comment

potiuk Oct 1, 2022

Choose a reason for hiding this comment

potiuk commented Oct 3, 2022

ashb Oct 3, 2022

Choose a reason for hiding this comment

potiuk Oct 3, 2022

Choose a reason for hiding this comment

ashb left a comment

Choose a reason for hiding this comment

potiuk commented Oct 3, 2022