Convert selective checks to Breeze Python (#24610)

Instead of bash-based, complex logic script to perform PR selective
checks we now integrated the whole logic into Breeze Python code.

It is now much simplified, when it comes to algorithm. We've
implemented simple rule-based decision tree. The rules describing
the decision tree are now are now much easier
to reason about and they correspond one-to-one with the rules
that are implemented in the code in rather straightforward way.

The code is much simpler and diagnostics of the selective checks
has also been vastly improved:

* The rule engine displays status of applying each rule and
  explains (with yellow warning message what decision was made
  and why. Informative messages are printed showing the resulting
  output

* List of files impacting the decision are also displayed

* The names of "ci file group" and "test type" were aligned

* Unit tests covering wide range of cases are added. Each test
  describes what is the case they demonstrate

* `breeze selective-checks` command that is used in CI can also
  be used locally by just providing commit-ish reference of the
  commit to check. This way you can very easily debug problems and
  fix them

Fixes: #19971

.github/workflows/build-images.yml

@@ -109,18 +109,20 @@ jobs:
         with:
           persist-credentials: false
           submodules: recursive
+      - name: "Setup python"
+        uses: actions/setup-python@v2
+        with:
+          # We do not have output from selective checks yet, so we need to hardcode python
+          python-version: 3.7
+          cache: 'pip'
+          cache-dependency-path: ./dev/breeze/setup*
+      - run: ./scripts/ci/install_breeze.sh
       - name: Selective checks
         id: selective-checks
         env:
-          PR_LABELS: ${{ steps.get-latest-pr-labels.outputs.pullRequestLabels }}
-        run: |
-          if [[ ${GITHUB_EVENT_NAME} == "pull_request_target" ]]; then
-            # Run selective checks
-            ./scripts/ci/selective_ci_checks.sh "${TARGET_COMMIT_SHA}"
-          else
-            # Run all checks
-            ./scripts/ci/selective_ci_checks.sh
-          fi
+          PR_LABELS: "$${{ steps.get-latest-pr-labels.outputs.pullRequestLabels }}"
+          COMMIT_REF: "${{ github.sha }}"
+        run: breeze selective-check
       - name: Compute dynamic outputs
         id: dynamic-outputs
         run: |

.github/workflows/ci.yml

@@ -192,18 +192,20 @@ jobs:
           fetch-depth: 2
           persist-credentials: false
         if: github.event_name  == 'pull_request'
+      - name: "Setup python"
+        uses: actions/setup-python@v2
+        with:
+          # We do not have output from selective checks yet, so we need to hardcode python
+          python-version: 3.7
+          cache: 'pip'
+          cache-dependency-path: ./dev/breeze/setup*
+      - run: ./scripts/ci/install_breeze.sh
       - name: Selective checks
         id: selective-checks
         env:
           PR_LABELS: "${{ steps.source-run-info.outputs.pullRequestLabels }}"
-        run: |
-          if [[ ${GITHUB_EVENT_NAME} == "pull_request" ]]; then
-            # Run selective checks
-            ./scripts/ci/selective_ci_checks.sh "${GITHUB_SHA}"
-          else
-            # Run all checks
-            ./scripts/ci/selective_ci_checks.sh
-          fi
+          COMMIT_REF: "${{ github.sha }}"
+        run: breeze selective-check
       # Avoid having to specify the runs-on logic every time. We use the custom
       # env var AIRFLOW_SELF_HOSTED_RUNNER set only on our runners, but never
       # on the public runners

.github/workflows/codeql-analysis.yml

@@ -43,19 +43,19 @@ jobs:
         with:
           fetch-depth: 2
           persist-credentials: false
+      - name: "Setup python"
+        uses: actions/setup-python@v2
+        with:
+          # We do not have output from selective checks yet, so we need to hardcode python
+          python-version: 3.7
+          cache: 'pip'
+          cache-dependency-path: ./dev/breeze/setup*
+      - run: ./scripts/ci/install_breeze.sh
       - name: Selective checks
         id: selective-checks
         env:
-          EVENT_NAME: ${{ github.event_name }}
-          TARGET_COMMIT_SHA: ${{ github.sha }}
-        run: |
-          if [[ ${EVENT_NAME} == "pull_request" ]]; then
-            # Run selective checks
-            ./scripts/ci/selective_ci_checks.sh "${TARGET_COMMIT_SHA}"
-          else
-            # Run all checks
-            ./scripts/ci/selective_ci_checks.sh
-          fi
+          COMMIT_REF: "${{ github.sha }}"
+        run: breeze selective-check
 
   analyze:
     name: Analyze

BREEZE.rst

@@ -565,13 +565,17 @@ Configuration and maintenance
 * Cleanup breeze with ``breeze cleanup`` command
 * Self-upgrade breeze with ``breeze self-upgrade`` command
 * Setup autocomplete for Breeze with ``breeze setup-autocomplete`` command
-* Checking available resources for docker with ``breeze resource-check`` command
-* Freeing space needed to run CI tests with ``breeze free-space`` command
-* Fixing ownership of files in your repository with ``breeze fix-ownership`` command
 * Print Breeze version with ``breeze version`` command
 * Outputs hash of commands defined by ``breeze`` with ``command-hash-export`` (useful to avoid needless
   regeneration of Breeze images)
 
+CI tasks
+--------
+* Freeing space needed to run CI tests with ``breeze free-space`` command
+* Fixing ownership of files in your repository with ``breeze fix-ownership`` command
+* Checking available resources for docker with ``breeze resource-check`` command
+* Deciding which tests should be run with ``breeze selective-check`` command
+
 Release tasks
 -------------
 
@@ -1295,8 +1299,8 @@ command but it is very similar to current ``breeze`` command):
       </a>
     </div>
 
-Resource check
-==============
+Running resource check
+----------------------
 
 Breeze requires certain resources to be available - disk, memory, CPU. When you enter Breeze's shell,
 the resources are checked and information if there is enough resources is displayed. However you can
@@ -1310,7 +1314,7 @@ Those are all available flags of ``resource-check`` command:
 
 
 Freeing the space
-=================
+-----------------
 
 When our CI runs a job, it needs all memory and disk it can have. We have a Breeze command that frees
 the memory and disk space used. You can also use it clear space locally but it performs a few operations
@@ -1323,8 +1327,26 @@ Those are all available flags of ``free-space`` command:
   :alt: Breeze free-space
 
 
+Selective check
+---------------
+
+When our CI runs a job, it needs to decide which tests to run, whether to build images and how much the test
+should be run on multiple combinations of Python, Kubernetes, Backend versions. In order to optimize time
+needed to run the CI Builds. You can also use the tool to test what tests will be run when you provide
+a specific commit that Breeze should run the tests on.
+
+More details about the algorithm used to pick the right tests can be
+found in `Selective Checks <SELECTIVE_CHECKS.md>`_.
+
+Those are all available flags of ``selective-check`` command:
+
+.. image:: ./images/breeze/output-selective-check.svg
+  :width: 100%
+  :alt: Breeze selective-check
+
+
 Tracking backtracking issues for CI builds
-==========================================
+------------------------------------------
 
 When our CI runs a job, we automatically upgrade our dependencies in the ``main`` build. However, this might
 lead to conflicts and ``pip`` backtracking for a long time (possibly forever) for dependency resolution.

dev/breeze/README.md

@@ -52,6 +52,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
 
 ---------------------------------------------------------------------------------------------------------
 
-Package config hash: a80a853b2c32c284a68ccd6d468804b892a69f14d2ad1886bdaa892755cf6262660e2b9fc582bcae27ae478910055267a76edea2df658196198a0365150e93e5
+Package config hash: 7279229e03b197f2bbd10ebb7b313f67bba3a704735d3688652efc5bdc1b3a60f2d1e0a144c89a2ecd11268b06888c5302a8774a8f392dc383bb940c99521db3
 
 ---------------------------------------------------------------------------------------------------------