feat: Run UATs from a remote commit

README.md

@@ -25,6 +25,7 @@ very least) of the following pieces:
     * MicroK8s
     * Charmed Kubernetes
     * EKS cluster
+    * AKS cluster <!-- codespell-ignore -->
 * **Charmed Kubeflow** deployed on top of it
 * **MLFlow (optional)** deployed alongside Kubeflow
 
@@ -77,26 +78,41 @@ In order to run the tests using the `driver`:
    source venv/bin/activate
    pip install tox
    ```
-* Run the UATs:
 
-   ```bash
-   # assumes an existing `kubeflow` Juju model
-   tox -e uats
-   ```
+Then in order to run UATs, there are two options:
 
-  You can also run a subset of the provided tests using the `--filter` option and passing a filter
-  that follows the same syntax as the pytest `-k` option, e.g.
+#### Run tests from a remote commit
+In this case, tests are fetched from a remote commit of `charmed-kubeflow-uats` repository. In order to define the commit, tests use the hash of the `HEAD`, where the repository is checked out locally. This means that when you want to run tests from a specific branch, you need to check out to that branch and then run the tests. Note that if the locally checked out commit is not pushed to the remote repository, then tests will fail.
 
-   ```bash
-   # run all tests containing 'kfp' or 'katib' in their name
-   tox -e uats -- --filter "kfp or katib"
-   # run any test that doesn't contain 'kserve' in its name
-   tox -e uats -- --filter "not kserve"
-   ```
+```bash
+# assumes an existing `kubeflow` Juju model
+tox -e uats-remote
+```
+
+#### Run tests from local copy
+
+This one works only when running the tests from the same node where the tests job is deployed (e.g. running from the same machine where the Microk8s cluster lives). In this case, the tests job instantiates a volume that is [mounted to the local directory of the repository where tests reside](https://github.com/canonical/charmed-kubeflow-uats/blob/ee0fa08931b11f40e97dbe3e340c413cf466a084/assets/test-job.yaml.j2#L34-L36). If unsure about your setup, use the `-remote` option.
+
+```bash
+# assumes an existing `kubeflow` Juju model
+tox -e uats-local
+```
+
+#### Run a subset of UATs
+
+You can also run a subset of the provided tests using the `--filter` option and passing a filter
+that follows the same syntax as the pytest `-k` option, e.g.
+
+```bash
+# run any test that doesn't contain 'kserve' in its name
+tox -e uats-remote -- --filter "not kserve"
+# run all tests containing 'kfp' or 'katib' in their name
+tox -e uats-local -- --filter "kfp or katib"
+```
 
-  This simulates the behaviour of running `pytest -k "some filter"` directly on the test suite.
-  You can read more about the options provided by Pytest in the corresponding section of the
-  [documentation](https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags).
+This simulates the behaviour of running `pytest -k "some filter"` directly on the test suite.
+You can read more about the options provided by Pytest in the corresponding section of the
+[documentation](https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags).
 
 #### Run Kubeflow UATs
 
@@ -105,7 +121,10 @@ dedicated `kubeflow` tox test environment:
 
 ```bash
 # assumes an existing `kubeflow` Juju model
-tox -e kubeflow
+# run tests from the checked out commit after fetching them remotely
+tox -e kubeflow-remote
+# run tests from the local copy of the repo
+tox -e kubeflow-local
 ```
 
 #### Developer Notes
@@ -117,7 +136,7 @@ a Kubernetes Job to run the tests. More specifically, the `driver` executes the
 1. Create a Kubeflow Profile (i.e. `test-kubeflow`) to run the tests in
 2. Submit a Kubernetes Job (i.e. `test-kubeflow`) that runs `tests`
    The Job performs the following:
-   * Mount the local `tests` directory to a Pod that uses `jupyter-scipy` as the container image
+   * If a `-local` tox environment is run, then it mounts the local `tests` directory to a Pod that uses `jupyter-scipy` as the container image. Else (in `-remote` tox environments), it creates an emptyDir volume which it syncs to the current commit that the repo is checked out locally, using a [git-sync](https://github.com/kubernetes/git-sync/) `initContainer`.
    * Install python dependencies specified in the [requirements.txt](tests/requirements.txt)
    * Run the test suite by executing `pytest`
 3. Wait until the Job completes (regardless of the outcome)

assets/test-job.yaml.j2

@@ -11,16 +11,28 @@ spec:
           access-ml-pipeline: "true"
           mlflow-server-minio: "true"
     spec:
+      {% if not tests_local_run %}
+      # securityContext is needed in order for test files to be writeable 
+      # since the tests save the notebooks. Setting it enables:
+      # * The test-volume to be group-owned by this GID.
+      # * The GID to be added to each container.
+      securityContext:
+        fsGroup: 101
+      {% endif %}
       serviceAccountName: default-editor
       containers:
         - name: {{ job_name }}
-          image: {{ test_image }}
+          image: {{ tests_image }}
           command:
             - bash
             - -c
           args:
             - |
+              {% if tests_local_run %}
               cd /tests;
+              {% else %}
+              cd /tests/charmed-kubeflow-uats/tests;
+              {% endif %}
               pip install -r requirements.txt >/dev/null;
               {{ pytest_cmd }};
               # Kill Istio Sidecar after workload completes to have the Job status properly updated
@@ -30,8 +42,28 @@ spec:
           volumeMounts:
             - name: test-volume
               mountPath: /tests
+      {% if not tests_local_run %}
+      initContainers:
+        - name: git-sync
+          # This container pulls git data and publishes it into volume
+          # "test-volume".
+          image: registry.k8s.io/git-sync/git-sync:v4.0.0
+          args:
+            - --repo=https://github.com/canonical/charmed-kubeflow-uats
+            - --ref={{ tests_remote_commit }}
+            - --root=/tests
+            - --group-write
+            - --one-time
+          volumeMounts:
+            - name: test-volume
+              mountPath: /tests
+      {% endif %}
       volumes:
         - name: test-volume
+          {% if tests_local_run %}
           hostPath:
-            path: {{ test_dir }}
+            path: {{ tests_local_dir }}
+          {% else %}
+          emptyDir: {}
+          {% endif %}
       restartPolicy: Never

driver/test_kubeflow_workloads.py

@@ -3,6 +3,7 @@
 
 import logging
 import os
+import subprocess
 from pathlib import Path
 
 import pytest
@@ -16,7 +17,9 @@
 JOB_TEMPLATE_FILE = ASSETS_DIR / "test-job.yaml.j2"
 PROFILE_TEMPLATE_FILE = ASSETS_DIR / "test-profile.yaml.j2"
 
-TESTS_DIR = os.path.abspath(Path("tests"))
+TESTS_LOCAL_RUN = eval(os.environ.get("LOCAL"))
+TESTS_LOCAL_DIR = os.path.abspath(Path("tests"))
+
 TESTS_IMAGE = "kubeflownotebookswg/jupyter-scipy:v1.7.0"
 
 NAMESPACE = "test-kubeflow"
@@ -39,6 +42,13 @@ def pytest_filter(request):
     return f"-k '{filter}'" if filter else ""
 
 
+@pytest.fixture(scope="session")
+def tests_checked_out_commit(request):
+    """Retrieve active git commit."""
+    head = subprocess.check_output(["git", "rev-parse", "HEAD"])
+    return head.decode("UTF-8").rstrip()
+
+
 @pytest.fixture(scope="session")
 def pytest_cmd(pytest_filter):
     """Format the Pytest command."""
@@ -95,16 +105,18 @@ async def test_create_profile(lightkube_client, create_profile):
     assert_namespace_active(lightkube_client, NAMESPACE)
 
 
-def test_kubeflow_workloads(lightkube_client, pytest_cmd):
+def test_kubeflow_workloads(lightkube_client, pytest_cmd, tests_checked_out_commit):
     """Run a K8s Job to execute the notebook tests."""
     log.info(f"Starting Kubernetes Job {NAMESPACE}/{JOB_NAME} to run notebook tests...")
     resources = list(
         codecs.load_all_yaml(
             JOB_TEMPLATE_FILE.read_text(),
             context={
                 "job_name": JOB_NAME,
-                "test_dir": TESTS_DIR,
-                "test_image": TESTS_IMAGE,
+                "tests_local_run": TESTS_LOCAL_RUN,
+                "tests_local_dir": TESTS_LOCAL_DIR,
+                "tests_image": TESTS_IMAGE,
+                "tests_remote_commit": tests_checked_out_commit,
                 "pytest_cmd": pytest_cmd,
             },
         )
@@ -121,7 +133,7 @@ def test_kubeflow_workloads(lightkube_client, pytest_cmd):
         )
     finally:
         log.info("Fetching Job logs...")
-        fetch_job_logs(JOB_NAME, NAMESPACE)
+        fetch_job_logs(JOB_NAME, NAMESPACE, TESTS_LOCAL_RUN)
 
 
 def teardown_module():

driver/utils.py

@@ -79,8 +79,14 @@ def wait_for_job(
         raise ValueError(f"Unknown status {job.status} for Job {namespace}/{job_name}!")
 
 
-def fetch_job_logs(job_name, namespace):
+def fetch_job_logs(job_name, namespace, tests_local_run):
     """Fetch the logs produced by a Kubernetes Job."""
+    if not tests_local_run:
+        print("##### git-sync initContainer logs #####")
+        command = ["kubectl", "logs", "-n", namespace, f"job/{job_name}", "-c", "git-sync"]
+        subprocess.check_call(command)
+
+    print("##### test-kubeflow container logs #####")
     command = ["kubectl", "logs", "-n", namespace, f"job/{job_name}"]
     subprocess.check_call(command)
 

tox.ini

@@ -52,7 +52,8 @@ commands =
     codespell {toxinidir}/. --skip {toxinidir}/./.git --skip {toxinidir}/./.tox \
       --skip {toxinidir}/./build --skip {toxinidir}/./lib --skip {toxinidir}/./venv \
       --skip {toxinidir}/./.mypy_cache \
-      --skip {toxinidir}/./icon.svg --skip *.json.tmpl
+      --skip {toxinidir}/./icon.svg --skip *.json.tmpl \
+      --ignore-regex=".*codespell-ignore."
     # pflake8 wrapper supports config from pyproject.toml
     pflake8 {[vars]all_path}
     isort --check-only --diff {[vars]all_path}
@@ -61,23 +62,30 @@ deps =
     -r requirements-lint.txt
 description = Check code against coding style standards
 
-[testenv:kubeflow]
+[testenv:kubeflow-{local,remote}]
 commands =
     # run all tests apart from the ones that use MLFlow
     pytest -vv --tb native {[vars]driver_path} -s --filter "not mlflow" --model kubeflow {posargs}
+setenv =
+    local: LOCAL = True
+    remote: LOCAL = False
 deps =
     -r requirements.txt
 description = Run UATs for Kubeflow
 
-[testenv:uats]
+[testenv:uats-{local,remote}]
 # provide a filter when calling tox to (de)select test cases based on their names, e.g.
 # * run all tests containing 'kfp' or 'katib' in their name:
 #   $ tox -e uats -- --filter "kfp or katib"
 # * run any test that doesn't contain 'kserve' in its name:
 #   $ tox -e uats -- --filter "not kserve"
 # this simulates the behaviour of running 'pytest -k "<filter>"' directly on the test suite:
 # https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags
-commands = pytest -vv --tb native {[vars]driver_path} -s --model kubeflow {posargs}
+commands = 
+    pytest -vv --tb native {[vars]driver_path} -s --model kubeflow {posargs}
+setenv =
+    local: LOCAL = True
+    remote: LOCAL = False
 deps =
     -r requirements.txt
 description = Run UATs for Kubeflow and Integrations