feat: Run UATs from a remote commit

README.md

@@ -25,6 +25,7 @@ very least) of the following pieces:
     * MicroK8s
     * Charmed Kubernetes
     * EKS cluster
+    * AKS cluster <!-- codespell-ignore -->
 * **Charmed Kubeflow** deployed on top of it
 * **MLFlow (optional)** deployed alongside Kubeflow
 
@@ -77,26 +78,41 @@ In order to run the tests using the `driver`:
    source venv/bin/activate
    pip install tox
    ```
-* Run the UATs:
 
-   ```bash
-   # assumes an existing `kubeflow` Juju model
-   tox -e uats
-   ```
+Then in order to run UATs, there are two options:
 
-  You can also run a subset of the provided tests using the `--filter` option and passing a filter
-  that follows the same syntax as the pytest `-k` option, e.g.
+#### Run tests from a remote branch
+In this case, tests are fetched from a remote branch of `charmed-kubeflow-uats` repository. In order to select a branch, use the `--branch` option. If no branch is defined, then the `main` branch is used.
 
-   ```bash
-   # run all tests containing 'kfp' or 'katib' in their name
-   tox -e uats -- --filter "kfp or katib"
-   # run any test that doesn't contain 'kserve' in its name
-   tox -e uats -- --filter "not kserve"
-   ```
+```bash
+# assumes an existing `kubeflow` Juju model
+tox -e uats-remote -- --branch tests-branch
+```
+
+#### Run tests from local copy
+
+This one works when running the tests from the same node where the tests job is deployed (e.g. running from the same machine where the Microk8s cluster lives). This happens because in this case, the tests job instantiates a volume that is [mounted to the local directory of the repository where tests reside](https://github.com/canonical/charmed-kubeflow-uats/blob/ee0fa08931b11f40e97dbe3e340c413cf466a084/assets/test-job.yaml.j2#L34-L36), while the job and the pod are scheduled in a remote node. If unsure about your setup, use the `remote branch`.
+
+```bash
+# assumes an existing `kubeflow` Juju model
+tox -e uats-local
+```
+
+#### Run a subset of UATs
+
+You can also run a subset of the provided tests using the `--filter` option and passing a filter
+that follows the same syntax as the pytest `-k` option, e.g.
+
+```bash
+# run all tests containing 'kfp' or 'katib' in their name
+tox -e uats-local -- --filter "kfp or katib"
+# run any test that doesn't contain 'kserve' in its name
+tox -e uats-remote -- --filter "not kserve"
+```
 
-  This simulates the behaviour of running `pytest -k "some filter"` directly on the test suite.
-  You can read more about the options provided by Pytest in the corresponding section of the
-  [documentation](https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags).
+This simulates the behaviour of running `pytest -k "some filter"` directly on the test suite.
+You can read more about the options provided by Pytest in the corresponding section of the
+[documentation](https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags).
 
 #### Run Kubeflow UATs
 
@@ -105,7 +121,10 @@ dedicated `kubeflow` tox test environment:
 
 ```bash
 # assumes an existing `kubeflow` Juju model
-tox -e kubeflow
+# use tests from the local copy of the repo
+tox -e kubeflow-local
+# use tests from a remote branch
+tox -e kubeflow-remote -- --branch=<remote-branch>
 ```
 
 #### Developer Notes
@@ -117,7 +136,7 @@ a Kubernetes Job to run the tests. More specifically, the `driver` executes the
 1. Create a Kubeflow Profile (i.e. `test-kubeflow`) to run the tests in
 2. Submit a Kubernetes Job (i.e. `test-kubeflow`) that runs `tests`
    The Job performs the following:
-   * Mount the local `tests` directory to a Pod that uses `jupyter-scipy` as the container image
+   * If a `-local` tox environment is run, then it mounts the local `tests` directory to a Pod that uses `jupyter-scipy` as the container image. Else (in `-remote` tox environments), it creates an emptyDir volume which it syncs to the remote branch of uats provided, using a [git-sync](https://github.com/kubernetes/git-sync/) `initContainer`.
    * Install python dependencies specified in the [requirements.txt](tests/requirements.txt)
    * Run the test suite by executing `pytest`
 3. Wait until the Job completes (regardless of the outcome)

assets/test-job.yaml.j2

@@ -11,16 +11,28 @@ spec:
           access-ml-pipeline: "true"
           mlflow-server-minio: "true"
     spec:
+      {% if tests_local_run == "false" %}
+      # securityContext is needed in order for test files to be writeable 
+      # since the tests save the notebooks. Setting it enables:
+      # * The test-volume to be group-owned by this GID.
+      # * The GID to be added to each container.
+      securityContext:
+        fsGroup: 101
+      {% endif %}
       serviceAccountName: default-editor
       containers:
         - name: {{ job_name }}
-          image: {{ test_image }}
+          image: {{ tests_image }}
           command:
             - bash
             - -c
           args:
             - |
+              {% if tests_local_run == "true" %}
               cd /tests;
+              {% else %}
+              cd /tests/charmed-kubeflow-uats/tests;
+              {% endif %}
               pip install -r requirements.txt >/dev/null;
               {{ pytest_cmd }};
               # Kill Istio Sidecar after workload completes to have the Job status properly updated
@@ -30,8 +42,28 @@ spec:
           volumeMounts:
             - name: test-volume
               mountPath: /tests
+      {% if tests_local_run == "false" %}
+      initContainers:
+        - name: git-sync
+          # This container pulls git data and publishes it into volume
+          # "test-volume".
+          image: registry.k8s.io/git-sync/git-sync:v4.0.0
+          args:
+            - --repo=https://github.com/canonical/charmed-kubeflow-uats
+            - --ref={{ tests_remote_branch }}
+            - --root=/tests
+            - --group-write
+            - --one-time
+          volumeMounts:
+            - name: test-volume
+              mountPath: /tests
+      {% endif %}
       volumes:
         - name: test-volume
+          {% if tests_local_run == "true" %}
           hostPath:
-            path: {{ test_dir }}
+            path: {{ tests_local_dir }}
+          {% else %}
+          emptyDir: {}
+          {% endif %}
       restartPolicy: Never

driver/conftest.py

@@ -9,6 +9,9 @@ def pytest_addoption(parser: Parser):
 
     * Add a `--filter` option to (de)select test cases based on their name (see also
       https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags)
+    * Add a `--branch` option to select which remote branch to run tests from (see also
+      https://github.com/canonical/charmed-kubeflow-uats/issues/65#issuecomment-2016884170).
+      Defaults to `main`.
     """
     parser.addoption(
         "--filter",
@@ -18,3 +21,8 @@ def pytest_addoption(parser: Parser):
         " any test that doesn't contain 'kserve' in its name. Essentially, the option simulates"
         " the behaviour of running `pytest -k '<filter>'` directly on the test suite.",
     )
+    parser.addoption(
+        "--branch",
+        help="Provide a remote branch from charmed-kubeflow-uats repository which will be used "
+        " to checkout and run the UATs from.",
+    )

driver/test_kubeflow_workloads.py

@@ -16,7 +16,9 @@
 JOB_TEMPLATE_FILE = ASSETS_DIR / "test-job.yaml.j2"
 PROFILE_TEMPLATE_FILE = ASSETS_DIR / "test-profile.yaml.j2"
 
-TESTS_DIR = os.path.abspath(Path("tests"))
+TESTS_LOCAL_RUN = os.environ.get("LOCAL").replace('"', "")
+TESTS_LOCAL_DIR = os.path.abspath(Path("tests"))
+
 TESTS_IMAGE = "kubeflownotebookswg/jupyter-scipy:v1.7.0"
 
 NAMESPACE = "test-kubeflow"
@@ -39,6 +41,13 @@ def pytest_filter(request):
     return f"-k '{filter}'" if filter else ""
 
 
+@pytest.fixture(scope="session")
+def tests_remote_branch(request):
+    """Retrieve active git branch from Pytest invocation."""
+    branch = request.config.getoption("branch")
+    return branch if branch else "main"
+
+
 @pytest.fixture(scope="session")
 def pytest_cmd(pytest_filter):
     """Format the Pytest command."""
@@ -95,16 +104,18 @@ async def test_create_profile(lightkube_client, create_profile):
     assert_namespace_active(lightkube_client, NAMESPACE)
 
 
-def test_kubeflow_workloads(lightkube_client, pytest_cmd):
+def test_kubeflow_workloads(lightkube_client, pytest_cmd, tests_remote_branch):
     """Run a K8s Job to execute the notebook tests."""
     log.info(f"Starting Kubernetes Job {NAMESPACE}/{JOB_NAME} to run notebook tests...")
     resources = list(
         codecs.load_all_yaml(
             JOB_TEMPLATE_FILE.read_text(),
             context={
                 "job_name": JOB_NAME,
-                "test_dir": TESTS_DIR,
-                "test_image": TESTS_IMAGE,
+                "tests_local_run": TESTS_LOCAL_RUN,
+                "tests_local_dir": TESTS_LOCAL_DIR,
+                "tests_image": TESTS_IMAGE,
+                "tests_remote_branch": tests_remote_branch,
                 "pytest_cmd": pytest_cmd,
             },
         )

tox.ini

@@ -52,7 +52,8 @@ commands =
     codespell {toxinidir}/. --skip {toxinidir}/./.git --skip {toxinidir}/./.tox \
       --skip {toxinidir}/./build --skip {toxinidir}/./lib --skip {toxinidir}/./venv \
       --skip {toxinidir}/./.mypy_cache \
-      --skip {toxinidir}/./icon.svg --skip *.json.tmpl
+      --skip {toxinidir}/./icon.svg --skip *.json.tmpl \
+      --ignore-regex=".*codespell-ignore."
     # pflake8 wrapper supports config from pyproject.toml
     pflake8 {[vars]all_path}
     isort --check-only --diff {[vars]all_path}
@@ -61,23 +62,30 @@ deps =
     -r requirements-lint.txt
 description = Check code against coding style standards
 
-[testenv:kubeflow]
+[testenv:kubeflow-{local,remote}]
 commands =
     # run all tests apart from the ones that use MLFlow
     pytest -vv --tb native {[vars]driver_path} -s --filter "not mlflow" --model kubeflow {posargs}
+setenv =
+    local: LOCAL = "true"
+    remote: LOCAL = "false"
 deps =
     -r requirements.txt
 description = Run UATs for Kubeflow
 
-[testenv:uats]
+[testenv:uats-{local,remote}]
 # provide a filter when calling tox to (de)select test cases based on their names, e.g.
 # * run all tests containing 'kfp' or 'katib' in their name:
 #   $ tox -e uats -- --filter "kfp or katib"
 # * run any test that doesn't contain 'kserve' in its name:
 #   $ tox -e uats -- --filter "not kserve"
 # this simulates the behaviour of running 'pytest -k "<filter>"' directly on the test suite:
 # https://docs.pytest.org/en/7.4.x/reference/reference.html#command-line-flags
-commands = pytest -vv --tb native {[vars]driver_path} -s --model kubeflow {posargs}
+commands = 
+    pytest -vv --tb native {[vars]driver_path} -s --model kubeflow {posargs}
+setenv =
+    local: LOCAL = "true"
+    remote: LOCAL = "false"
 deps =
     -r requirements.txt
 description = Run UATs for Kubeflow and Integrations