End to end tests are ridiculously complicated and maybe impossible to run locally

[bfjelds]

The CI e2e tests use workflow-generated files and python scripts to configure runs for PRs, releases, pushes.

To create a new test, you need to understand: python, github action yaml, shell scripts, etc.

As a minimum bar, the tests should have documentation that:

1. describes how they run, when containers are downloaded vs created
2. describes how to run locally
3. describes how to modify them

A better test "system" would probably be if the tests:

1. depended on env vars rather than files
2. were written in shell script rather than python

[github-actions]

Issue has been automatically marked as stale due to inactivity for 45 days. Update the issue to remove label, otherwise it will be automatically closed.

[kate-goldenring]

The e2e tests can be run locally but they require a cluster to be set up an test configuration. Ideally, this e2e could include cluster set up and tear down.

Current flow of running tests:

1. Set up cluster
2. Install Python and pip.
3. Install the kubernetes Python package pip install kubernetes
4. Set up the test inputs

Setting up test inputs

1. Tell test to start immediately

   echo 0 > /tmp/sleep_duration.txt
   

2. Specify what version to test in /tmp/version_to_test.txt. To test latest version in main:

   curl https://raw.githubusercontent.com/deislabs/akri/main/version.txt | head -n1 > /tmp/version_to_test.txt
   

3. Specify what helm chart to test. This could be one from the output of a helm package command or it could be the latest built (main) like the following:

   echo "akri-helm-charts/akri-dev --version $(cat /tmp/version_to_test.txt)" > /tmp/helm_chart_location.txt
   

4. Set up crictl, KUBECONFIG, and installation alias. Should vary by k8s distribution.
   For k3s:

   echo '--set agent.host.crictl=/usr/local/bin/crictl --set agent.host.dockerShimSock=/run/k3s/containerd/containerd.sock' > /tmp/cri_args_to_test.txt
   echo '/etc/rancher/k3s/k3s.yaml' > /tmp/kubeconfig_path_to_test.txt
   echo 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl' > /tmp/runtime_cmd_to_test.txt
   

   For MicroK8s:

   echo '--set agent.host.crictl=/usr/local/bin/crictl --set agent.host.dockerShimSock=/var/snap/microk8s/common/run/containerd.sock' > /tmp/cri_args_to_test.txt
   echo '~/.kube/config' > /tmp/kubeconfig_path_to_test.txt
   echo 'microk8s kubectl' > /tmp/runtime_cmd_to_test.txt
   

5. Run the tests: python akri/test/run-end-to-end.py

Desired functionality

This is clearly very involved and not documented anywhere. The tests should be improved by

• Documenting how to run e2e tests locally
• Modified to use cli instead of files for passing parameters
• Include optional cluster creation and clean up
• Clear output on test results/logging
• Still be able to be executed as part of Akri's test cases workflow
• Reach: Support running Akri's components locally rather than only as containerized components. Essentially, you should be able to choose what helm deploys and what is run locally.

Examples

• Krustlet has a simple command for running it's end to end tests locally.

[diconico07]

Can we consider this done with #623 and project-akri/akri-docs#74 ?

[kate-goldenring]

Yes, thanks @diconico07!