Skip to content

Latest commit

 

History

History
225 lines (161 loc) · 8.41 KB

automated_tests.md

File metadata and controls

225 lines (161 loc) · 8.41 KB
Raw

EKS Workshop - Automated End-to-End Tests

Automated end-to-end tests are used to ensure the stability of the workshop content over time, and are integrated in to the Pull Request process to allow validation of PRs before they are merged. This document outlines how to use automated tests in various scenarios.

Running tests

This section documents running the tests locally as well as via PRs.

Locally

You should run the tests locally before raising a PR, this can be done with some convenience scripts.

To use this utility make sure you have:

  • Run make create-infrastructure as outlined here
  • Run make shell as outlined here

First, ensure you have the workshop infrastructure running in your AWS account:

make create-infrastructure

Then run make test specifying the top-level module that you wish to test. This limits the tests to the a specific directory relative to website/docs. For example to test only the content for the autoscaling module located at `website/docs/autoscaling' the command would be:

➜  eks-workshop-v2 git:(main) ✗ make test module="autoscaling"
bash hack/run-tests.sh
Generating temporary AWS credentials...
Building container images...
sha256:62fb5cc6e348d854a59a5e00429554eab770adc24150753df2e810355e678899
sha256:a6b3c8675c79804587b5e4d5b8dfc5cfd6d01b40c89ee181ad662902e0cb650d
Running test suite...
Added new context arn:aws:eks:us-west-2:111111111:cluster/eks-workshop to /root/.kube/config
✔ Generating test cases
✔ Building Mocha suites

Executing tests...


EKS Workshop
    Autoscaling
      - Autoscaling
      Workloads
        - Workloads
        Horizontal Pod Autoscaler
          - Horizontal Pod Autoscaler
          ✔ Configure HPA
          ✔ Generate load
        Cluster Proportional Autoscaler
          - Cluster Proportional Autoscaler
          ✔ Installing CPA
          ✔ Autoscaling CoreDNS
      Compute
        - Compute
        Cluster Autoscaler (CA)
          ✔ Cluster Autoscaler (CA)
          ✔ Enable CA
          ✔ Scale with CA
          ✔ Clean Up
        Karpenter
          ✔ Karpenter
          ✔ Set up the Provisioner
          ✔ Automatic Node Provisioning
          ✔ Remove Provisioner
        Cluster Over-Provisioning
          ✔ Cluster Over-Provisioning
          ✔ Introduction
          ✔ Setting up Over-Provisioning
          ✔ Scale a workload


  13 passing (2m)
  8 pending

success

When you start testing your content you should likely test your module in isolation but prior to raising a PR you should check that the top-level module your content is a part of runs in its entirety.

For example if you are working on content at website/docs/autoscaling/compute/karpenter you should start by testing like so:

make test module="autoscaling/compute/karpenter"

But prior to raising a PR should verify the autoscaling module in its entirety:

make test module="autoscaling"

You can pass extra flag to the test framework by using environment variable AWS_EKS_WORKSHOP_TEST_FLAGS

AWS_EKS_WORKSHOP_TEST_FLAGS="--debug" make test module="gitops"

Finally, once you are done if needed you can destroy the infrastructure:

make destroy-infrastructure

Writing tests

The automated tests primarily work by extracting all of the bash code snippets out of the Markdown files and executing them in order. It is aware of weight and sidebar_position so does not need any further help to figure this out. This means that by adhering to the recommendations in the style guide your commands will have tests generated for them without any further action from you.

Any commands that return an error code will be marked as a failed unit test case.

Note: The framework will only extract commands from bash sections that are prefixed with $ , see the style guide section on 'Command blocks' for more information.

Take the following Markdown example:

# Some example

This is some example content to demonstrate testing.

```bash
$ ls -la
Some example output
```

In this case the test framework will pull out and run ls -la, ignoring Some example output.

Hooks

The test framework introduces the concept of 'hooks', which are used to decorate test cases with additional logic or assertions that should not be shown to the end-user of the content. Most coding unit test frameworks have similar concepts, for example hooks in Mocha.

This is especially important when operating with an asynchronous system such as Kubernetes, where running kubectl apply [...] can often involve waiting several minutes for something to happen after the command returns success. For example creating a Service of type LoadBalancer.

Hooks are introduced to a bash snippet using the optional hook annotation, which uses a convention to pick up bash script from a directory relative to the Markdown file.

For example take this case where the Markdown is in my-content.md:

# Some example hook

This is some example content to demonstrate testing with hooks.

Lets tell the user to create a file with `touch`:

```bash hook=example
$ touch /tmp/my-file
Some example output
```

The directory structure might look like this:

--- website
    `-- docs
        `-- my-modules
            |-- tests
            |   |-- hook-example.sh
            `-- my-content.md

The framework will automatically look in the tests directory for a file called hook-${hook-name}.sh, and will execute it before and after the bash snippet. The only argument passed is before or after to provide context which it is getting called.

Here is an example implementation of a hook file hook-example.sh that checks /tmp/my-file doesn't exist when the test starts and was created after the bash snippet in the Markdown is completed.

set -Eeuo pipefail

before() {
  if [[ -f /tmp/my-file ]]; then
    # Try to provide an error message where possible to help understanding failure
    echo 'Error: Expected file to not be present'
    # Exit with a non-zero code for failure
    exit 1
  fi

  # Otherwise let the hook exit naturally
}

after() {
  if [[ ! -f /tmp/my-file ]]; then
    echo 'Error: Expected file does not exist'
    exit 1
  fi
}

"$@"

Other annotations

The framework includes support for a number of annotations that can be added to the code fence blocks in Markdown to customize the behavior for certain shell code snippets.

For example, if there are bash segments you do not want to run you can indicate they should be skipped:

```bash test=false
$ echo "This won't get run"
```

For cases where theres a concerned a script might run for too long or not finish, you can specify a timeout in seconds (default is 60 seconds):

```bash timeout=120
$ echo "This test case will fail"
$ sleep 150
```

Here is a complete list of the available annotations:

Annotation Description Default
test This script block should be executed as a test true
timeout Time limit in seconds before the script block will be marked as failed 120
hook Name of the hook to execute for this script section
hookTimeout Time limit in seconds for the hooks to complete before the script block will be marked as failed 300
expectError Ignore any errors that occur when the script block is executed false

What if my module can't be tested?

This is not something we generally want to accept in to the workshop content as any manual testing represents significant toil in terms of maintenance when it comes to content changes and other events like upgrading EKS versions. PRs raised that do not have adequate considerations for testing will generally not be accepted.

However, in some cases it is necessary to prevent a module or chapter from being tested. In that case you can place a file called .notest in the directory which should be excluded. This will cause the framework to ignore that directry and any sub-directories automatically.