Merge pull request #2670 from chuckha/testing-doc

📖 Update testing document

docs/book/src/developer/testing.md

@@ -1,22 +1,52 @@
-# Running Cluster API tests
+# Testing Cluster API
 
-## Unit tests
+## Basic tests
 
-Unit tests run very quickly. They don't require any additional services and can be run using default go tools or through
-the `test` make target, e.g. `make test`.
+### Unit tests
+
+Unit tests run very quickly. They don't require any additional services to execute. They focus on individual pieces of
+logic and allow integration bugs to slip through. They are fast and great for getting the initial implementation worked
+out.
+
+### `envtest`
+
+`envtest` is a testing environment that is provided by `kubebuilder`. This environment spins up a local instance of
+etcd and the kube-apiserver. This allows reconcilers to do all the things they need to do to function very similarly
+to how they would in a real environment. These tests provide integration testing between reconcilers and kubernetes.
+
+### Running unit and `envtest` tests
+
+Using the `test` target through `make` will run all of the unit and `envtest` tests.
 
 ## Integration tests
 
 Integration tests use a real cluster and real dependencies to run tests. The dependencies are managed manually and are
-not meant to be run locally. See `scripts/ci-integration.sh` for more details.
+not meant to be run locally. This is used during CI to ensure basic functionality works as expected.
+
+See `scripts/ci-integration.sh` for more details.
 
 ## End-to-end tests
 
 The end-to-end tests are similar to the integration tests except they are designed to manage dependencies for you and 
-have a more complete test using a docker provider to test the Cluster API mechanisms. The end-to-end tests run best on 
-CI, but work locally as well.
+have a more complete test using a docker provider to test the Cluster API mechanisms. The end-to-end tests can run
+locally without modifying the host machine or on a CI platform.
+
+### Running the end-to-end tests
 
-### Running the tests
+#### Environment variables
+
+The test run can be controlled to some degree by environment variables.
+
+* `SKIP_RESOURCE_CLEANUP`: Will prevent the end-to-end tests from cleaning up the local resources they create. This is
+useful when debugging an error that requires cluster inspection. After the error occurs and the tests fail, the cluster
+remains. You can then `docker exec` into a running container to get information to assist in debugging. This parameter
+is also useful if you simply want a local cluster to experiment with. After the tests run and succeed you are left
+with a cluster in a known working state for experimentation.
+
+* `FOCUS`: The `FOCUS` variable allows running of certain tests. The default on CI is `FOCUS='Basic'`. In order to run
+all the tests, set `FOCUS='Basic|Full'`. This will run the basic test and then run all other tests.
+
+### Types of end-to-end runs
 
 `make test-capd-e2e-full` will build all manifests and provider images then use those manifests and images in the test
 run. This is great if you're modifying API types in any of the providers.
@@ -27,3 +57,9 @@ use whatever manifests exist on disk. This is good if you're only updating contr
 `make test-capd-e2e` will only build the docker provider image and use whatever provider images already exist on disk.
 This is good if you're working on the test framework itself. You'll likely want to build the images at least once
 and then use this for a faster test run.
+
+### Examples
+
+* `make test-capd-e2e-full SKIP_RESOURCE_CLEANUP=true`
+* `make test-capd-e2e-images`
+* `make test-capd-e2e FOCUS='Basic|Full'`