Skip to content

Commit

Permalink
feat: Add initial E2E test and structure
Browse files Browse the repository at this point in the history 
This adds a single test to create a CAPMVM workload cluster and a bit of
structure so that more tests can be added later.

Note that only CAPMVM on the current branch is tested, not released
versions or upgrades between versions. I have deliberately left it so we
can add those bit by bit after the general setup is merged in.

I have a lot more to do in the e2e saga, but this is a first step which
is useable.

I used the CAPI test framework[1] which seems to be something of a
standard across providers, although it is a bit unwieldy.

Everything can be found under `test/e2e/`.

The contents of `test/e2e/data` are mostly yaml and can be ignored.

The dir `test/e2e/config` contains the config file used by the CAPI test
framework. Other config files can be passed in via test params but this
is the default.

The test itself is at `test/e2e/e2e_test.go` and
`test/e2e/e2e_suite_test.go`. The suite performs some env setup and
teardown by way of a manager. It is assumed for now that the tests are
not run in parallel (there is only one of them) so I did not set up any
synchronization between nodes, but that will need to be done as we add
more tests and parallelization.

The test creates a cilium flavour cluster, mainly because I could not
get the other one to work and did not want to spend a lot of time fixing
that. After the cluster is created it checks that all flintlock hosts
were used, and that a simple application can be successfully deployed.

In `test/e2e/utils` there are a bunch of helper things, which will
probably end up being broken up as they expand:
- `params.go` sets up test flags
- `defaults.go` holds defaults for test flags
- `utils.go` is a bunch of helper funcs used by the test
- `clusterctl.go` contains some functions I copied from the framework
  because I had to edit them and they are not hugely flexible.
- `manager.go` holds some objects and methods which handle environment
  setup/teardown and shared test information.

The test can be run locally by:
- Pulling the images listed at the top of
  `test/e2e/config/e2e_conf.yaml` to the local machine running the tests
  (except for the capmvm image, which will be built)
- Starting 1 or more flintlock servers
- Exporting the server addresses `export
  FLINTLOCK_HOSTS="1.2.3.4:9090,5.6.7.8:9090"`
- Running `make e2e`

[1]: https://pkg.go.dev/sigs.k8s.io/cluster-api/test/framework
  • Loading branch information
@Callisto13
Callisto13 committed Jul 27, 2022
1 parent 18f3e38 commit 50f983e
Show file tree
Hide file tree
Showing 21 changed files with 2,038 additions and 116 deletions.
21 changes: 19 additions & 2 deletions Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ COUNTERFEITER := $(TOOLS_BIN_DIR)/counterfeiter
GOLANGCI_LINT := $(TOOLS_BIN_DIR)/golangci-lint
CONTROLLER_GEN := $(TOOLS_BIN_DIR)/controller-gen
DEFAULTER_GEN := $(TOOLS_BIN_DIR)/defaulter-gen
GINKGO := $(TOOLS_BIN_DIR)/ginkgo

.DEFAULT_GOAL := help

Expand All @@ -69,9 +70,19 @@ endif
lint: $(GOLANGCI_LINT) ## Lint
$(GOLANGCI_LINT) run -v --fast=false

##@ Testing

# TODO fix this to use tags or something
.PHONY: test
test: ## Run tests.
go test -v ./...
go test -v ./controllers/... ./internal/...

TEST_ARTEFACTS := $(REPO_ROOT)/test/e2e/_artefacts
FLINTLOCK_HOSTS := ${FLINTLOCK_HOSTS}

.PHONY: e2e
e2e: $(GINKGO) e2e-image ## Run end to end test suite.
$(GINKGO) -tags=e2e -v -r test/e2e -- -e2e.artefact-dir $(TEST_ARTEFACTS) -e2e.flintlock-hosts $(FLINTLOCK_HOSTS)

##@ Binaries

Expand All @@ -82,7 +93,6 @@ build: managers ## Build manager binary.
managers: ## Build manager binary.
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags "${LDFLAGS} -extldflags '-static'" -o $(BIN_DIR)/manager .


##@ Docker

.PHONY: docker-build
Expand All @@ -92,6 +102,10 @@ docker-build: docker-pull-prerequisites ## Build docker image with the manager.
docker-push: ## Push docker image with the manager.
docker push $(CONTROLLER_IMAGE):$(TAG)

.PHONY: e2e-image
e2e-image: docker-pull-prerequisites ## Build docker image with the manager.
docker build --build-arg ARCH=$(ARCH) --build-arg LDFLAGS="$(LDFLAGS)" . -t $(CONTROLLER_IMAGE):e2e

.PHONY: docker-pull-prerequisites
docker-pull-prerequisites:
docker pull docker.io/docker/dockerfile:1.1-experimental
Expand Down Expand Up @@ -147,6 +161,9 @@ $(GOLANGCI_LINT): $(TOOLS_DIR)/go.mod # Get and build golangci-lint
$(COUNTERFEITER): $(TOOLS_DIR)/go.mod # Get and build counterfieter
cd $(TOOLS_DIR); go build -tags=tools -o $(subst hack/tools/,,$@) github.com/maxbrunsfeld/counterfeiter/v6

$(GINKGO): $(TOOLS_DIR)/go.mod # Get and build ginkgo v1
cd $(TOOLS_DIR); go build -tags=tools -o $(subst hack/tools/,,$@) github.com/onsi/ginkgo/ginkgo

##@ Utility

.PHONY: help
Expand Down
51 changes: 49 additions & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ go 1.17

require (
github.com/go-logr/logr v1.2.3
github.com/onsi/ginkgo v1.16.5
github.com/onsi/gomega v1.19.0
github.com/spf13/pflag v1.0.5
github.com/weaveworks-liquidmetal/flintlock/api v0.0.0-20220628141946-264f4544f49f
Expand All @@ -18,26 +19,44 @@ require (
k8s.io/klog/v2 v2.70.1
k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9
sigs.k8s.io/cluster-api v1.1.5
sigs.k8s.io/cluster-api/test v1.1.5
sigs.k8s.io/controller-runtime v0.12.3
)

// pinning this because cluster-api needs 0.23.5, but something else is
// pulling in 0.24.2 which is breaking the e2es.
replace k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.23.5

require (
cloud.google.com/go v0.93.3 // indirect
cloud.google.com/go v0.99.0 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest v0.11.18 // indirect
github.com/Azure/go-autorest/autorest/adal v0.9.13 // indirect
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/BurntSushi/toml v1.0.0 // indirect
github.com/MakeNowJust/heredoc v1.0.0 // indirect
github.com/Masterminds/goutils v1.1.1 // indirect
github.com/Masterminds/semver/v3 v3.1.1 // indirect
github.com/Masterminds/sprig/v3 v3.2.2 // indirect
github.com/Microsoft/go-winio v0.5.0 // indirect
github.com/PuerkitoBio/purell v1.1.1 // indirect
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/alessio/shellescape v1.4.1 // indirect
github.com/antlr/antlr4/runtime/Go/antlr v0.0.0-20210826220005-b48c857c3a0e // indirect
github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/coredns/caddy v1.1.0 // indirect
github.com/coredns/corefile-migration v1.0.17 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/docker/distribution v2.8.1+incompatible // indirect
github.com/docker/docker v20.10.16+incompatible // indirect
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/drone/envsubst/v2 v2.0.0-20210730161058-179042472c46 // indirect
github.com/emicklei/go-restful v2.9.5+incompatible // indirect
github.com/evanphx/json-patch v4.12.0+incompatible // indirect
github.com/evanphx/json-patch/v5 v5.6.0 // indirect
Expand All @@ -50,25 +69,49 @@ require (
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/cel-go v0.10.1 // indirect
github.com/google/gnostic v0.5.7-v3refs // indirect
github.com/google/go-cmp v0.5.6 // indirect
github.com/google/go-github/v33 v33.0.0 // indirect
github.com/google/go-querystring v1.0.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.1.2 // indirect
github.com/google/uuid v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.6.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/huandu/xstrings v1.3.2 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/magiconair/properties v1.8.5 // indirect
github.com/mailru/easyjson v0.7.6 // indirect
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/mapstructure v1.4.2 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/pelletier/go-toml v1.9.4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/prometheus/client_golang v1.12.1 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/common v0.32.1 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
github.com/shopspring/decimal v1.2.0 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spf13/afero v1.6.0 // indirect
github.com/spf13/cast v1.4.1 // indirect
github.com/spf13/cobra v1.4.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/viper v1.9.0 // indirect
github.com/stoewer/go-strcase v1.2.0 // indirect
github.com/subosito/gotenv v1.2.0 // indirect
github.com/valyala/fastjson v1.6.3 // indirect
golang.org/x/crypto v0.0.0-20220214200702-86341886e292 // indirect
golang.org/x/net v0.0.0-20220225172249-27dd8689420f // indirect
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
Expand All @@ -80,12 +123,16 @@ require (
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/ini.v1 v1.63.2 // indirect
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
k8s.io/apiextensions-apiserver v0.24.2 // indirect
k8s.io/apiserver v0.24.2 // indirect
k8s.io/cluster-bootstrap v0.23.0 // indirect
k8s.io/component-base v0.24.2 // indirect
k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 // indirect
sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect
sigs.k8s.io/kind v0.14.0 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)
Loading

0 comments on commit 50f983e

Please sign in to comment.