Switching to kuttl for MWT (#138)

* Switching to kuttl for MWT

Signed-off-by: Ken Sipe <[email protected]>

* mods based on PR review Co-authored-by: Zain Malik <[email protected]>

Signed-off-by: Ken Sipe <[email protected]>

* Switching to kuttl for MWT

Signed-off-by: Ken Sipe <[email protected]>

* mods based on PR review Co-authored-by: Zain Malik <[email protected]>

Signed-off-by: Ken Sipe <[email protected]>

* adding test config files

Signed-off-by: Ken Sipe <[email protected]>

* switch to cassandra 1.0.0

Signed-off-by: Ken Sipe <[email protected]>

* Ran ./tools/format_files.sh to make TC happy

Signed-off-by: Andreas Neumann <[email protected]>

Co-authored-by: Andreas Neumann <[email protected]>

benchmarks/README.md

@@ -4,61 +4,29 @@
 
 ### Set up some KUDO Cassandra instance information
 
+1. Ensure a configuration with clsuter
+2. cd mwt
+3. read [README](mwt/README.md)
+
 ```bash
 kudo_cassandra_operator_name="cassandra"
 kudo_cassandra_operator_version="1.0.0"
 kudo_cassandra_instance_name="cassandra"
 kudo_cassandra_instance_namespace="cassandra"
-kudo_cassandra_statefulset_name="node"
 
 svc_endpoint="${kudo_cassandra_instance_name}-svc.${kudo_cassandra_instance_namespace}.svc.cluster.local"
 ```
 
-### Create namespace
+### Setup and Verify KUDO Cassandra
 
-```bash
-kubectl create namespace "${kudo_cassandra_instance_namespace}"
-```
+From `mwt`  
+Run `kubectl kuttl test setup/ --parallel 1 --skip-delete`  
+Which will:
 
-### Install KUDO Cassandra with some parameters
-
-Tweak the parameters as desired.
-
-```bash
-kubectl kudo install ./operator \
-        --instance="${kudo_cassandra_instance_name}" \
-        --namespace="${kudo_cassandra_instance_namespace}" \
-        -p "NODE_COUNT=10" \
-        -p "NODE_MEM_MIB=8196" \
-        -p "NODE_MEM_LIMIT_MIB=8196" \
-        -p "NODE_CPU_MC=6000" \
-        -p "NODE_CPU_LIMIT_MC=6000" \
-        -p "NODE_DISK_SIZE_GIB=100" \
-        -p "PROMETHEUS_EXPORTER_CPU_MC=1000" \
-        -p "PROMETHEUS_EXPORTER_CPU_LIMIT_MC=1000"
-```
-
-### Wait for deploy plan to be `COMPLETE`
-
-```bash
-kubectl kudo plan status \
-        --instance="${kudo_cassandra_instance_name}" \
-        --namespace="${kudo_cassandra_instance_namespace}"
-```
-
-### Make sure `nodetool status` looks good
-
-```bash
-pod="0"
-container="cassandra"
-kubectl exec "pod/${kudo_cassandra_instance_name}-${kudo_cassandra_statefulset_name}-${pod}" \
-        -n "${kudo_cassandra_instance_namespace}" \
-        -c "${container}" \
-        -- \
-        bash -c "\
-          nodetool status
-        "
-```
+1. verify setup
+2. install cassandra
+3. wait for deployment to finish
+4. output nodetool status
 
 ### Run `kassandra-stress`
 
@@ -106,9 +74,5 @@ kubectl delete <deployment> -n "${kudo_cassandra_instance_namespace}"
 
 ### Uninstall operator
 
-```bash
-./scripts/uninstall_operator.sh \
-  --operator "${kudo_cassandra_operator_name}" \
-  --version "${kudo_cassandra_operator_version}" \
-  --instance "${kudo_cassandra_instance_name}" \
-```
+From `mwt`  
+Run `kubectl kuttl test teardown/`

benchmarks/mwt/.gitignore

@@ -0,0 +1 @@
+kubeconfig

benchmarks/mwt/README.md

@@ -0,0 +1,62 @@
+1. assert kudo  
+   k kudo init --wait  
+   k kudo init --wait --unsafe-self-signed-webhook-ca
+
+2. create ns
+3. install CR  
+   assert
+4. install with params  
+   with instance name  
+   assert
+
+## Prerequisites
+
+- Kubernetes cluster up
+- KUDO CLI installed on node kuttl is running from
+- KUDO manager installed in the cluster
+
+# TestSuites
+
+- **Setup:** used to setup a MWT test use the following command
+
+`kubectl kuttl test setup/ --parallel 1 --skip-delete`
+
+or
+
+`kubectl kuttl test --config kuttl-setup.yaml`
+
+The `parallel 1` is important for setup, as the order of the folders and the
+order of there execution matters. The tests are designed to allow for a restart.
+If kudo confirmation is guaranteed, it is possible to run:
+`kubectl kuttl test setup/ --parallel 1 --skip-delete --test 01-cassandra-install`.
+The tests inside "cassandra-install" are also allowed to run multiple tests with
+good end results. This is the reason command failures are ignored. If a command
+fails we don't care unless the desired asserted state is not reached (which is
+built into the test).
+
+- **Teardown:** used to remove cassandra (and verify it has been removed)
+
+`kubectl kuttl test --config kuttl-teardown.yaml`
+
+## Tests under **Setup**
+
+- 00-kudo-check: confirms
+
+  1. commands locally are present
+  2. kudo manager is ready at the server
+
+- 01-cassandra-install
+  1. first creates and asserts "cassandra" namespace exists
+  2. installs the cassandra CR and asserts they exist
+  3. installs cassandra with parameters and asserts that the deployment plan is
+     complete
+  4. runs the nodetool on node 0 for output to stdout
+
+**NOTES:**
+
+1. The MWT parameter values need to be changed. This was built and tested on
+   konvoy but not on a MWT env
+2. The timeouts may need to change. In particular the wait for namespace, the
+   wait for deploy to finish and the wait for deletes.
+3. This was tested with konvoy with config setting established with
+   `./konvoy apply kubeconfig --force-overwrite`

benchmarks/mwt/kuttl-setup.yaml

@@ -0,0 +1,7 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestSuite
+parallel: 1
+skipDelete: true
+timeout: 30
+testDirs:
+  - setup

benchmarks/mwt/kuttl-teardown.yaml

@@ -0,0 +1,6 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestSuite
+parallel: 1
+timeout: 30
+testDirs:
+  - teardown

benchmarks/mwt/setup/00-kudo-check/00-assert.yaml

@@ -0,0 +1,17 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  labels:
+    app: kudo-manager
+  name: kudo-controller-manager-0
+  namespace: kudo-system
+status:
+  containerStatuses:
+  - name: manager
+    ready: true
+
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: instances.kudo.dev

benchmarks/mwt/setup/00-kudo-check/00-kudo-setup.yaml

@@ -0,0 +1,5 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestStep
+commands:
+  - command: kubectl kudo init --client-only
+# ensures all commands are present and that kudo is installed locally... test failure if not true

benchmarks/mwt/setup/01-cassandra-install/00-assert.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: cassandra
+status:
+  phase: Active

benchmarks/mwt/setup/01-cassandra-install/00-install-namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestStep
+commands:
+  - command: kubectl create namespace cassandra
+    ignoreFailure: true
+#  if we can't create a namespace, it is likely because it already exists, if that is the case, 00-assert will catch it.

benchmarks/mwt/setup/01-cassandra-install/01-assert.yaml

@@ -0,0 +1,5 @@
+apiVersion: kudo.dev/v1beta1
+kind: Operator
+metadata:
+  name: cassandra
+  namespace: cassandra

benchmarks/mwt/setup/01-cassandra-install/01-install-crd.yaml

@@ -0,0 +1,4 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestStep
+commands:
+  - command: kubectl kudo install cassandra --skip-instance --namespace cassandra  

benchmarks/mwt/setup/01-cassandra-install/02-assert.yaml

@@ -0,0 +1,15 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestAssert
+timeout: 600
+---
+
+apiVersion: kudo.dev/v1beta1
+kind: Instance
+metadata:
+  name: cassandra
+  namespace: cassandra
+status:
+  planStatus:
+    deploy:
+      name: deploy
+      status: COMPLETE

benchmarks/mwt/setup/01-cassandra-install/02-install-cassandra.yaml

@@ -0,0 +1,20 @@
+apiVersion: kudo.dev/v1beta1
+kind: Instance
+metadata:
+  name: cassandra
+  namespace: cassandra
+spec:
+  operatorVersion:
+    name: cassandra-1.0.0
+    namespace: cassandra
+    kind: OperatorVersion
+  name: "cassandra"
+  parameters:
+    NODE_COUNT: "10"
+    NODE_MEM_MIB: "8196"
+    NODE_MEM_LIMIT_MIB: "8196"
+    NODE_CPU_MC: "6000"
+    NODE_CPU_LIMIT_MC: "6000"
+    NODE_DISK_SIZE_GIB: "100"
+    PROMETHEUS_EXPORTER_CPU_MC: "1000"
+    PROMETHEUS_EXPORTER_CPU_LIMIT_MC: "1000"

benchmarks/mwt/setup/01-cassandra-install/03-nodetool.yaml

@@ -0,0 +1,4 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestStep
+commands:
+  - command: kubectl exec "pod/cassandra-node-0" -n cassandra -c cassandra -- bash -c "nodetool status" 

benchmarks/mwt/setup/01-cassandra-install/params.yaml

@@ -0,0 +1,7 @@
+  parameters:
+    NODE_COUNT: "3"
+    NODE_MEM_MIB: "4096"
+    NODE_MEM_LIMIT_MIB: "4096"
+    NODE_CPU_MC: "1000"
+    NODE_CPU_LIMIT_MC: "1000"
+    NODE_DISK_SIZE_GIB: "20"

benchmarks/mwt/teardown/teardown/00-delete.yaml

@@ -0,0 +1,25 @@
+apiVersion: kuttl.dev/v1beta1
+kind: TestStep
+commands:
+  - command: kubectl delete instance cassandra --namespace cassandra
+    ignoreFailure: true
+  - command: kubectl delete operatorversion --all --namespace cassandra
+    ignoreFailure: true
+  - command: kubectl delete operator cassandra --namespace cassandra
+    ignoreFailure: true
+
+#  kuttl doesn't support custom namespace deletes... yet
+# delete:
+# - apiVersion: kudo.dev/v1beta1
+#   kind: Instance
+#   metadata:
+#     name: cassandra
+#     namespace: cassandra
+# - apiVersion: kudo.dev/v1beta1
+#   kind: OperatorVersion
+#   metadata:
+#     namespace: cassandra
+# - apiVersion: kudo.dev/v1beta1
+#   kind: Operator
+#   metadata:
+#     namespace: cassandra

benchmarks/mwt/teardown/teardown/00-errors.yaml

@@ -0,0 +1,17 @@
+apiVersion: kudo.dev/v1beta1
+kind: Instance
+metadata:
+  name: cassandra
+  namespace: cassandra
+
+---
+apiVersion: kudo.dev/v1beta1
+kind: OperatorVersion
+metadata:
+  namespace: cassandra
+
+---
+apiVersion: kudo.dev/v1beta1
+kind: Operator
+metadata:
+  namespace: cassandra