From 855b60f54d23f1cd0339837dff110642eda942f3 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Sun, 29 Oct 2017 17:22:33 -0400 Subject: [PATCH 01/13] Support copy pasta --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index dd1e8e0c..813cfc76 100644 --- a/README.md +++ b/README.md @@ -67,9 +67,9 @@ At termination time, kube-monkey will: Clone the repository and build the container. ``` -$ go get github.com/asobti/kube-monkey -$ cd $GOPATH/src/github.com/asobti/kube-monkey -$ make container +go get github.com/asobti/kube-monkey +cd $GOPATH/src/github.com/asobti/kube-monkey +make container ``` ## Configuring From 1e958a14488cdb13a6cb6f6a802b4e4f85970b36 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Sun, 29 Oct 2017 17:22:55 -0400 Subject: [PATCH 02/13] KISS added some extra guiding instructions --- README.md | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 813cfc76..53610e1e 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,15 @@ -## kube-monkey -kube-monkey is an implementation of [Netflix's Chaos Monkey](https://github.com/Netflix/chaosmonkey) for [Kubernetes](http://kubernetes.io/) -clusters. It randomly deletes Kubernetes pods in the cluster encouraging and validating the development of failure-resilient -services. +# kube-monkey +kube-monkey is an implementation of [Netflix's Chaos Monkey](https://github.com/Netflix/chaosmonkey) for [Kubernetes](http://kubernetes.io/) clusters. It randomly deletes Kubernetes pods in the cluster encouraging and validating the development of failure-resilient services. --- +--- -kube-monkey runs at a pre-configured hour (`run_hour`, defaults to 8am) on weekdays, and builds a schedule of deployments that will face a random -Pod death sometime during the same day. The time-range during the day when the random pod Death might occur is configurable and -defaults to 10am to 4pm. +kube-monkey runs at a pre-configured hour (`run_hour`, defaults to 8am) on weekdays, and builds a schedule of deployments that will face a random Pod death sometime during the same day. The time-range during the day when the random pod Death might occur is configurable and defaults to 10am to 4pm. -kube-monkey can be configured with a list of namespaces to blacklist - any deployments within a blacklisted namespace will not -be touched. +kube-monkey can be configured with a list of namespaces to blacklist - any deployments within a blacklisted namespace will not be touched. ## Opting-In to Chaos -kube-monkey works on an opt-in model and will only schedule terminations for Deployments that have explicitly agreed -to have their pods terminated by kube-monkey. +kube-monkey works on an opt-in model and will only schedule terminations for Deployments that have explicitly agreed to have their pods terminated by kube-monkey. Opt-in is done by setting the following labels on a Kubernetes Deployment: @@ -66,18 +60,18 @@ At termination time, kube-monkey will: Clone the repository and build the container. -``` +```bash go get github.com/asobti/kube-monkey cd $GOPATH/src/github.com/asobti/kube-monkey make container ``` ## Configuring -kube-monkey is configured by a toml file placed at `/etc/kube-monkey/config.toml`. -Configuration keys and descriptions can be found in [`config/param/param.go`](https://github.com/asobti/kube-monkey/blob/master/config/param/param.go) +kube-monkey is configured by a toml file placed at `/etc/kube-monkey/config.toml` and expects the configmap to exist before deployment. -#### Example config file +Configuration keys and descriptions can be found in [`config/param/param.go`](https://github.com/asobti/kube-monkey/blob/master/config/param/param.go) +#### Example config.toml file ```toml [kubemonkey] dry_run = true # Terminations are only logged @@ -89,8 +83,11 @@ blacklisted_namespaces = ["kube-system"] # Critical deployments live here ## Deploying -Run kube-monkey as a Deployment within the Kubernetes cluster, in a namespace that has permissions to kill Pods -in other namespaces (eg. `kube-system`). +First deploy the expected `kube-monkey-config-map` configmap in the `kube-system` namespace. Make sure to define the keyname as `config.toml` + +For example `kubectl create configmap km-config --from-file=config.toml=km-config.toml` + +Run kube-monkey as a Deployment within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`). See dir [`examples/`](https://github.com/asobti/kube-monkey/tree/master/examples) for example Kubernetes yaml files. From ed47ef1019b6cede191309306e25aeff64789187 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Sun, 29 Oct 2017 17:50:47 -0400 Subject: [PATCH 03/13] Clarifying CM Namespace Technically shouldn't force the user to use the kube-monkey namespace --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 53610e1e..a2f3ae5c 100644 --- a/README.md +++ b/README.md @@ -83,11 +83,11 @@ blacklisted_namespaces = ["kube-system"] # Critical deployments live here ## Deploying -First deploy the expected `kube-monkey-config-map` configmap in the `kube-system` namespace. Make sure to define the keyname as `config.toml` +1. First deploy the expected `kube-monkey-config-map` configmap in the namespace you intend to run kube-monkey in (for example, the `kube-system` namespace). Make sure to define the keyname as `config.toml` -For example `kubectl create configmap km-config --from-file=config.toml=km-config.toml` +> For example `kubectl create configmap km-config --from-file=config.toml=km-config.toml` -Run kube-monkey as a Deployment within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`). +2. Run kube-monkey as a Deployment within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`). See dir [`examples/`](https://github.com/asobti/kube-monkey/tree/master/examples) for example Kubernetes yaml files. From 46c90e57fe823f93c00927272cdc5a529be4ba5f Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Sun, 29 Oct 2017 17:51:28 -0400 Subject: [PATCH 04/13] Including apiserver overload info --- README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/README.md b/README.md index a2f3ae5c..725ff54a 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,17 @@ spec: [... omitted ...] ``` +### Overriding the apiserver +#### Use cases: +* Since client-go does not support [cluster dns](https://github.com/kubernetes/client-go/blob/master/rest/config.go#L336) explicitly with a `// TODO: switch to using cluster DNS.` note in the code, you may need to override the apiserver. +* If you are running an unauthenticated system, you may need to force the http apiserver enpoint. + +#### To override the apiserver specify in the config.toml file +```toml +[kubernetes] +host="https://your-apiserver-url.com" +``` + ## How kube-monkey works #### Scheduling time From 0390bc12be6cbff6915c9533aa906759fb4506ee Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Mon, 30 Oct 2017 09:29:10 -0400 Subject: [PATCH 05/13] Todo --> Contribute --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 725ff54a..ccd463bd 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ kube-monkey is built using v1.5 of [kubernetes/client-go](https://github.com/kub [Compatibility Matrix](https://github.com/kubernetes/client-go#compatibility-matrix) to see which versions of Kubernetes are compatible. -## To do +## Ways to contribute -- Add tests +- Add unit [tests](https://golang.org/pkg/testing/) - Use a logging library like [glog](https://github.com/golang/glog) From 73d6e3ee97ab9a01f8f5943d17953e1e4c6bac15 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Thu, 9 Nov 2017 17:00:49 -0500 Subject: [PATCH 06/13] Updated after logging merge --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ccd463bd..d03b6fc4 100644 --- a/README.md +++ b/README.md @@ -111,4 +111,4 @@ versions of Kubernetes are compatible. ## Ways to contribute - Add unit [tests](https://golang.org/pkg/testing/) -- Use a logging library like [glog](https://github.com/golang/glog) +- Standardize/establish logging convention for [glog](https://github.com/golang/glog) usage From bdebdee3a7398bdcfc0807fbcf7696a4ff917398 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Wed, 15 Nov 2017 10:59:28 -0500 Subject: [PATCH 07/13] Spelling Fix --- kubernetes/kubernetes.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kubernetes/kubernetes.go b/kubernetes/kubernetes.go index 7fdda8c9..13b25b69 100644 --- a/kubernetes/kubernetes.go +++ b/kubernetes/kubernetes.go @@ -15,7 +15,7 @@ func NewInClusterClient() (*kube.Clientset, error) { } if apiserverHost, override := cfg.ClusterAPIServerHost(); override { - fmt.Printf("API server host overriden to: %s\n", apiserverHost) + fmt.Printf("API server host overridden to: %s\n", apiserverHost) config.Host = apiserverHost } From d824e55a645e39a1345ca7e63fae09553648c526 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Wed, 20 Dec 2017 10:47:23 -0500 Subject: [PATCH 08/13] Deployment identifier example https://github.com/asobti/kube-monkey/issues/22 --- README.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d03b6fc4..2f29e607 100644 --- a/README.md +++ b/README.md @@ -35,11 +35,33 @@ spec: metadata: labels: kube-monkey/enabled: enabled - kube-monkey/identifier: monkey-victim + kube-monkey/identifier: monkey-victim-pods kube-monkey/mtbf: '2' [... omitted ...] ``` +For newer versions of kubernetes you may need to add the labels to the deployment metadata as well. + +```yaml +--- +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: monkey-victim + namespace: app-namespace + labels: + kube-monkey/enabled: enabled + kube-monkey/identifier: monkey-victim + kube-monkey/mtbf: '2' +spec: + template: + metadata: + labels: + kube-monkey/enabled: enabled + kube-monkey/identifier: monkey-victim +[... omitted ...] +``` + ### Overriding the apiserver #### Use cases: * Since client-go does not support [cluster dns](https://github.com/kubernetes/client-go/blob/master/rest/config.go#L336) explicitly with a `// TODO: switch to using cluster DNS.` note in the code, you may need to override the apiserver. From dcfff066b5ad7089f5acab24229606b8f8fb7499 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Wed, 27 Dec 2017 17:07:03 -0500 Subject: [PATCH 09/13] Add timezone to example config Allow people to copy pasta ;) https://github.com/asobti/kube-monkey/issues/23 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2f29e607..843b8217 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,7 @@ run_hour = 8 # Run scheduling at 8am on weekdays start_hour = 10 # Don't schedule any pod deaths before 10am end_hour = 16 # Don't schedule any pod deaths after 4pm blacklisted_namespaces = ["kube-system"] # Critical deployments live here +time_zone = "America/New_York" # Set tzdata timezone example. Note the field is time_zone not timezone ``` ## Deploying From a75c85d7878c75bac6f28c2c67af53ce25ad8ee5 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Wed, 27 Dec 2017 17:34:06 -0500 Subject: [PATCH 10/13] Update Logging Conventions Only relevant after https://github.com/asobti/kube-monkey/pull/24 --- README.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d1e7fa10..ef3440fd 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,22 @@ See dir [`examples/`](https://github.com/asobti/kube-monkey/tree/master/examples kube-monkey uses glog and supports all command-line features for glog. To specify a custom v level or a custom log directory on the pod, see `args: ["-v=5", "-log_dir=/path/to/custom/log"]` in the [example deployment file](https://github.com/asobti/kube-monkey/tree/master/examples/deployment.yaml) +> **Standardized glog levels `grep -r V\([0-9]\) *`** +> +> L0: None +> +> L1: Highest Level current status info and Errors with Terminations +> +> L2: Successful terminations +> +> L3: More detailed schedule status info +> +> L4: Debugging verbose schedule and config info +> +> L5: Auto-resolved inconsequential issues + +More resources: See the [k8 logging page](https://kubernetes.io/docs/concepts/cluster-administration/logging/) suggesting [community conventions for logging severity](https://github.com/kubernetes/community/blob/master/contributors/devel/logging.md) + ## Compatibility with Kubernetes kube-monkey is built using v1.5 of [kubernetes/client-go](https://github.com/kubernetes/client-go). Refer to the @@ -138,4 +154,3 @@ versions of Kubernetes are compatible. ## Ways to contribute - Add unit [tests](https://golang.org/pkg/testing/) -- Standardize/establish logging convention for [glog](https://github.com/golang/glog) usage From 801e7b0e18614947b24cac2de1f3bad5fafb6c10 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Mon, 1 Jan 2018 19:36:03 -0500 Subject: [PATCH 11/13] Upgraded client-go https://github.com/asobti/kube-monkey/pull/25 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ef3440fd..bd67eae8 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ More resources: See the [k8 logging page](https://kubernetes.io/docs/concepts/cl ## Compatibility with Kubernetes -kube-monkey is built using v1.5 of [kubernetes/client-go](https://github.com/kubernetes/client-go). Refer to the +kube-monkey is built using v6.0 of [kubernetes/client-go](https://github.com/kubernetes/client-go). Refer to the [Compatibility Matrix](https://github.com/kubernetes/client-go#compatibility-matrix) to see which versions of Kubernetes are compatible. From 0d348f0d6be89ec00d18cdafc82b1c1a53a98f96 Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Mon, 1 Jan 2018 19:42:03 -0500 Subject: [PATCH 12/13] Add new contribution types Possible after templatizing in #27 and reference issue #10 --- README.md | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index bd67eae8..97f57a37 100644 --- a/README.md +++ b/README.md @@ -3,21 +3,21 @@ kube-monkey is an implementation of [Netflix's Chaos Monkey](https://github.com/ --- -kube-monkey runs at a pre-configured hour (`run_hour`, defaults to 8am) on weekdays, and builds a schedule of deployments that will face a random Pod death sometime during the same day. The time-range during the day when the random pod Death might occur is configurable and defaults to 10am to 4pm. +kube-monkey runs at a pre-configured hour (`run_hour`, defaults to 8am) on weekdays, and builds a schedule of k8 apps (kubernetes kinds) that will face a random Pod death sometime during the same day. The time-range during the day when the random pod Death might occur is configurable and defaults to 10am to 4pm. -kube-monkey can be configured with a list of namespaces to blacklist - any deployments within a blacklisted namespace will not be touched. +kube-monkey can be configured with a list of namespaces to blacklist - any k8 apps within a blacklisted namespace will not be touched. ## Opting-In to Chaos -kube-monkey works on an opt-in model and will only schedule terminations for Deployments that have explicitly agreed to have their pods terminated by kube-monkey. +kube-monkey works on an opt-in model and will only schedule terminations for k8 apps that have explicitly agreed to have their pods terminated by kube-monkey. -Opt-in is done by setting the following labels on a Kubernetes Deployment: +Opt-in is done by setting the following labels on a Kubernetes k8 app: **`kube-monkey/enabled`**: Set to **`"enabled"`** to opt-in to kube-monkey -**`kube-monkey/mtbf`**: Mean time between failure (in days). For example, if set to **`"3"`**, the Deployment can expect to have a Pod +**`kube-monkey/mtbf`**: Mean time between failure (in days). For example, if set to **`"3"`**, the k8 app can expect to have a Pod killed approximately every third weekday. -**`kube-monkey/identifier`**: A unique identifier for the deployment (eg. the deployment's name). This is used to identify the pods -that belong to a Deployment as Pods inherit labels from their Deployment. +**`kube-monkey/identifier`**: A unique identifier for the k8 app (eg. the k8 app's name). This is used to identify the pods +that belong to a k8 app as Pods inherit labels from their k8 app. **`kube-monkey/kill-all`**: Set this label's value to `"kill-all"` if you want kube-monkey to kill ALL of your pods. Default behavior in the absence of this label is to kill only ONE pod. **Use this label carefully.** @@ -40,7 +40,7 @@ spec: [... omitted ...] ``` -For newer versions of kubernetes you may need to add the labels to the deployment metadata as well. +For newer versions of kubernetes you may need to add the labels to the k8 app metadata as well. ```yaml --- @@ -78,15 +78,15 @@ host="https://your-apiserver-url.com" #### Scheduling time Scheduling happens once a day on Weekdays - this is when a schedule for terminations for the current day is generated. During scheduling, kube-monkey will: -1. Generate a list of eligible deployments (deployments that have opted-in and are not blacklisted) -2. For each eligible deployment, flip a biased coin (bias determined by `kube-monkey/mtbf`) to determine if a pod for that deployment should be killed today +1. Generate a list of eligible k8 apps (k8 apps that have opted-in and are not blacklisted) +2. For each eligible k8 app, flip a biased coin (bias determined by `kube-monkey/mtbf`) to determine if a pod for that k8 app should be killed today 3. For each victim, calculate a random time when a pod will be killed #### Termination time -This is the randomly generated time during the day when a victim Deployment will have a pod killed. +This is the randomly generated time during the day when a victim k8 app will have a pod killed. At termination time, kube-monkey will: -1. Check if the deployment is still eligible (has not opted-out or been blacklisted since scheduling) -2. Get a list of running pods for the deployment +1. Check if the k8 app is still eligible (has not opted-out or been blacklisted since scheduling) +2. Get a list of running pods for the k8 app 3. Select one random pod and delete it ## Building @@ -100,7 +100,7 @@ make container ``` ## Configuring -kube-monkey is configured by a toml file placed at `/etc/kube-monkey/config.toml` and expects the configmap to exist before deployment. +kube-monkey is configured by a toml file placed at `/etc/kube-monkey/config.toml` and expects the configmap to exist before the kubemonkey deployment. Configuration keys and descriptions can be found in [`config/param/param.go`](https://github.com/asobti/kube-monkey/blob/master/config/param/param.go) @@ -111,7 +111,7 @@ dry_run = true # Terminations are only logged run_hour = 8 # Run scheduling at 8am on weekdays start_hour = 10 # Don't schedule any pod deaths before 10am end_hour = 16 # Don't schedule any pod deaths after 4pm -blacklisted_namespaces = ["kube-system"] # Critical deployments live here +blacklisted_namespaces = ["kube-system"] # Critical apps live here time_zone = "America/New_York" # Set tzdata timezone example. Note the field is time_zone not timezone ``` @@ -121,7 +121,7 @@ time_zone = "America/New_York" # Set tzdata timezone example. Note the > For example `kubectl create configmap km-config --from-file=config.toml=km-config.toml` -2. Run kube-monkey as a Deployment within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`). +2. Run kube-monkey as a k8 app within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`). See dir [`examples/`](https://github.com/asobti/kube-monkey/tree/master/examples) for example Kubernetes yaml files. @@ -154,3 +154,8 @@ versions of Kubernetes are compatible. ## Ways to contribute - Add unit [tests](https://golang.org/pkg/testing/) +- Support more k8 types + - ~~deployments~~ + - statefulsets + - dameonsets + - etc From c70db49343f98acd4116fb18441c51f5ecd9ea1b Mon Sep 17 00:00:00 2001 From: Spellchaser Date: Tue, 2 Jan 2018 19:13:45 -0500 Subject: [PATCH 13/13] Added statefulsets See https://github.com/asobti/kube-monkey/pull/29 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 97f57a37..564d7a75 100644 --- a/README.md +++ b/README.md @@ -156,6 +156,6 @@ versions of Kubernetes are compatible. - Add unit [tests](https://golang.org/pkg/testing/) - Support more k8 types - ~~deployments~~ - - statefulsets + - ~~statefulsets~~ - dameonsets - etc