Skip to content

Commit

Permalink
Add new contribution types
Browse files Browse the repository at this point in the history
Possible after templatizing in asobti#27 and reference issue #10
  • Loading branch information
Aergonus authored Jan 2, 2018
1 parent 801e7b0 commit 0d348f0
Showing 1 changed file with 21 additions and 16 deletions.
37 changes: 21 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,21 @@ kube-monkey is an implementation of [Netflix's Chaos Monkey](https://github.com/

---

kube-monkey runs at a pre-configured hour (`run_hour`, defaults to 8am) on weekdays, and builds a schedule of deployments that will face a random Pod death sometime during the same day. The time-range during the day when the random pod Death might occur is configurable and defaults to 10am to 4pm.
kube-monkey runs at a pre-configured hour (`run_hour`, defaults to 8am) on weekdays, and builds a schedule of k8 apps (kubernetes kinds) that will face a random Pod death sometime during the same day. The time-range during the day when the random pod Death might occur is configurable and defaults to 10am to 4pm.

kube-monkey can be configured with a list of namespaces to blacklist - any deployments within a blacklisted namespace will not be touched.
kube-monkey can be configured with a list of namespaces to blacklist - any k8 apps within a blacklisted namespace will not be touched.

## Opting-In to Chaos

kube-monkey works on an opt-in model and will only schedule terminations for Deployments that have explicitly agreed to have their pods terminated by kube-monkey.
kube-monkey works on an opt-in model and will only schedule terminations for k8 apps that have explicitly agreed to have their pods terminated by kube-monkey.

Opt-in is done by setting the following labels on a Kubernetes Deployment:
Opt-in is done by setting the following labels on a Kubernetes k8 app:

**`kube-monkey/enabled`**: Set to **`"enabled"`** to opt-in to kube-monkey
**`kube-monkey/mtbf`**: Mean time between failure (in days). For example, if set to **`"3"`**, the Deployment can expect to have a Pod
**`kube-monkey/mtbf`**: Mean time between failure (in days). For example, if set to **`"3"`**, the k8 app can expect to have a Pod
killed approximately every third weekday.
**`kube-monkey/identifier`**: A unique identifier for the deployment (eg. the deployment's name). This is used to identify the pods
that belong to a Deployment as Pods inherit labels from their Deployment.
**`kube-monkey/identifier`**: A unique identifier for the k8 app (eg. the k8 app's name). This is used to identify the pods
that belong to a k8 app as Pods inherit labels from their k8 app.
**`kube-monkey/kill-all`**: Set this label's value to `"kill-all"` if you want kube-monkey to kill ALL of your pods. Default behavior in the absence of this label is to kill only ONE pod. **Use this label carefully.**


Expand All @@ -40,7 +40,7 @@ spec:
[... omitted ...]
```

For newer versions of kubernetes you may need to add the labels to the deployment metadata as well.
For newer versions of kubernetes you may need to add the labels to the k8 app metadata as well.

```yaml
---
Expand Down Expand Up @@ -78,15 +78,15 @@ host="https://your-apiserver-url.com"
#### Scheduling time
Scheduling happens once a day on Weekdays - this is when a schedule for terminations for the current day is generated.
During scheduling, kube-monkey will:
1. Generate a list of eligible deployments (deployments that have opted-in and are not blacklisted)
2. For each eligible deployment, flip a biased coin (bias determined by `kube-monkey/mtbf`) to determine if a pod for that deployment should be killed today
1. Generate a list of eligible k8 apps (k8 apps that have opted-in and are not blacklisted)
2. For each eligible k8 app, flip a biased coin (bias determined by `kube-monkey/mtbf`) to determine if a pod for that k8 app should be killed today
3. For each victim, calculate a random time when a pod will be killed

#### Termination time
This is the randomly generated time during the day when a victim Deployment will have a pod killed.
This is the randomly generated time during the day when a victim k8 app will have a pod killed.
At termination time, kube-monkey will:
1. Check if the deployment is still eligible (has not opted-out or been blacklisted since scheduling)
2. Get a list of running pods for the deployment
1. Check if the k8 app is still eligible (has not opted-out or been blacklisted since scheduling)
2. Get a list of running pods for the k8 app
3. Select one random pod and delete it

## Building
Expand All @@ -100,7 +100,7 @@ make container
```

## Configuring
kube-monkey is configured by a toml file placed at `/etc/kube-monkey/config.toml` and expects the configmap to exist before deployment.
kube-monkey is configured by a toml file placed at `/etc/kube-monkey/config.toml` and expects the configmap to exist before the kubemonkey deployment.

Configuration keys and descriptions can be found in [`config/param/param.go`](https://github.com/asobti/kube-monkey/blob/master/config/param/param.go)

Expand All @@ -111,7 +111,7 @@ dry_run = true # Terminations are only logged
run_hour = 8 # Run scheduling at 8am on weekdays
start_hour = 10 # Don't schedule any pod deaths before 10am
end_hour = 16 # Don't schedule any pod deaths after 4pm
blacklisted_namespaces = ["kube-system"] # Critical deployments live here
blacklisted_namespaces = ["kube-system"] # Critical apps live here
time_zone = "America/New_York" # Set tzdata timezone example. Note the field is time_zone not timezone
```

Expand All @@ -121,7 +121,7 @@ time_zone = "America/New_York" # Set tzdata timezone example. Note the

> For example `kubectl create configmap km-config --from-file=config.toml=km-config.toml`
2. Run kube-monkey as a Deployment within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`).
2. Run kube-monkey as a k8 app within the Kubernetes cluster, in a namespace that has permissions to kill Pods in other namespaces (eg. `kube-system`).

See dir [`examples/`](https://github.com/asobti/kube-monkey/tree/master/examples) for example Kubernetes yaml files.

Expand Down Expand Up @@ -154,3 +154,8 @@ versions of Kubernetes are compatible.
## Ways to contribute

- Add unit [tests](https://golang.org/pkg/testing/)
- Support more k8 types
- ~~deployments~~
- statefulsets
- dameonsets
- etc

0 comments on commit 0d348f0

Please sign in to comment.