diff --git a/AUTHORS.md b/AUTHORS.md deleted file mode 100644 index 6e62a3b8b..000000000 --- a/AUTHORS.md +++ /dev/null @@ -1,9 +0,0 @@ -# Credits - -## Development Lead - -- Raúl Sevilla Cañavate [rsevilla87](https://github.com/rsevilla87) - -## Contributors - -None yet. Why not be the first? diff --git a/AUTHORS.md b/AUTHORS.md new file mode 120000 index 000000000..9921bebcb --- /dev/null +++ b/AUTHORS.md @@ -0,0 +1 @@ +docs/AUTHORS.md \ No newline at end of file diff --git a/README.md b/README.md index 1bd45d312..835a80b2f 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,12 @@ + + [![Build Status](https://github.com/cloud-bulldozer/kube-burner/workflows/Go/badge.svg?branch=master)](https://github.com/cloud-bulldozer/kube-burner/actions?query=workflow%3AGo) [![Go Report Card](https://goreportcard.com/badge/github.com/cloud-bulldozer/kube-burner)](https://goreportcard.com/report/github.com/cloud-bulldozer/kube-burner) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) - - ---- -- [What's this?](#whats-this) -- [Quick start](#quick-start) -- [Building](#building) -- [Getting started](#getting-started) -- [Configuration](#configuration) - - [Objects](#objects) - - [Job types](#job-types) - - [Injected variables](#injected-variables) - - [Metrics profile](#metrics-profile) - - [Job Summary](#job-summary) - - [Indexers](#indexers) - - [Measurements](#measurements) - - [Pod latency](#pod-latency) - - [Pprof collection](#pprof-collection) -- [Contributing to kube-burner](#contributing-to-kube-burner) - - [Requirements](#requirements) +# What is Kube-burner? - -## What's this? - -Kube-burner is a tool aimed at stressing kubernetes clusters. An overview of its behaviour can be summarized with these three steps: +Kube-burner is a tool aimed at stressing kubernetes clusters. The main functionallity it provides can be summareized these three steps: - Create/delete the objects declared in the jobs. - Collect desired on-cluster prometheus metrics. @@ -33,459 +14,14 @@ Kube-burner is a tool aimed at stressing kubernetes clusters. An overview of its [![asciicast](https://asciinema.org/a/KksoK5voK3al1FuOza89t1JAp.svg)](https://asciinema.org/a/KksoK5voK3al1FuOza89t1JAp) +## Documentation + +Documentation is available at https://kube-burner.readthedocs.io/ -## Quick start +## Downloading Kube-burner In case you want to start tinkering with `kube-burner` now: -- You can find the binaries in the [releases section of this repository](https://github.com/cloud-bulldozer/kube-burner/releases). +- You can find the binaries in the [releases section of the repository](https://github.com/cloud-bulldozer/kube-burner/releases). - There's also a container image available at [quay](https://quay.io/repository/cloud-bulldozer/kube-burner?tab=tags). -- A valid example of a configuration file can be found at [./examples/cfg.yml](./examples/cfg.yml) - - -## Building - -To build kube-burner just execute `make build`, once finished `kube-burner`'s binary should be available at `./bin/kube-burner` - -```console -$ make build -building kube-burner 0.1.0 -GOPATH=/home/rsevilla/go -CGO_ENABLED=0 go build -v -mod vendor -ldflags "-X github.com/cloud-bulldozer/kube-burner/version.GitCommit=d91c8cc35cb458a4b80a5050704a51c7c6e35076 -X github.com/cloud-bulldozer/kube-burner/version.BuildDate=2020-08-19-19:10:09 -X github.com/cloud-bulldozer/kube-burner/version.GitBranch=master" -o bin/kube-burner -``` - -## Getting started - -kube-burner is basically a binary client with currently the following optionsn - -```console -./bin/kube-burner help -Kube-burner 🔥 - -Tool aimed at stressing a kubernetes cluster by creating or deleting lot of objects. - -Usage: - kube-burner [command] - -Available Commands: - completion Generates completion scripts for bash shell - destroy Destroy old namespaces labeled with the given UUID. - help Help about any command - index Index metrics from the given time range - init Launch benchmark - version Print the version number of kube-burner - -Flags: - -h, --help help for kube-burner - -Use "kube-burner [command] --help" for more information about a command. -``` - -- The **init** option supports the following flags: - - config: Path or URL to a valid configuration file. - - log-level: Logging level. Default `info` - - prometheus-url: Prometheus full URL. i.e. `https://prometheus-k8s-openshift-monitoring.apps.rsevilla.stress.mycluster.example.com` - - metrics-profile: Path to a valid metrics profile file. Default `metrics.yaml` - - token: Prometheus Bearer token. - - username: Prometheus username for basic authentication. - - password: Prometheus password for basic authentication. - - skip-tls-verify: Skip TLS verification for prometheus. Default `true` - - step: Prometheus step size. Default `30s` - - UUID: Benchmark UUID. - -**Note**: Both basic authentication and Bearer authentication need credentials able to query given Prometheus API. - -With the above, triggering kube-burner would be as simple as: - -```console -$ kube-burner init -c cfg.yml -u https://prometheus-k8s-openshift-monitoring.apps.rsevilla.stress.mycluster.example.com -t ${token} --uuid 67f9ec6d-6a9e-46b6-a3bb-065cde988790` -``` - -or -```console -$ kube-burner init -c http://web.domain.com:8080/cfg.yml -u https://prometheus-k8s-openshift-monitoring.apps.rsevilla.stress.mycluster.example.com -t ${token} --uuid 67f9ec6d-6a9e-46b6-a3bb-065cde988790` -``` - - -If you have no interest in collecting prometheus metrics, kube-burner can also be launched w/o any prometheus endpoint. - -```console -$ kube-burner init -c cfg.yml --uuid 67f9ec6d-6a9e-46b6-a3bb-065cde988790` -``` - -- The **index** option can be used to collect and index the metrics from a given time range. This option supports the same flags as the **init** option. The time range is given by: - - start: Epoch start time. Defaults to one hour before the current time. - - End: Epoch end time. Defaults to the current time. - -- The **destroy** option requires the above `config` and `UUID` flags to destroy all namespaces labeled with `kube-burner-uuid=`. - -- The **completion** option generates bash a completion script that can be imported with: - - `. <(kube-burner completion)` - - `kube-burner completion > /etc/bash_completion.d/kube-burner` - -## Configuration - -All the magic `kube-burner` does is described in the configuration file. This file is written in YAML format and has the following sections: - -* global: This section describes the global job configuration, it holds the following parameters: - -| Option | Description | Type | Example | Default | -|------------------|----------------------------------------------------------------------------------------------------------|----------------|----------------|-------------| -| kubeconfig | Points to a valid kubeconfig file. Can be omitted if using the KUBECONFIG environment variable | String | ~/mykubeconfig | in-cluster | | -| writeToFile | Whether to dump collected metrics to files | Boolean | true | true | -| metricsDirectory | Directory where collected metrics will be dumped into. It will be created if it doesn't exist previously | String | ./metrics | ./collected-metrics | -| measurements | List of measurements. Detailed in the [measurements section](#Measurements) | List | - | [] | -| indexerConfig | Holds the indexer configuration. Detailed in the [indexers section](#Indexers) | Object | - | - | - -* jobs: This section contains a list of jobs that `kube-burner` will execute. Each job can hold the following parameters. - -| Option | Description | Type | Example | Default | -|----------------------|----------------------------------------------------------------------------------|---------|----------|---------| -| name | Job name | String | myjob | "" | -| jobType | Type of job to execute. More details at [job types](#job-types) | string | create | create | -| jobIterations | How many times to execute the job | Integer | 10 | 0 | -| namespace | Namespace base name to use | String | firstjob | "" | -| namespacedIterations | Whether to create a namespace per job iteration | Boolean | true | true | -| cleanup | Cleanup clean up old namespaces | Boolean | true | true | -| podWait | Wait for all pods to be running before moving forward to the next job iteration | Boolean | true | true | -| waitWhenFinished | Wait for all pods to be running when all iterations are completed | Boolean | true | false | -| maxWaitTimeout | Maximum wait timeout in seconds. (If podWait is enabled this timeout will be reseted with each iteration) | Integer | 1h | 12h | -| waitFor | List containing the objects Kind wait for. Wait for all if empty | List | ["Deployment", "Build", "DaemonSet"]| [] | -| jobIterationDelay | How long to wait between each job iteration | Duration| 2s | 0s | -| jobPause | How long to pause after finishing the job | Duration| 10s | 0s | -| qps | Limit object creation queries per second | Integer | 25 | 0 | -| burst | Maximum burst for throttle | Integer | 50 | 0 | -| objects | List of objects the job will create. Detailed on the [objects section](#objects) | List | - | [] | -| verifyObjects | Verify object count after running each job. Return code will be 1 if failed | Boolean | true | true | -| errorOnVerify | Exit with rc 1 before indexing when objects verification fails | Boolean | true | false | - - -A valid example of a configuration file can be found at [./examples/cfg.yml](./examples/cfg.yml) - -### Objects - -The objects created by `kube-burner` are rendered using the default golang's [template library](https://golang.org/pkg/text/template/). -Each object element supports the following parameters: - -| Option | Description | Type | Example | Default | -|----------------------|-------------------------------------------------------------------|---------|-----------------------------------------------------|---------| -| objectTemplate | Object template file or URL | String | deployment.yml or https://domain.com/deployment.yml | "" | -| replicas | How replicas of this object to create per job iteration | Integer | 10 | - | -| inputVars | Map of arbitrary input variables to inject to the object template | Object | - | - | - -It's important to note that all objects created by kube-burner are labeled with. `kube-burner-uuid=,kube-burner-job=,kube-burner-index=` - - -### Job types - -kube-burner support two types of jobs with different parameters each. The default job type is __create__. Which basically creates objects as described in the section [objects](#objects). - -The other type is __delete__, this type of job deletes objects described in the objects list. Using delete as job type the objects list would have the following structure. - -```yaml -objects: -- kind: Deployment - labelSelector: {kube-burner-job: cluster-density} - apiVersion: apps/v1 - -- kind: Secret - labelSelector: {kube-burner-job: cluster-density} -``` -Where: -- kind: Object kind of the k8s object to delete. -- labelSelector: Map with the labelSelector. -- apiVersion: API version from the k8s object. - -As mentioned previously, all objects created by kube-burner are labeled with `kube-burner-uuid=,kube-burner-job=,kube-burner-index=`. Thanks to this we could design a workload with one job to create objects and another one able to remove the objects created by the previous - -```yaml -jobs: -- name: create-objects - namespace: job-namespace - jobIterations: 100 - objects: - - objectTemplate: deployment.yml - replicas: 10 - - - objectTemplate: service.yml - replicas: 10 - -- name: remove-objects - jobType: delete - objects: - - kind: Deployment - labelSelector: {kube-burner-job: create-objects} - apiVersion: apps/v1 - - - kind: Secret - labelSelector: {kube-burner-job: create-objects} -``` - -This job type supports the some of the same parameters as the create job type: -- **waitForDeletion**: Wait for objects to be deleted before finishing the job. Defaults to true -- name -- qps -- burst -- jobPause - -### Injected variables - -All object templates are injected a series of variables by default: - -- Iteration: Job iteration number. -- Replica: Object replica number. Keep in mind that this number is reset to 1 with each job iteration. -- JobName: Job name. -- UUID: Benchmark UUID. - -In addition, you can also inject arbitrary variables with the option **inputVars** from the objectTemplate object: - -```yaml - - objectTemplate: service.yml - replicas: 2 - inputVars: - port: 80 - targetPort: 8080 -``` - -The following code snippet shows an example of a k8s service using these variables: - - -```yaml -apiVersion: v1 -kind: Service -metadata: - name: sleep-app-{{ .Iteration }}-{{ .Replica }} - labels: - name: my-app-{{ .Iteration }}-{{ .Replica }} -spec: - selector: - app: sleep-app-{{ .Iteration }}-{{ .Replica }} - ports: - - name: serviceport - protocol: TCP - port: "{{ .port }}" - targetPort: "{{ .targetPort }}" - type: ClusterIP -``` - -It's worth to say that you can also use [golang template semantics](https://golang.org/pkg/text/template/) in your *objectTemplate* files. - -```yaml -kind: ImageStream -apiVersion: image.openshift.io/v1 -metadata: - name: {{.prefix}}-{{.Replica}} -spec: -{{ if .image }} - dockerImageRepository: {{.image}} -{{ end }} -``` - -### Metrics profile - -The metrics-profile flag points to a YAML or URL of a file containing a list of the prometheus queries kube-burner will collect for each job. -As soon one of job finishes, `kube-burner` makes a range query for each query described in this file, and indexes it in the index configured by the parameter `defaultIndex`. -We can use the parameter `indexName` in a metrics-profile file to make `kube-burner` to index the resulting metrics to a different index. -An example of a valid metrics profile file is shown below: -The parameter **metricName** is added to the indexed documents, it will allow us to identify documents from a certain query more easily. - -```yaml -metrics: - - query: irate(process_cpu_seconds_total{job=~".*(crio|etcd|controller-manager|apiserver|scheduler).*"}[2m]) - metricName: controlPlaneCPU - - - query: process_resident_memory_bytes{job=~".*(crio|etcd|controller-manager|apiserver|scheduler).*"} - metricName: controlPlaneMemory - - - query: sum(irate(node_cpu_seconds_total[2m])) by (mode,instance) - metricName: nodeCPU -``` - - -It's also possible to execute instant queries from kube-burner by adding the flag instant to the desired metric. These kind of queries are useful to get only one sample for a static metric such as the number of nodes or the kube-apiserver version. - -```yaml -metrics: - - query: kube_node_role - metricName: nodeRoles - instant: true -``` - -### Job Summary - -In case indexing is enabled, at the end of each job, a document holding the job summary is indexed. This is useful to identify the parameters the job was executed with: - -This document looks like: - -```json -{ - "timestamp": "2020-11-13T13:55:31.654185032+01:00", - "uuid": "bdb7584a-d2cd-4185-8bfa-1387cc31f99e", - "metricName": "jobSummary", - "elapsedTime": 8.768932955, - "jobConfig": { - "jobIterations": 10, - "jobIterationDelay": 0, - "jobPause": 0, - "name": "kubelet-density", - "objects": [ - { - "objectTemplate": "templates/pod.yml", - "replicas": 1, - "inputVars": { - "containerImage": "gcr.io/google_containers/pause-amd64:3.0" - } - } - ], - "jobType": "create", - "qps": 5, - "burst": 5, - "namespace": "kubelet-density", - "waitFor": null, - "maxWaitTimeout": 43200000000000, - "waitForDeletion": true, - "podWait": false, - "waitWhenFinished": true, - "cleanup": true, - "namespacedIterations": false, - "verifyObjects": true, - "errorOnVerify": false - } -} -``` - -### Indexers - -`kube-burner` is able to **index the collected prometheus metrics** into a given Indexer. -The indexer configuration is described in the `indexerConfig` section and can be configured with the following parameters: - - -| Option | Description | Type | Example | Default | -|----------------------|-----------------------|----------|------------|---------| -| enabled | Enable indexing | Boolean | true | false | -| type | Type of indexer | String | elastic | "" | - - -The following indexers are currently supported: - -- `elastic`: Index documents in Elasticsearch 7 instances. - -In addition, each indexer has its own configuration parameters. - ----- - -The `elastic` indexer is configured by the parameters below: - -| Option | Description | Type | Example | Default | -|----------------------|---------------------------------------------------|-------------|------------------------------------------|---------| -| esServers | List of ES instances | List | [https://elastic.apps.rsevilla.org:9200] | "" | -| defaultIndex | Default index to send the prometheus metrics into | String | kube-burner | "" | -| insecureSkipVerify | TLS certificate verification | Boolean | true | false | - -**Note**: It's possible to index documents in an authenticated ES instance using the notation `http(s)://[username]:[password]@[address]:[port]` in the *esServers* parameter. - -### Measurements - -Apart from prometheus metrics collection, `kube-burner` allows to get further metrics using other mechanisms or data sources such as the -own kubernetes API, these mechanisms are called measurements. -Measurements are enabled in the measurements section of the configuration file. This section contains a list of measurements with their options. -'kube-burner' supports the following measurements so far: - -#### Pod latency - -Collects latencies from the different pod startup phases, these **latency metrics are in ms**. Can be enabled with: - -```yaml - measurements: - - name: podLatency - esIndex: kube-burner-podlatency -``` - -This measurement sends its metrics to the index configured by *esIndex*. The metrics collected are pod latency histograms and pod latency quantiles P99, P95 and P50. - -Pod latency sample: -```json -{ - "timestamp": "2020-11-15T20:28:59.598727718Z", - "schedulingLatency": 4, - "initializedLatency": 20, - "containersReadyLatency": 2997, - "podReadyLatency": 2997, - "metricName": "podLatencyMeasurement", - "jobName": "kubelet-density", - "uuid": "c40b4346-7af7-4c63-9ab4-aae7ccdd0616", - "namespace": "kubelet-density", - "podName": "kubelet-density-13", - "jobName": "kube-burner-job" -} -``` - -Pod latency quantile sample: - -```json -{ - "quantileName": "podReady", - "uuid": "23c0b5fd-c17e-4326-a389-b3aebc774c82", - "P99": 3774, - "P95": 3510, - "P50": 2897, - "max": 3774, - "avg": 2876.3, - "timestamp": "2020-11-15T22:26:51.553221077+01:00", - "metricName": "podLatencyQuantilesMeasurement", - "jobName": "kubelet-density" -}, -{ - "quantileName": "scheduling", - "uuid": "23c0b5fd-c17e-4326-a389-b3aebc774c82", - "P99": 64, - "P95": 8, - "P50": 5, - "max": 64, - "avg": 5.38, - "timestamp": "2020-11-15T22:26:51.553225151+01:00", - "metricName": "podLatencyQuantilesMeasurement", - "jobName": "kubelet-density" -} -``` - -More information about the pod lifecycle can be found in the [kubernetes docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/). - -**Note**: The __esIndex__ option can be used to configure the ES index where metrics will be indexed. - -#### Pprof collection - -This measurement takes care of collecting golang profiling information from pods. To do so, kube-burner connects to pods with the given labels running in certain namespaces. This measurement uses an implementation similar to `kubectl exec`, and as soon as it connects to one pod it executes the command `curl ` to get the pprof data. Pprof files are collected in a regular basis given by the parameter `pprofInterval` and these files are stored in the directory configured by the parameter `pprofDirectory` which by default is `pprof`. -It's also possible to configure a token to get pprof data from authenticated endoints such as kube-apiserver with the variable `bearerToken`. - -An example of how to configure this measurement to collect pprof HEAP and CPU profiling data from kube-apiserver is shown below: - -```yaml - measurements: - - name: pprof - pprofInterval: 5m - pprofDirectory: pprof-data - pprofTargets: - - name: kube-apiserver-heap - namespace: "openshift-kube-apiserver" - labelSelector: {app: openshift-kube-apiserver} - bearerToken: thisIsNotAValidToken - url: https://localhost:6443/debug/pprof/heap - - - name: kube-apiserver-cpu - namespace: "openshift-kube-apiserver" - labelSelector: {app: openshift-kube-apiserver} - bearerToken: thisIsNotAValidToken - url: https://localhost:6443/debug/pprof/profile?timeout=30 -``` - -**Note**: As mentioned before, this measurement requires cURL to be installed in the target pods. - -## Contributing to kube-burner - -If you want to contribute to kube-burner, submit a Pull Request or Issue. - -### Requirements - -- `golang >= 1.13` -- `make` +- A valid example of a configuration file can be found at [examples/cfg.yml](https://github.com/cloud-bulldozer/kube-burner/blob/master/examples/cfg.yml) diff --git a/docs/AUTHORS.md b/docs/AUTHORS.md new file mode 100644 index 000000000..ac43b2e25 --- /dev/null +++ b/docs/AUTHORS.md @@ -0,0 +1,7 @@ +# Development Lead + +- Raúl Sevilla Cañavate [rsevilla87](https://github.com/rsevilla87) + +# Contributors + +None yet. Why not be the first? diff --git a/docs/CLI.md b/docs/CLI.md new file mode 100644 index 000000000..4cd1f5426 --- /dev/null +++ b/docs/CLI.md @@ -0,0 +1,77 @@ +kube-burner is basically a binary client with currently the following options: + +```console +$ kube-burner help +Kube-burner 🔥 + +Tool aimed at stressing a kubernetes cluster by creating or deleting lot of objects. + +Usage: + kube-burner [command] + +Available Commands: + completion Generates completion scripts for bash shell + destroy Destroy old namespaces labeled with the given UUID. + help Help about any command + index Index metrics from the given time range + init Launch benchmark + version Print the version number of kube-burner + +Flags: + -h, --help help for kube-burner + +Use "kube-burner [command] --help" for more information about a command. +``` + +# Init + +This option is meant to run Kube-burner benchmark, and it supports the these flags: + + - config: Path or URL to a valid configuration file. + - log-level: Logging level. Default `info` + - prometheus-url: Prometheus full URL. i.e. `https://prometheus-k8s-openshift-monitoring.apps.rsevilla.stress.mycluster.example.com` + - metrics-profile: Path to a valid metrics profile file. Default `metrics.yaml` + - token: Prometheus Bearer token. + - username: Prometheus username for basic authentication. + - password: Prometheus password for basic authentication. + - skip-tls-verify: Skip TLS verification for prometheus. Default `true` + - step: Prometheus step size. Default `30s` + - UUID: Benchmark UUID. + +**Note**: Both basic authentication and Bearer authentication need credentials able to query the given Prometheus API. + +With the above, triggering kube-burner would be as simple as: + +```console +$ kube-burner init -c cfg.yml -u https://prometheus-k8s-openshift-monitoring.apps.rsevilla.stress.mycluster.example.com -t ${token} --uuid 67f9ec6d-6a9e-46b6-a3bb-065cde988790` +``` + +Kube-burner also supports remote configuration files served by a web server, to use it, rather than a path pass a URL like below: + +```console +$ kube-burner init -c http://web.domain.com:8080/cfg.yml -t ${token} --uuid 67f9ec6d-6a9e-46b6-a3bb-065cde988790` +``` + +If you have no interest in collecting prometheus metrics, kube-burner can also be launched w/o any prometheus endpoint, this will disable metrics collection. + +```console +$ kube-burner init -c cfg.yml --uuid 67f9ec6d-6a9e-46b6-a3bb-065cde988790` +``` + +# Index + +This option can be used to collect and index the metrics from a given time range. The time range is given by: + + - start: Epoch start time. Defaults to one hour before the current time. + - End: Epoch end time. Defaults to the current time. + +# Destroy + +This option requires the above `config` and `UUID` flags to destroy all namespaces labeled with `kube-burner-uuid=`. + +# Completion +Generates bash a completion script that can be imported with: +`. <(kube-burner completion)` + +Or permanently imported with: +`kube-burner completion > /etc/bash_completion.d/kube-burner` \ No newline at end of file diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md new file mode 100644 index 000000000..564086261 --- /dev/null +++ b/docs/CONFIGURATION.md @@ -0,0 +1,160 @@ + +All the magic `kube-burner` does is described in the configuration file. As previoysly mentioned the location of this configuration file is provided by the flag `-c`. This file is written in YAML format and has several sections. + +# Global + +In this section is described global job configuration, it holds the following parameters: + +| Option | Description | Type | Example | Default | +|------------------|----------------------------------------------------------------------------------------------------------|----------------|----------------|-------------| +| kubeconfig | Points to a valid kubeconfig file. Can be omitted if using the KUBECONFIG environment variable, or running from a pod | String | ~/mykubeconfig | in-cluster | | +| writeToFile | Whether to dump collected metrics to files | Boolean | true | true | +| metricsDirectory | Directory where collected metrics will be dumped into. It will be created if it doesn't exist previously | String | ./metrics | ./collected-metrics | +| measurements | List of measurements. Detailed in the [measurements section](#Measurements) | List | - | [] | +| indexerConfig | Holds the indexer configuration. Detailed in the [indexers section](#Indexers) | Object | - | - | + +# Jobs + +This section contains the list of jobs `kube-burner` will execute. Each job can hold the following parameters. + +| Option | Description | Type | Example | Default | +|----------------------|----------------------------------------------------------------------------------|---------|----------|---------| +| name | Job name | String | myjob | "" | +| jobType | Type of job to execute. More details at [job types](#job-types) | string | create | create | +| jobIterations | How many times to execute the job | Integer | 10 | 0 | +| namespace | Namespace base name to use | String | firstjob | "" | +| namespacedIterations | Whether to create a namespace per job iteration | Boolean | true | true | +| cleanup | Cleanup clean up old namespaces | Boolean | true | true | +| podWait | Wait for all pods to be running before moving forward to the next job iteration | Boolean | true | true | +| waitWhenFinished | Wait for all pods to be running when all iterations are completed | Boolean | true | false | +| maxWaitTimeout | Maximum wait timeout in seconds. (If podWait is enabled this timeout will be reseted with each iteration) | Integer | 1h | 12h | +| waitFor | List containing the objects Kind wait for. Wait for all if empty | List | ["Deployment", "Build", "DaemonSet"]| [] | +| jobIterationDelay | How long to wait between each job iteration | Duration| 2s | 0s | +| jobPause | How long to pause after finishing the job | Duration| 10s | 0s | +| qps | Limit object creation queries per second | Integer | 25 | 0 | +| burst | Maximum burst for throttle | Integer | 50 | 0 | +| objects | List of objects the job will create. Detailed on the [objects section](#objects) | List | - | [] | +| verifyObjects | Verify object count after running each job. Return code will be 1 if failed | Boolean | true | true | +| errorOnVerify | Exit with rc 1 before indexing when objects verification fails | Boolean | true | false | + + +A valid example of a configuration file can be found at [./examples/cfg.yml](https://github.com/cloud-bulldozer/kube-burner/blob/master/examples/cfg.yml) + +# Objects + +The objects created by `kube-burner` are rendered using the default golang's [template library](https://golang.org/pkg/text/template/). +Each object element supports the following parameters: + +| Option | Description | Type | Example | Default | +|----------------------|-------------------------------------------------------------------|---------|-----------------------------------------------------|---------| +| objectTemplate | Object template file or URL | String | deployment.yml or https://domain.com/deployment.yml | "" | +| replicas | How replicas of this object to create per job iteration | Integer | 10 | - | +| inputVars | Map of arbitrary input variables to inject to the object template | Object | - | - | + +It's important to note that all objects created by kube-burner are labeled with. `kube-burner-uuid=,kube-burner-job=,kube-burner-index=` + + +# Job types + +kube-burner support two types of jobs with different parameters each. The default job type is __create__. Which basically creates objects as described in the section [objects](#objects). + +The other type is __delete__, this type of job deletes objects described in the objects list. Using delete as job type the objects list would have the following structure. + +```yaml +objects: +- kind: Deployment + labelSelector: {kube-burner-job: cluster-density} + apiVersion: apps/v1 + +- kind: Secret + labelSelector: {kube-burner-job: cluster-density} +``` +Where: +- kind: Object kind of the k8s object to delete. +- labelSelector: Map with the labelSelector. +- apiVersion: API version from the k8s object. + +As mentioned previously, all objects created by kube-burner are labeled with `kube-burner-uuid=,kube-burner-job=,kube-burner-index=`. Thanks to this we could design a workload with one job to create objects and another one able to remove the objects created by the previous + +```yaml +jobs: +- name: create-objects + namespace: job-namespace + jobIterations: 100 + objects: + - objectTemplate: deployment.yml + replicas: 10 + + - objectTemplate: service.yml + replicas: 10 + +- name: remove-objects + jobType: delete + objects: + - kind: Deployment + labelSelector: {kube-burner-job: create-objects} + apiVersion: apps/v1 + + - kind: Secret + labelSelector: {kube-burner-job: create-objects} +``` + +This job type supports the some of the same parameters as the create job type: +- **waitForDeletion**: Wait for objects to be deleted before finishing the job. Defaults to true +- name +- qps +- burst +- jobPause + +# Injected variables + +All object templates are injected a series of variables by default: + +- Iteration: Job iteration number. +- Replica: Object replica number. Keep in mind that this number is reset to 1 with each job iteration. +- JobName: Job name. +- UUID: Benchmark UUID. + +In addition, you can also inject arbitrary variables with the option **inputVars** from the objectTemplate object: + +```yaml + - objectTemplate: service.yml + replicas: 2 + inputVars: + port: 80 + targetPort: 8080 +``` + +The following code snippet shows an example of a k8s service using these variables: + + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: sleep-app-{{ .Iteration }}-{{ .Replica }} + labels: + name: my-app-{{ .Iteration }}-{{ .Replica }} +spec: + selector: + app: sleep-app-{{ .Iteration }}-{{ .Replica }} + ports: + - name: serviceport + protocol: TCP + port: "{{ .port }}" + targetPort: "{{ .targetPort }}" + type: ClusterIP +``` + +It's worth to say that you can also use [golang template semantics](https://golang.org/pkg/text/template/) in your *objectTemplate* files. + +```yaml +kind: ImageStream +apiVersion: image.openshift.io/v1 +metadata: + name: {{.prefix}}-{{.Replica}} +spec: +{{ if .image }} + dockerImageRepository: {{.image}} +{{ end }} +``` diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md new file mode 100644 index 000000000..29b89b57c --- /dev/null +++ b/docs/CONTRIBUTING.md @@ -0,0 +1,19 @@ +If you want to contribute to kube-burner, submit a Pull Request or Issue. +Kube-burner uses `golangci-lint` to run linters. Prior to send a PR, ensure to execute linters with `make lint`. + +# Requirements + +- `golang >= 1.13` +- `golang-ci-lint` +- `make` + +# Building + +To build kube-burner just execute `make build`, once finished `kube-burner`'s binary should be available at `./bin/kube-burner` + +```console +$ make build +building kube-burner 0.1.0 +GOPATH=/home/rsevilla/go +CGO_ENABLED=0 go build -v -mod vendor -ldflags "-X github.com/cloud-bulldozer/kube-burner/version.GitCommit=d91c8cc35cb458a4b80a5050704a51c7c6e35076 -X github.com/cloud-bulldozer/kube-burner/version.BuildDate=2020-08-19-19:10:09 -X github.com/cloud-bulldozer/kube-burner/version.GitBranch=master" -o bin/kube-burner +``` diff --git a/docs/INDEXERS.md b/docs/INDEXERS.md new file mode 100644 index 000000000..cd1bbac0f --- /dev/null +++ b/docs/INDEXERS.md @@ -0,0 +1,31 @@ +# Configuration + +`kube-burner` is able to **index the collected prometheus metrics** into a given Indexer. +The indexer configuration is described in the `indexerConfig` section and can be configured with the following parameters: + + +| Option | Description | Type | Example | Default | +|----------------------|-----------------------|----------|------------|---------| +| enabled | Enable indexing | Boolean | true | false | +| type | Type of indexer | String | elastic | "" | + + +# Elastic + +Index documents in Elasticsearch 7 instances. + +In addition, each indexer has its own configuration parameters. + +---- + +The `elastic` indexer is configured by the parameters below: + +| Option | Description | Type | Example | Default | +|----------------------|---------------------------------------------------|-------------|------------------------------------------|---------| +| esServers | List of ES instances | List | [https://elastic.apps.rsevilla.org:9200] | "" | +| defaultIndex | Default index to send the prometheus metrics into | String | kube-burner | "" | +| insecureSkipVerify | TLS certificate verification | Boolean | true | false | + +**Note**: It's possible to index documents in an authenticated ES instance using the notation `http(s)://[username]:[password]@[address]:[port]` in the *esServers* parameter. + + diff --git a/docs/MEASUREMENTS.md b/docs/MEASUREMENTS.md new file mode 100644 index 000000000..685a08214 --- /dev/null +++ b/docs/MEASUREMENTS.md @@ -0,0 +1,93 @@ +Apart from prometheus metrics collection, Kube-burner allows to get further metrics using other mechanisms or data sources such as the own kubernetes API, these mechanisms are called measurements. +Measurements are enabled in the measurements section of the configuration file. This section contains a list of measurements with their options. +'kube-burner' supports the following measurements so far: + +# Pod latency + +Collects latencies from the different pod startup phases, these **latency metrics are in ms**. Can be enabled with: + +```yaml + measurements: + - name: podLatency + esIndex: kube-burner-podlatency +``` + +This measurement sends its metrics to the index configured by *esIndex*. The metrics collected are pod latency histograms and pod latency quantiles P99, P95 and P50. + +Pod latency sample: +```json +{ + "timestamp": "2020-11-15T20:28:59.598727718Z", + "schedulingLatency": 4, + "initializedLatency": 20, + "containersReadyLatency": 2997, + "podReadyLatency": 2997, + "metricName": "podLatencyMeasurement", + "jobName": "kubelet-density", + "uuid": "c40b4346-7af7-4c63-9ab4-aae7ccdd0616", + "namespace": "kubelet-density", + "podName": "kubelet-density-13", + "jobName": "kube-burner-job" +} +``` + +Pod latency quantile sample: + +```json +{ + "quantileName": "podReady", + "uuid": "23c0b5fd-c17e-4326-a389-b3aebc774c82", + "P99": 3774, + "P95": 3510, + "P50": 2897, + "max": 3774, + "avg": 2876.3, + "timestamp": "2020-11-15T22:26:51.553221077+01:00", + "metricName": "podLatencyQuantilesMeasurement", + "jobName": "kubelet-density" +}, +{ + "quantileName": "scheduling", + "uuid": "23c0b5fd-c17e-4326-a389-b3aebc774c82", + "P99": 64, + "P95": 8, + "P50": 5, + "max": 64, + "avg": 5.38, + "timestamp": "2020-11-15T22:26:51.553225151+01:00", + "metricName": "podLatencyQuantilesMeasurement", + "jobName": "kubelet-density" +} +``` + +More information about the pod lifecycle can be found in the [kubernetes docs](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/). + +**Note**: The __esIndex__ option can be used to configure the ES index where metrics will be indexed. + +# Pprof collection + +This measurement takes care of collecting golang profiling information from pods. To do so, kube-burner connects to pods with the given labels running in certain namespaces. This measurement uses an implementation similar to `kubectl exec`, and as soon as it connects to one pod it executes the command `curl ` to get the pprof data. Pprof files are collected in a regular basis given by the parameter `pprofInterval` and these files are stored in the directory configured by the parameter `pprofDirectory` which by default is `pprof`. +It's also possible to configure a token to get pprof data from authenticated endoints such as kube-apiserver with the variable `bearerToken`. + +An example of how to configure this measurement to collect pprof HEAP and CPU profiling data from kube-apiserver is shown below: + +```yaml + measurements: + - name: pprof + pprofInterval: 5m + pprofDirectory: pprof-data + pprofTargets: + - name: kube-apiserver-heap + namespace: "openshift-kube-apiserver" + labelSelector: {app: openshift-kube-apiserver} + bearerToken: thisIsNotAValidToken + url: https://localhost:6443/debug/pprof/heap + + - name: kube-apiserver-cpu + namespace: "openshift-kube-apiserver" + labelSelector: {app: openshift-kube-apiserver} + bearerToken: thisIsNotAValidToken + url: https://localhost:6443/debug/pprof/profile?timeout=30 +``` + +**Note**: As mentioned before, this measurement requires cURL to be installed in the target pods. diff --git a/docs/METRICS.md b/docs/METRICS.md new file mode 100644 index 000000000..58050ccbf --- /dev/null +++ b/docs/METRICS.md @@ -0,0 +1,72 @@ +# Metrics profile + +The metrics-profile flag points to a YAML or URL of a file containing a list of the prometheus queries kube-burner will collect for each job. +As soon one of job finishes, `kube-burner` makes a range query for each query described in this file, and indexes it in the index configured by the parameter `defaultIndex`. +We can use the parameter `indexName` in a metrics-profile file to make `kube-burner` to index the resulting metrics to a different index. +An example of a valid metrics profile file is shown below: +The parameter **metricName** is added to the indexed documents, it will allow us to identify documents from a certain query more easily. + +```yaml +metrics: + - query: irate(process_cpu_seconds_total{job=~".*(crio|etcd|controller-manager|apiserver|scheduler).*"}[2m]) + metricName: controlPlaneCPU + + - query: process_resident_memory_bytes{job=~".*(crio|etcd|controller-manager|apiserver|scheduler).*"} + metricName: controlPlaneMemory + + - query: sum(irate(node_cpu_seconds_total[2m])) by (mode,instance) + metricName: nodeCPU +``` + + +It's also possible to execute instant queries from kube-burner by adding the flag instant to the desired metric. These kind of queries are useful to get only one sample for a static metric such as the number of nodes or the kube-apiserver version. + +```yaml +metrics: + - query: kube_node_role + metricName: nodeRoles + instant: true +``` + +# Job Summary + +In case indexing is enabled, at the end of each job, a document holding the job summary is indexed. This is useful to identify the parameters the job was executed with: + +This document looks like: + +```json +{ + "timestamp": "2020-11-13T13:55:31.654185032+01:00", + "uuid": "bdb7584a-d2cd-4185-8bfa-1387cc31f99e", + "metricName": "jobSummary", + "elapsedTime": 8.768932955, + "jobConfig": { + "jobIterations": 10, + "jobIterationDelay": 0, + "jobPause": 0, + "name": "kubelet-density", + "objects": [ + { + "objectTemplate": "templates/pod.yml", + "replicas": 1, + "inputVars": { + "containerImage": "gcr.io/google_containers/pause-amd64:3.0" + } + } + ], + "jobType": "create", + "qps": 5, + "burst": 5, + "namespace": "kubelet-density", + "waitFor": null, + "maxWaitTimeout": 43200000000000, + "waitForDeletion": true, + "podWait": false, + "waitWhenFinished": true, + "cleanup": true, + "namespacedIterations": false, + "verifyObjects": true, + "errorOnVerify": false + } +} +``` diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..b6b08723c --- /dev/null +++ b/docs/README.md @@ -0,0 +1,23 @@ + + +[![Build Status](https://github.com/cloud-bulldozer/kube-burner/workflows/Go/badge.svg?branch=master)](https://github.com/cloud-bulldozer/kube-burner/actions?query=workflow%3AGo) +[![Go Report Card](https://goreportcard.com/badge/github.com/cloud-bulldozer/kube-burner)](https://goreportcard.com/report/github.com/cloud-bulldozer/kube-burner) +[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) + + +Kube-burner is a tool aimed at stressing kubernetes clusters. The main functionallity it provides can be summareized these three steps: + +- Create/delete the objects declared in the jobs. +- Collect desired on-cluster prometheus metrics. +- Write and/or index them to the configured TSDB. + +[![asciicast](https://asciinema.org/a/KksoK5voK3al1FuOza89t1JAp.svg)](https://asciinema.org/a/KksoK5voK3al1FuOza89t1JAp) + + +## Downloading Kube-burner + +In case you want to start tinkering with `kube-burner` now: + +- You can find the binaries in the [releases section of the repository](https://github.com/cloud-bulldozer/kube-burner/releases). +- There's also a container image available at [quay](https://quay.io/repository/cloud-bulldozer/kube-burner?tab=tags). +- A valid example of a configuration file can be found at [examples/cfg.yml](https://github.com/cloud-bulldozer/kube-burner/blob/master/examples/cfg.yml) diff --git a/docs/kube-burner-logo.png b/docs/kube-burner-logo.png new file mode 120000 index 000000000..d1e0220bd --- /dev/null +++ b/docs/kube-burner-logo.png @@ -0,0 +1 @@ +../media/logo/kube-burner-logo.png \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 000000000..f2cc963fb --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,20 @@ +docs_dir: docs/ +repo_url: https://github.com/cloud-bulldozer/kube-burner +nav: +- Home: + - Quick start: README.md + - CLI: CLI.md +- Configuration: + - General: CONFIGURATION.md + - Measurements: MEASUREMENTS.md + - Metrics: METRICS.md + - Indexers: INDEXERS.md +- Development: + - Contributing: CONTRIBUTING.md + - Authors: AUTHORS.md +site_name: Kube-burner +markdown_extensions: +- toc: + permalink: "#" +theme: + name: readthedocs