Documentation for Iter8 metrics (#627)

* multidb

* Creating new Iter8 metrics

* prometheus metric descriptions

* prometheus with basic auth

* new relic documentation

* block chomped yaml

* sysdig documentation

* complete documentation for metrics

* Elastic documentation

* Updated sample experiment in metrics documentation

* relative slack unfurl path

* DBs

* Providers in slack  expansion

* Shorter slack description

* Fixed installation instructions

* Fixed install instructions

* Updated install and conformance tests

* Updated installation and platform setup

* Updated check.sh

* Fixed documentation for metrics

* check.sh for mirroring

* image url

* Fixing illustration.png absolute path

* Updated install and quickstart instructions

mkdocs/docs/concepts/buildingblocks.md

@@ -39,7 +39,7 @@ When two or more versions participate in an experiment, Iter8 **recommends a ver
 
 ## Metrics
 
-Metric backends like Prometheus, New Relic, Sysdig and Elastic collect metrics for deployed versions and serve them through REST APIs. Iter8 defines a new Kubernetes resource called **Metric** that makes it easy to use metrics in experiments from any RESTful metrics backend.
+Metric providers like **Prometheus**, **New Relic**, **Sysdig** and **Elastic** can collect metrics for versions and serve them through REST APIs. Iter8 defines a new Kubernetes resource called **Metric** that makes it easy to use metrics in experiments from these and any other RESTful metrics provider.
 
 ## Objectives
 

mkdocs/docs/concepts/features.md

@@ -14,7 +14,7 @@ Iter8 makes it easy to achieve the following goals.
 - **Progressive** traffic shifting.
 - **Dark launches**, **traffic mirroring** and **traffic segmentation**.
 - Use Helm, Kustomize, and plain YAML/JSON for defining your app manifests.
-- Use out-of-the-box metrics or define custom metrics based on data in **Prometheus**.
+- Use metrics from any RESTful provider including **Prometheus**, **New Relic**, **Sysdig**, and **Elastic**.
 - Statistically rigorous evaluation of versions, traffic splitting, and promotion/rollback decisions using **Bayesian learning** and **multi-armed bandit** algorithms.
 - Observe experiments in realtime.
 

mkdocs/docs/getting-started/install.md

@@ -5,43 +5,18 @@ title: Installation
 
 # Installation
 
-## Step 1: Iter8
+## Iter8
 
-!!! example "Prerequisites"
-
-    1. A Kubernetes cluster
-    2. [kubectl CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
-
-Install Iter8 in your Kubernetes cluster as follows.
+Install Iter8 in your Kubernetes cluster as follows. This installation requires [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
 
 ```shell
-export TAG=v0.3.2
+export TAG=v0.4.3
 curl -s https://raw.githubusercontent.com/iter8-tools/iter8-install/main/install.sh | bash
 ```
 
-## (Optional) Step 2: Prometheus add-on
-
-Install Iter8's Prometheus add-on in your cluster as follows. This step assumes you have installed Iter8 following Step 1 above.
-
-```shell
-export TAG=v0.3.2
-curl -s https://raw.githubusercontent.com/iter8-tools/iter8-install/main/install-prom-add-on.sh | bash
-```
-
-??? note "Running Iter8 tutorials without Iter8's Prometheus add-on"
-    When you installed Iter8 in the first step above, you also installed several *out-of-the-box* Iter8 metric resources. They are required for running the tutorials documented on this site. 
-
-    The out-of-the-box metric resources have a urlTemplate field. This field is configured as the URL of the Prometheus instance created in this step. 
-
-    You can skip this step and still run Iter8 tutorials using your own Prometheus instance. To do so, ensure that your Prometheus instance scrapes the end-points that would have been scraped by the Prometheus instance created in this step, and configure the urlTemplate fields of Iter8 metric resources to match the URL of your Prometheus instance.
-
-## (Optional) Step 3: iter8ctl
-The iter8ctl CLI enables real-time observability of Iter8 experiments. 
-
-!!! example "Prerequisites"
-
-    Go 1.13+
+## iter8ctl (optional)
+The `iter8ctl` client facilitates real-time observability of Iter8 experiments. Install `iter8ctl` on your local machine as follows. This installation requires Go 1.13+.
 
 ```shell
-GO111MODULE=on GOBIN=/usr/local/bin go get github.com/iter8-tools/[email protected].2
+GO111MODULE=on GOBIN=/usr/local/bin go get github.com/iter8-tools/[email protected].3
 ```

mkdocs/docs/getting-started/quick-start/with-knative.md

@@ -251,7 +251,7 @@ Observe the experiment in realtime. Paste commands from the tabs below in separa
 === "iter8ctl"
     Install `iter8ctl`. You can change the directory where `iter8ctl` binary is installed by changing `GOBIN` below.
     ```shell
-    GO111MODULE=on GOBIN=/usr/local/bin go get github.com/iter8-tools/[email protected].2
+    GO111MODULE=on GOBIN=/usr/local/bin go get github.com/iter8-tools/[email protected].3
     ```
 
     Periodically describe the experiment.
@@ -261,7 +261,7 @@ Observe the experiment in realtime. Paste commands from the tabs below in separa
     sleep 4
     done
     ```
-    ??? info "iter8ctl output"
+    ??? info "Look inside `iter8ctl` output"
         The `iter8ctl` output will be similar to the following.
         ```shell
         ****** Overview ******

mkdocs/docs/metrics/using-metrics.md

@@ -5,7 +5,7 @@ template: main.html
 # Using Metrics in Experiments
 
 !!! tip "Iter8 metrics API"    
-    Iter8 defines a new Kubernetes resource called Metric that makes it easy to use metrics in experiments from RESTful metric backends like Prometheus, New Relic, Sysdig and Elastic.
+    Iter8 defines a new Kubernetes resource called Metric that makes it easy to use metrics in experiments from RESTful metric providers like Prometheus, New Relic, Sysdig and Elastic.
 
     List metrics available in your cluster using the `kubectl get metrics.iter8.tools` command. Use metrics in experiments by referencing them in experiment criteria.
 
@@ -18,11 +18,11 @@ kubectl get metrics.iter8.tools --all-namespaces
 
 ```shell
 NAMESPACE       NAME                           TYPE      DESCRIPTION
-iter8-knative   95th-percentile-tail-latency   gauge     95th percentile tail latency
-iter8-knative   error-count                    counter   Number of error responses
-iter8-knative   error-rate                     gauge     Fraction of requests with error responses
-iter8-knative   mean-latency                   gauge     Mean latency
-iter8-knative   request-count                  counter   Number of requests
+iter8-knative   95th-percentile-tail-latency   Gauge     95th percentile tail latency
+iter8-knative   error-count                    Counter   Number of error responses
+iter8-knative   error-rate                     Gauge     Fraction of requests with error responses
+iter8-knative   mean-latency                   Gauge     Mean latency
+iter8-knative   request-count                  Counter   Number of requests
 ```
 
 ## Referencing metrics

mkdocs/docs/reference/apispec.md

@@ -8,7 +8,7 @@ template: main.html
     The Iter8 API provides two [Kubernetes custom resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) to automate metrics and AI-driven experiments, progressive delivery, and rollout of Kubernetes and OpenShift apps.
 
     1. The **Experiment** resource provides expressive controls required by application developers and service operators who wish to automate new releases of their apps in a robust, principled and metrics-driven manner. These controls encompass [testing, deployment, traffic shaping, and version promotion functions](../../../concepts/buildingblocks/) and can be flexibly composed to automate [diverse use-cases](../../../tutorials/knative/canary-progressive/).
-    2. The **Metric** resource encapsulates the REST query that is used by Iter8 for retrieving a metric value from the metrics backend. Metrics are referenced in experiments.
+    2. The **Metric** resource encapsulates the REST query that is used by Iter8 for retrieving a metric value from the metrics provider. Metrics are referenced in experiments.
 
 
 !!! note "API Version"    
@@ -132,16 +132,19 @@ Standard Kubernetes [meta.v1/ObjectMeta](https://kubernetes.io/docs/reference/ge
 #### Spec
 | Field name | Field type         | Description | Required |
 | ----- | ------------ | ----------- | -------- |
-| params | [][NamedValue](#namedvalue) | List of name/value pairs corresponding to the name and value of the HTTP query parameters used by Iter8 when querying the metrics backend. Each name represents a parameter name; the corresponding value is a string template with placeholders, which will be interpolated by Iter8 at query time. | No |
-| description | string | Human readable description. | No |
-| units | string | Units of measurement. Units are used only for display purposes. | No |
-| type | string | Metric type. Valid values are `counter` and `gauge`. Default value = `gauge`. | No |
-| sampleSize | string | Reference to a metric that represents the number of data points over which the metric value is computed. This field applies only to `gauge` metrics. References can be expressed in the form 'name' or 'namespace/name'. If just `name` is used, the implied namespace is the namespace of the referring metric. | No |
-| provider | string | Type of the metrics database. Provider is used only for display purposes. | No |
-| jqExpression | string | The [jq](https://stedolan.github.io/jq/) expression used by Iter8 to extract the metric value from the JSON response of the metrics backend to a metrics query. | Yes |
-| secret | string | Reference to a secret that contains information used for authenticating with the metrics database. In particular, Iter8 uses data in this secret to interpolate the HTTP headers and URL while querying the database. References can be expressed in the form 'name' or 'namespace/name'. If just `name` is used, the implied namespace is the namespace where Iter8 is installed (which is `iter8-system` by default). | No |
-| headerTemplates | [][NamedValue](#namedvalue) | List of templates for headers that should be added to metrics queries. Variable portions of the headers, expressed in the form `{.name}` will be replaced at runtime with the value of the `name` entry defined in the secret. If no value can be found in the secret, no replacement will be done. | No |
-| urlTemplate | string | Template for URL of metrics server. Variable portions of the URL, expressed in the form `{.name}` will be replaced at runtimme with the value of the `name` entry defined in the secret. If no value can be found in the secret, no replacement will be done. | Yes |
+| description | string | Human readable description. This field is meant for informational purposes. | No |
+| units | string | Units of measurement. This field is meant for informational purposes. | No |
+| provider | string | Type of the metrics provider. This field is meant for informational purposes. | No |
+| params | [][NamedValue](#namedvalue) | List of name/value pairs corresponding to the name and value of the HTTP query parameters used by Iter8 when querying the metrics provider. Each name represents a parameter name; the corresponding value is a string template with placeholders; the placeholders will be dynamically substituted by Iter8 with values at query time. | No |
+| body | string | String used to construct the JSON body of the HTTP request. Body may be templated, in which Iter8 will attempt to substitute placeholders in the template at query time using version information. | No |
+| type | string | Metric type. Valid values are `Counter` and `Gauge`. Default value = `Gauge`. A `Counter` metric is one whose value never decreases over time. A `Gauge` metric is one whose value may increase or decrease over time. | No |
+| method | string | HTTP method (verb) used in the HTTP request. Valid values are `GET` and `POST`. Default value = `GET`. | No |
+| authType | string | Identifies the type of authentication used in the HTTP request. Valid values are `Basic`, `Bearer` and `APIKey` which correspond to HTTP authentication with these respective methods. | No |
+| sampleSize | string | Reference to a metric that represents the number of data points over which the value of this metric is computed. This field applies only to `Gauge` metrics. References can be expressed in the form 'name' or 'namespace/name'. If just `name` is used, the implied namespace is the namespace of the referring metric. | No |
+| secret | string | Reference to a secret that contains information used for authenticating with the metrics provider. In particular, Iter8 uses data in this secret to substitute placeholders in the HTTP headers and URL while querying the provider. References can be expressed in the form 'name' or 'namespace/name'. If just `name` is used, the implied namespace is the namespace where Iter8 is installed (which is `iter8-system` by default). | No |
+| headerTemplates | [][NamedValue](#namedvalue) | List of name/value pairs corresponding to the name and value of the HTTP request headers used by Iter8 when querying the metrics provider. Each name represents a header field name; the corresponding value is a string template with placeholders; the placeholders will be dynamically substituted by Iter8 with values at query time. Placeholder substitution is attempted only if `authType` and `secret` fields are present. | No |
+| jqExpression | string | The [jq](https://stedolan.github.io/jq/) expression used by Iter8 to extract the metric value from the JSON response returned by the provider. | Yes |
+| urlTemplate | string | Template for the metric provider's URL. Typically, urlTemplate is expected to be the actual URL without any placeholders. However, urlTemplate may be templated, in which case, Iter8 will attempt to substitute placeholders in the urlTemplate at query time using the `secret` referenced in the metric. Placeholder substitution will not be attempted if `secret` is not specified. | Yes |
 
 ## Experiment field types
 
@@ -214,7 +217,7 @@ Standard Kubernetes [meta.v1/ObjectMeta](https://kubernetes.io/docs/reference/ge
 | Field name | Field type | Description | Required |
 | ----- | ---- | ----------- | -------- |
 | name | string | Name of the version. | Yes |
-| variables | [][NamedValue](#namedvalue) | Variables are name-value pairs associated with a version. Metrics and tasks within experiment specs can contain strings with placeholders. Iter8 uses variables to interpolate these strings. | No |
+| variables | [][NamedValue](#namedvalue) | Variables are name-value pairs associated with a version. Metrics and tasks within experiment specs can contain strings with placeholders. Iter8 uses variables to substitute placeholders in these strings. | No |
 | weightObjRef | [corev1.ObjectReference](https://pkg.go.dev/k8s.io/[email protected]/core/v1#ObjectReference) | Reference to a Kubernetes resource and a field-path within the resource. Iter8 uses `weightObjRef` to get or set weight (traffic percentage) for the version. | No |
 
 
@@ -534,7 +537,7 @@ The `common` task library provides the `exec` task. Use this task to execute she
               - "sample-app" # release name
               - "--namespace=iter8-system" # release namespace
               - "sample-app" # chart name
-              - "--values=https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/canaryprogressive/{{ .promote }}-values.yaml" # values URL dynamically interpolated
+              - "--values=https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/canaryprogressive/{{ .promote }}-values.yaml" # placeholder is substituted dynamically
     ```
 
 === "Kustomize"
@@ -562,21 +565,21 @@ The `common` task library provides the `exec` task. Use this task to execute she
               kustomize build github.com/iter8-tools/iter8/samples/knative/canaryfixedsplit/{{ .name }}?ref=master | kubectl apply -f -
     ```
 
-### Interpolation of task inputs
+### Placeholder substitution in task inputs
 
 Inputs to tasks can contain placeholders, or template variables, which will be dynamically substituted when the task is executed by Iter8. For example, in the sample experiment above, one input is:
 
 ```bash 
 "https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/{{ .promote }}.yaml"
 ```
 
-In this case, the placeholder is `{{ .promote }}`. Variable interpolation works as follows.
+In this case, the placeholder is `{{ .promote }}`. Placeholder substitution in task inputs works as follows.
 
 1. Iter8 will find the version recommended for promotion. This information is stored in the `status.versionRecommendedForPromotion` field of the experiment. The version recommended for promotion is the `winner`, if a `winner` has been found in the experiment. Otherwise, it is the baseline version supplied in the `spec.versionInfo` field of the experiment.
 
 2. If the placeholder is `{{ .name }}`, Iter8 will substitute it with the name of the version recommended for promotion. Else, if it is any other variable, Iter8 will substitute it with the value of the corresponding variable for the version recommended for promotion. Variable values are specified in the `variables` field of the version detail. Note that variable values could have been supplied by the creator of the experiment, or by other tasks such as `init-experiment` that may already have been executed by Iter8 as part of the experiment.
 
-??? example "Interpolation Example 1"
+??? example "Placeholder substitution Example 1"
 
     Consider the sample experiment above. Suppose the `winner` of this experiment was `candidate`. Then:
 
@@ -585,7 +588,7 @@ In this case, the placeholder is `{{ .promote }}`. Variable interpolation works
     3. The value of the placeholder for the version recommended for promotion is `candid`.
     4. The command executed by the `exec` task is then `kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/candid.yaml`.
 
-??? example "Interpolation Example 2"
+??? example "Placeholder substitution Example 2"
 
     Consider the sample experiment above. Suppose the `winner` of this experiment was `current`. Then:
 
@@ -594,7 +597,7 @@ In this case, the placeholder is `{{ .promote }}`. Variable interpolation works
     3. The value of the placeholder for the version recommended for promotion is `base`.
     4. The command executed by the `exec` task is then `kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/base.yaml`.
 
-??? example "Interpolation Example 3"
+??? example "Placeholder substitution Example 3"
 
     Consider the sample experiment above. Suppose the experiment did not yield a `winner`. Then:
 
@@ -604,7 +607,7 @@ In this case, the placeholder is `{{ .promote }}`. Variable interpolation works
     4. The command executed by the `exec` task is then `kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/base.yaml`.
 
 ### Disable Interpolation (always do this in a `start` action)
-By default, the `common/exec` task will attempt to find the version recommended for promotion, and use its values to interpolate the inputs to the task. However, this behavior will lead to task failure since version recommended for promotion will be generally undefined at this stage of the experiment. To use the `common/exec` task as part of an experiment `start` action, set `disableInterpolation` to `true` as illustrated in the `kubectl/Helm/Kustomize` samples above.
+By default, the `common/exec` task will attempt to find the version recommended for promotion, and use its values to substitute placeholders in the inputs to the task. However, this behavior will lead to task failure since version recommended for promotion will be generally undefined at this stage of the experiment. To use the `common/exec` task as part of an experiment `start` action, set `disableInterpolation` to `true` as illustrated in the `kubectl/Helm/Kustomize` samples above.
 
 ### Error handling in tasks
 When a task exits with an error, it will result in the failure of the experiment to which it belongs.