Architectural overview of KLT

[StackScribe]

Move the architectural description that is currently in the README.md file (https://github.com/keptn/lifecycle-toolkit/blob/main/README.md) into the documentation.

Strategy is to quickly pull in the existing material somewhat as it is. Long term, we may want to rework it.

Part of #770

[StackScribe]

In 0.7.0, we add the Metrics Operator. That needs to be added into this section. @odubajDT has provided good source info in the following:

• docs: adapt KeptnEvaluationDefinition and introduce KeptnMetricsProvider #944
• docs: migrator for KeptnEvaluationProvider -> KeptnMetricsProvider #945
• test: introduce integration test with KeptnEvaluation #947

Also, it might be good if the first use of the term "Operator" links to https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ . We don't want to be duplicating k8s docs but it doesn't hurt to give people a link, in case they are not total k8s experts.

[StackScribe]

From Florian Bacher:
There have also been some changes in the docs about KeptnMetrics can be used with an HorizontalPodAutoscaler: https://github.com/keptn/lifecycle-toolkit/blob/main/docs/content/en/docs/concepts/metrics/_index.md - this is more user related, but I think the metrics API endpoint might also be something worth to be mentioned in the architectural overview
_index.md

Keptn Metric

A KeptnMetric is a CRD representing a metric. The metric will be collected from the provider specified in the
specs.provider.name field. The query is a string in the provider-specific query language, used to obtain a metric.
Providing the metrics as CRD into a K8s cluster will facilitate the reusability of this data across multiple components.
Furthermore, this allows using multiple observability platforms for different metrics.
A KeptnMetric looks like the following:

apiVersion: [metrics.keptn.sh/v1alpha1](http://metrics.keptn.sh/v1alpha1)
kind: KeptnMetric
metadata:
  name: keptnmetric-sample
  namespace: podtato-kubectl
spec:
  provider:
    name: "prometheus"
  query: "sum(kube_pod_container_resource_limits{resource='cpu'})"
  fetchIntervalSeconds: 5

In this example, the provider is set to prometheus, which is one of the currently supported KeptnMetricProviders.
The provider tells the metrics-operator where to get the value for the KeptnMetric, and its configuration looks follows:

apiVersion: [metrics.keptn.sh/v1alpha2](http://metrics.keptn.sh/v1alpha2)
kind: KeptnMetricsProvider
metadata:
  name: prometheus
spec:
  targetServer: "http://prometheus-k8s.monitoring.svc.cluster.local:9090/"
Other supported providers are dynatrace, and dql:
apiVersion: [metrics.keptn.sh/v1alpha2](http://metrics.keptn.sh/v1alpha2)
kind: KeptnMetricsProvider
metadata:
  name: dynatrace
  namespace: podtato-kubectl
spec:
  targetServer: "[dynatrace-tenant-url](https://app.slack.com/client/T73ASNB9Q/C04LY8V5H7G/thread/dynatrace-tenant-url)"
  secretKeyRef:
    name: dt-api-token
    key: DT_TOKEN

---

apiVersion: [metrics.keptn.sh/v1alpha2](http://metrics.keptn.sh/v1alpha2)
kind: KeptnMetricsProvider
metadata:
  name: dql
  namespace: podtato-kubectl
spec:
  secretKeyRef:
    key: CLIENT_SECRET
    name: dt-third-gen-secret 
  targetServer: "[dynatrace-third-gen-target-server](https://app.slack.com/client/T73ASNB9Q/C04LY8V5H7G/thread/dynatrace-third-gen-target-server)"

Keptn metrics can be exposed as OTel metrics via port 9999 of the KLT metrics-operator. To expose them, the env
variable EXPOSE_KEPTN_METRICS in the metrics-operator manifest needs to be set to true. The default value of this variable
is true. To access the metrics, use the following command:

kubectl port-forward deployment/metrics-operator 9999 -n keptn-lifecycle-toolkit-system

and access the metrics via your browser with:

http://localhost:9999/metrics

Accessing Metrics via the Kubernetes Custom Metrics API

KeptnMetrics can also be retrieved via the Kubernetes Custom Metrics API.
This makes it possible to refer to these metrics via the Kubernetes HorizontalPodAutoscaler, as in the following
example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: podtato-head-entry
  namespace: podtato-kubectl
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podtato-head-entry
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Object
      object:
        metric:
          name: keptnmetric-sample
        describedObject:
          apiVersion: [metrics.keptn.sh/v1alpha1](http://metrics.keptn.sh/v1alpha1)
          kind: KeptnMetric
          name: keptnmetric-sample
        target:
          type: Value
          value: "10"

You can also use the kubectl raw command to retrieve the values of a KeptnMetric, as in the following example:
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/podtato-kubectl/keptnmetrics.metrics.sh/keptnmetric-sample/keptnmetric-sample" | jq .

{
  "kind": "MetricValueList",
  "apiVersion": "[custom.metrics.k8s.io/v1beta2](http://custom.metrics.k8s.io/v1beta2)",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "KeptnMetric",
        "namespace": "podtato-kubectl",
        "name": "keptnmetric-sample",
        "apiVersion": "[metrics.keptn.sh/v1alpha1](http://metrics.keptn.sh/v1alpha1)"
      },
      "metric": {
        "name": "keptnmetric-sample",
        "selector": {
          "matchLabels": {
            "app": "frontend"
          }
        }
      },
      "timestamp": "2023-01-25T09:26:15Z",
      "value": "10"
    }
  ]
}

You can also filter based on matching labels. So to e.g. retrieve all metrics that are labelled with app=frontend, you
can use the following command:
$ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta2/namespaces/podtato-kubectl/keptnmetrics.metrics.sh//?labelSelector=app%3Dfrontend" | jq .

{
  "kind": "MetricValueList",
  "apiVersion": "[custom.metrics.k8s.io/v1beta2](http://custom.metrics.k8s.io/v1beta2)",
  "metadata": {},
  "items": [
    {
      "describedObject": {
        "kind": "KeptnMetric",
        "namespace": "keptn-lifecycle-toolkit-system",
        "name": "keptnmetric-sample",
        "apiVersion": "[metrics.keptn.sh/v1alpha1](http://metrics.keptn.sh/v1alpha1)"
      },
      "metric": {
        "name": "keptnmetric-sample",
        "selector": {
          "matchLabels": {
            "app": "frontend"
          }
        }
      },
      "timestamp": "2023-01-25T09:26:15Z",
      "value": "10"
    }
  ]
}

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be
closed if no further activity occurs. Thank you for your contributions.