feat: Added tutorial for setting up observability example (#145)

examples/observability/README.md

@@ -0,0 +1,69 @@
+# Sending Traces and Metrics to the OpenTelemetry Collector
+
+In this example, we will show you an example configuration for enabling the operator to send OpenTelemetry traces and metrics to the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector).
+The Collector will then be used to forward the gathered data to [Jaeger](https://www.jaegertracing.io) and [Prometheus](https://prometheus.io).
+
+## Prerequisites:
+
+This tutorial assumes that you have both Jaeger and the Prometheus Operator installed in your Cluster. 
+Also, please ensure that the Prometheus Operator has the required permissions to watch resources of the `keptn-lifecycle-controller-system` namespace (see https://prometheus-operator.dev/docs/kube/monitoring-other-namespaces/ as a reference).
+For setting up both Jaeger and Prometheus, please refer to their docs:
+
+- [Jaeger Setup](https://github.com/jaegertracing/jaeger-operator) 
+- [Prometheus Operator Setup](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizing.md)
+
+In this tutorial, we will assume that Jaeger will be installed in the `keptn-lifecycle-controller-system`, and the Jaeger collector endpoint is reachable under `http://jaeger-collector:14250`.
+
+## Configuring the OpenTelemetry Collector and Prometheus ServiceMonitor
+
+Once Jaeger and Prometheus are installed, you can deploy and configure the OpenTelemetry collector using the manifests in the `config` directory:
+
+```sh 
+kubectl apply -f config/
+```
+
+Also, please ensure that the `OTEL_COLLECTOR_URL` env vars of both the `klc-controller-manager`, 
+as well as the `keptn-scheduler` deployments are set appropriately. 
+By default, they are set to `otel-collector:4317`, which should be the correct value for this tutorial.
+
+Eventually, there should be a pod for the `otel-collector` deployment up and running:
+
+```sh
+$ kubectl get pods -lapp=opentelemetry -n keptn-lifecycle-controller-system
+
+NAME                              READY   STATUS    RESTARTS      AGE
+otel-collector-6fc4cc84d6-7hnvp   1/1     Running   6 (51m ago)   92m
+```
+
+## Seeing the OpenTelemetry Collector in action
+
+After everything has been set up, use the lifecycle operator to deploy a workload (e.g. using the `single-service` or `podtato-head` example in the `examples` folder).
+Once either of these examples have been deployed, you can view the generated traces in Jaeger. To do so, please create a port-forward for the `jaeger-query` service:
+
+```sh
+kubectl port-forward -n keptn-lifecycle-controller-system svc/jaeger-query 16686 
+```
+
+Afterwards, you can view the Jaeger UI in the browser at [localhost:16686](http://localhost:16686). There you should see the traces generated by the lifecycle controller, which should look like this:
+
+**Traces overview** 
+
+![](./assets/traces_overview.png)
+
+**Trace details**
+
+![](./assets/trace_detail.png)
+
+In Prometheus, do a port forward to the prometheus service inside your cluster (the exact name and namespace of the prometheus service will depend on your Prometheus setup - we are using the defaults that come with the example of the Prometheus Operator tutorial).
+
+```sh
+kubectl -n monitoring port-forward svc/prometheus-k8s 9090
+```
+
+Afterwards, you can view the Prometheus UI in the browser at [localhost:9090](http://localhost:9090). There, in the [Targets](http://localhost:9090/targets?search=) section, you should see an entry for the otel-collector:
+
+![](./assets/prometheus_targets.png)
+
+Also, in the [Graph](http://localhost:9090/graph?g0.expr=&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h) section, you can retrieve metrics reported by the Keptn Lifecycle Controller (all of the available metrics start with the `keptn` prefix):
+
+![](./assets/metrics.png)

examples/observability/config/otel-collector.yaml

@@ -0,0 +1,155 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: otel-collector-conf
+  namespace: keptn-lifecycle-controller-system
+  labels:
+    app: opentelemetry
+    component: otel-collector-conf
+data:
+  otel-collector-config: |
+    receivers:
+      # Make sure to add the otlp receiver.
+      # This will open up the receiver on port 4317
+      otlp:
+        protocols:
+          grpc:
+            endpoint: "0.0.0.0:4317"
+      prometheus:
+        config:
+          scrape_configs:
+            - job_name: 'otel-collector'
+              scrape_interval: 5s
+              static_configs:
+                - targets: ['klc-controller-manager-metrics-service:2222']
+    processors:
+    extensions:
+      health_check: {}
+    exporters:
+      jaeger:
+        endpoint: "jaeger-collector:14250"
+        tls:
+          insecure: true
+      prometheus:
+        endpoint: 0.0.0.0:8889
+      logging:
+
+    service:
+      extensions: [health_check]
+      pipelines:
+        traces:
+          receivers: [otlp]
+          processors: []
+          exporters: [jaeger]
+
+        metrics:
+          receivers: [otlp,prometheus]
+          processors: []
+          exporters: [prometheus, logging]
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: otel-collector
+  namespace: keptn-lifecycle-controller-system
+  labels:
+    app: opentelemetry
+    component: otel-collector
+spec:
+  ports:
+    - name: otlp # Default endpoint for otlp receiver.
+      port: 4317
+      protocol: TCP
+      targetPort: 4317
+      nodePort: 30080
+    - name: metrics # Default endpoint for metrics.
+      port: 8889
+      protocol: TCP
+      targetPort: 8889
+  selector:
+    component: otel-collector
+  type: NodePort
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: otel-collector
+  namespace: keptn-lifecycle-controller-system
+  labels:
+    app: opentelemetry
+    component: otel-collector
+spec:
+  selector:
+    matchLabels:
+      app: opentelemetry
+      component: otel-collector
+  minReadySeconds: 5
+  progressDeadlineSeconds: 120
+  replicas: 1
+  template:
+    metadata:
+      annotations:
+        prometheus.io/path: "/metrics"
+        prometheus.io/port: "8889"
+        prometheus.io/scrape: "true"
+      labels:
+        app: opentelemetry
+        component: otel-collector
+    spec:
+      containers:
+        - command:
+            - "/otelcol"
+            - "--config=/conf/otel-collector-config.yaml"
+          env:
+            - name: GOGC
+              value: "80"
+          image: otel/opentelemetry-collector:0.60.0
+          name: otel-collector
+          resources:
+            limits:
+              cpu: 1
+              memory: 2Gi
+            requests:
+              cpu: 200m
+              memory: 400Mi
+          ports:
+            - containerPort: 4317 # Default endpoint for otlp receiver.
+            - containerPort: 8889 # Default endpoint for querying metrics.
+          volumeMounts:
+            - name: otel-collector-config-vol
+              mountPath: /conf
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 13133 # Health Check extension default port.
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 13133 # Health Check extension default port.
+      volumes:
+        - configMap:
+            name: otel-collector-conf
+            items:
+              - key: otel-collector-config
+                path: otel-collector-config.yaml
+          name: otel-collector-config-vol
+---
+
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  labels:
+    serviceapp: otel-collector
+  name: otel-collector
+  namespace: keptn-lifecycle-controller-system
+spec:
+  endpoints:
+    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      interval: 30s
+      port: metrics
+  namespaceSelector:
+    matchNames:
+      - keptn-lifecycle-controller-system
+  selector:
+    matchLabels:
+      app: opentelemetry

operator/config/rbac/auth_proxy_service.yaml

@@ -11,5 +11,9 @@ spec:
     port: 8443
     protocol: TCP
     targetPort: https
+  - name: metrics
+    port: 2222
+    protocol: TCP
+    targetPort: 2222
   selector:
     control-plane: controller-manager