Working OpenTelemetry sidecar (without base nginx image)

docs/user-guide/nginx-configuration/annotations.md

@@ -806,6 +806,22 @@ sometimes need to be overridden to enable it or disable it for a specific ingres
 nginx.ingress.kubernetes.io/opentracing-trust-incoming-span: "true"
 ```
 
+### Enable Opentelemetry
+
+Opentelemetry can be enabled or disabled globally through the ConfigMap
+
+```yaml
+nginx.ingress.kubernetes.io/enable-opentelemetry: "true"
+```
+
+### Opentelemetry config
+
+Opentelemetry config
+
+```yaml
+nginx.ingress.kubernetes.io/opentelemetry-config: "/conf/otel-nginx.toml"
+```
+
 ### X-Forwarded-Prefix Header
 To add the non-standard `X-Forwarded-Prefix` header to the upstream request with a string value, the following annotation can be used:
 

docs/user-guide/nginx-configuration/configmap.md

@@ -130,6 +130,8 @@ The following table shows a configuration option's name, type, and the default v
 |[enable-opentracing](#enable-opentracing)|bool|"false"|
 |[opentracing-operation-name](#opentracing-operation-name)|string|""|
 |[opentracing-location-operation-name](#opentracing-location-operation-name)|string|""|
+|[enable-opentelemetry](#enable-opentelemetry)|bool|"false"|
+|[opentelemetry-config](#opentelemetry-config)|string|`Mandatory`|
 |[zipkin-collector-host](#zipkin-collector-host)|string|""|
 |[zipkin-collector-port](#zipkin-collector-port)|int|9411|
 |[zipkin-service-name](#zipkin-service-name)|string|"nginx"|
@@ -896,6 +898,15 @@ Specifies a custom name for the location span. _**default:**_ is empty
 
 For example, set to "HTTP $request_method $uri".
 
+## enable-opentelemetry
+
+Enables the nginx Opentelemetry extension. _**default:**_ is disabled
+
+## opentelemetry-config
+
+Opentelemetry exporters, processors, etc config. Mandatory, if enable-opentelemetry is enabled
+For more details see docs/user-guide/third-party-addons/opentelemetry.md 
+
 ## zipkin-collector-host
 
 Specifies the host to use when uploading traces. It must be a valid URL.

docs/user-guide/third-party-addons/opentelemetry.md

@@ -0,0 +1,126 @@
+# OpenTelemetry sidecar
+
+Enables distributed tracing using [OpenTelemetry](https://opentelemetry.io/).
+
+Using the OpenTelemetry sidecar enables the OpenTelemetry nginx module [OpenTelemetry nginx module](https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx) for the NGINX ingress controller.
+By default this feature is disabled.
+
+## Usage
+
+To enable the OpenTelemetry nginx module, the sidecar must be activated and configured.
+
+### Enable sidecar
+
+The OpenTelemetry sidecar load the OpenTelemetry module into the ingress-nginx container.
+To enable the sidecar use the [helm chart](https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx):
+```
+  extraModules: 
+  - name: opentelemetry
+    image: registry.k8s.io/ingress-nginx/opentelemetry
+```
+<!-- TODO: correct image name -->
+
+Otherwise add to the controller deployment configuration:
+```
+spec:
+  template:
+    spec:
+      containers:
+          volumeMounts:
+            - name: modules
+              mountPath: /modules_mount
+      initContainers:
+        - name: opentelemetry
+          image: registry.k8s.io/ingress-nginx/opentelemetry
+          command: ['sh', '-c', '/usr/local/bin/init_module.sh']
+          volumeMounts:
+            - name: modules
+              mountPath: /modules_mount
+      volumes:
+        - name: modules
+          emptyDir: {}
+```
+
+### enable module
+
+To enable the Opentelemetry nginx module add to the configuration ConfigMap:
+```
+data:
+  enable-opentelemetry: "true"
+```
+
+The [OpenTelemetry C++ library](https://github.com/open-telemetry/opentelemetry-cpp), base of the OpenTelemetry nginx module, requires a configuration file for processor and exporter configuration. The configuration file is `mandatory`.
+```
+data:
+  opentelemetry-config: /conf/otel-nginx.toml
+```
+
+#### module configuration
+The module configuration for processors and exporters can be added by an additional ConfigMap:
+```
+{
+    "kind": "ConfigMap",
+    "apiVersion": "v1",
+    "metadata": {
+        "name": "nginx-otel",
+        "namespace": "ingress-nginx",
+        "creationTimestamp": null
+    },
+    "data": {
+        "otel-nginx.toml": "exporter = \"otlp\"\nprocessor = \"batch\"\n\n[exporters.otlp]\n# Alternatively the OTEL_EXPORTER_OTLP_ENDPOINT environment variable can also be used.\nhost = \"localhost\"\nport = 4317\n\n[processors.batch]\nmax_queue_size = 2048\nschedule_delay_millis = 5000\nmax_export_batch_size = 512\n\n[service]\nname = \"nginx-proxy\" # Opentelemetry resource name\n\n[sampler]\nname = \"AlwaysOn\" # Also: AlwaysOff, TraceIdRatioBased\nratio = 0.1\nparent_based = false\n"
+    }
+}
+```
+Currently this ConfigMap must be added manually.
+The configuration here is equal to [OpenTelemetry C++ usage description](https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx#usage)
+
+Additionally the `otel-nginx.toml` file must be mounted to the ingress-nginx container
+```
+spec:
+  template:
+    spec:
+      containers:
+          volumeMounts:
+            - name: otel-nginx
+              readOnly: true
+              mountPath: /conf
+      volumes:
+        - name: otel-nginx
+          configMap:
+            name: nginx-otel
+            items:
+              - key: otel-nginx.toml
+                path: otel-nginx.toml
+```
+
+### nginx configuration
+The OpenTelemetry nginx module provides several [nginx directives](https://github.com/open-telemetry/opentelemetry-cpp-contrib/tree/main/instrumentation/nginx#nginx-directives).
+
+`Comming soon.`
+
+### helm chart
+
+The easiest way to activate the sidecar is to use the [helm chart](https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx). The values.yaml file have to contain
+
+```
+  extraModules: 
+  - name: opentelemetry
+    image: registry.k8s.io/ingress-nginx/opentelemetry:v20220415-controller-v1.2.0-beta.0-2-g81c2afd97@sha256:ce61e2cf0b347dffebb2dcbf57c33891d2217c1bad9c0959c878e5be671ef941
+
+  config:
+    enable-opentelemetry: true
+    opentelemetry-config: "/conf/otel-nginx.toml"
+
+  extraVolumeMounts:
+  - name: otel-nginx
+    readOnly: true
+    mountPath: /conf
+
+  extraVolumes:
+   - name: otel-nginx
+     configMap:
+      name: nginx-otel
+```
+
+## Collector
+As OpenTelemetry collector use for example [OpenTelemetry Collector Helm Chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector)

images/opentelemetry/rootfs/Dockerfile

@@ -42,4 +42,4 @@ RUN bash /opt/third_party/build.sh -n
 FROM alpine:3.14.6
 COPY --from=base /opt/third_party/init_module.sh /usr/local/bin/init_module.sh
 COPY --from=nginx /etc/nginx/modules /etc/nginx/modules
-COPY --from=nginx /opt/third_party/install/lib /etc/nginx/modules
+COPY --from=nginx /opt/third_party/install/lib /otel/lib

images/opentelemetry/rootfs/init_module.sh

@@ -18,5 +18,8 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
-mkdir -p /modules_mount/etc/nginx/modules
-cp -R /etc/nginx/modules /modules_mount/etc/nginx/modules
+for path in /etc/nginx/modules /otel/lib; 
+do 
+    mkdir -p  /modules_mount${path}
+    cp -R $path/* /modules_mount${path}/; 
+done

internal/ingress/annotations/opentelemetry/main.go

@@ -0,0 +1,75 @@
+/*
+Copyright 2019 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package opentelemetry
+
+import (
+	networking "k8s.io/api/networking/v1"
+	"k8s.io/klog/v2"
+
+	"k8s.io/ingress-nginx/internal/ingress/annotations/parser"
+	"k8s.io/ingress-nginx/internal/ingress/resolver"
+)
+
+type opentelemetry struct {
+	r resolver.Resolver
+}
+
+// Config contains the configuration to be used in the Ingress
+type Config struct {
+	OpenTelemetryEnabled	bool 	`json:"enabled"`
+	OpenTelemetryConfig		string 	`json:"config"`
+}
+
+// Equal tests for equality between two Config types
+func (bd1 *Config) Equal(bd2 *Config) bool {
+
+	if bd1.OpenTelemetryEnabled != bd2.OpenTelemetryEnabled {
+		return false
+	}
+
+	if bd1.OpenTelemetryConfig != bd2.OpenTelemetryConfig {
+		return false
+	}
+
+	return true
+}
+
+
+func NewParser(r resolver.Resolver) parser.IngressAnnotation {
+	return opentelemetry{r}
+}
+
+// Parse parses the annotations to look for opentelemetry configurations
+func (c opentelemetry) Parse(ing *networking.Ingress) (interface{}, error) {
+	var err error
+	config := &Config{}
+
+	config.OpenTelemetryEnabled, err = parser.GetBoolAnnotation("enable-opentelemetry", ing)
+	if err != nil {
+		config.OpenTelemetryEnabled = false
+	}
+
+	if config.OpenTelemetryEnabled {
+		config.OpenTelemetryConfig, err = parser.GetStringAnnotation("opentelemetry-config", ing)
+		if err != nil {
+			klog.Errorf("OpenTelementry config file (opentelemetry_config) must be set - %v.", err)
+			return nil, err
+		}
+	}
+
+	return config, nil
+}