From ae227f6da9087297d7612f7d046300644cc0366d Mon Sep 17 00:00:00 2001
From: Severin Neumann <neumanns@cisco.com>
Date: Fri, 1 Dec 2023 14:04:50 +0100
Subject: [PATCH] [blog] Receive any custom metric with the OpenTelemetry
 Collector (#3600)

Signed-off-by: svrnm <neumanns@cisco.com>
Co-authored-by: Fabrizio Ferri-Benedetti <fferribenedetti@splunk.com>
Co-authored-by: Patrice Chalin <chalin@users.noreply.github.com>
---
 content/en/blog/2023/any-metric-receiver.md | 492 ++++++++++++++++++++
 static/refcache.json                        |  12 +
 2 files changed, 504 insertions(+)
 create mode 100644 content/en/blog/2023/any-metric-receiver.md

diff --git a/content/en/blog/2023/any-metric-receiver.md b/content/en/blog/2023/any-metric-receiver.md
new file mode 100644
index 000000000000..388b7cde0186
--- /dev/null
+++ b/content/en/blog/2023/any-metric-receiver.md
@@ -0,0 +1,492 @@
+---
+title: Receive any custom metric with the OpenTelemetry Collector
+linkTitle: Any Metric Receiver
+date: 2023-11-30
+author: '[Severin Neumann](https://github.com/svrnm), Cisco'
+# prettier-ignore
+cSpell:ignore: carbonreceiver datapoint debugexporter enddate gomod helmuth noout openssl otlpexporter otlphttpexporter otlpreceiver ottl servername transformprocessor webserver
+---
+
+While OpenTelemetry (OTel) is here to help you with troubleshooting and handling
+the _"unknown unknowns"_, it is also instrumental for managing route tasks like
+monitoring system metrics, like disk usage, server availability or SSL
+certificate expiration dates. This can be achieved by utilizing any one of the
+[90+ receivers](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver)
+available for the [OpenTelemetry Collector](/docs/collector), such as the
+[Host Metrics Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver)
+or the [HTTP Check Receiver](/blog/2023/synthetic-testing/).
+
+But what if the available receivers don't meet your specific needs? Suppose you
+have a collection of shell scripts that provide custom metrics, and you want to
+export these to the OpenTelemetry Collector. You could write your own receiver,
+but this would require proficiency in Go.
+
+Before embarking on this path, consider examining the available receivers more
+closely: Some of them are capable of assimilating metrics in different
+formats—like
+[Carbon](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/carbonreceiver),
+[StatsD](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver),
+[InfluxDB](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/influxdbreceiver),
+[Prometheus](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver),
+and even
+[SNMP](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/snmpreceiver)—and
+integrating them into the OpenTelemetry ecosystem. With minor tweaks to your
+shell scripts, you can use one of these receivers to achieve your objective. For
+instance, the Carbon Receiver, with its simple
+[plaintext protocol](https://graphite.readthedocs.io/en/stable/feeding-carbon.html#the-plaintext-protocol),
+is ideal for use with shell scripts. Its protocol is incredibly straightforward:
+
+> The plaintext protocol is the most straightforward protocol supported by
+> Carbon. The data sent must be in the following format:
+> `<metric path> <metric value> <metric timestamp>`.
+
+## Example script: Check certificate expiration
+
+Consider the following shell script, which accepts a host name as an argument,
+and uses
+[`openssl s_client`](https://www.openssl.org/docs/manmaster/man1/openssl-s_client.html)
+to retrieve the certificate and compute the remaining time until certificate
+expiration:
+
+```shell
+#!/bin/bash
+HOST=${1}
+PORT=${2:-443}
+
+now=$(date +%s)
+str=$(echo q | openssl s_client -servername "${HOST}" "${HOST}:${PORT}" 2>/dev/null | openssl x509 -noout -enddate | awk -F"=" '{ print $2; }')
+if [[ "$(uname)" == "Darwin" ]] ; then
+  notAfter=$(date -j -f "%b %d %H:%M:%S %Y %Z" "${notAfterString}" +%s)
+else
+  notAfter=$(date -d "${notAfterString}" +%s)
+fi
+
+secondsLeft=$(($notAfter-$now))
+
+echo ${secondsLeft}
+```
+
+You can test this script as follows:
+
+```shell
+$ ./ssl_check.sh opentelemetry.io
+4357523
+```
+
+## Use Carbon's plaintext protocol
+
+To adapt this script to use Carbon's plaintext protocol, you'll need to modify
+the script's last few lines to output a metric in Carbon format:
+
+```shell {hl_lines=[12]}
+#!/bin/bash
+HOST=${1}
+PORT=${2:-443}
+
+now=$(date +%s)
+str=$(echo q | openssl s_client -servername "${HOST}" "${HOST}:${PORT}" 2>/dev/null | openssl x509 -noout -enddate | awk -F"=" '{ print $2; }')
+if [[ "$(uname)" == "Darwin" ]] ; then
+  notAfter=$(date -j -f "%b %d %H:%M:%S %Y %Z" "${notAfterString}" +%s)
+else
+  notAfter=$(date -d "${notAfterString}" +%s)
+fi
+
+secondsLeft=$(($notAfter-$now))
+
+metricPath="tls.server.not_after.time_left;unit=s"
+echo "${metricPath} ${secondsLeft} ${now}"
+```
+
+In doing so, the script will output `<metric path>` as
+`tls.server.not_after.time_left;unit=s`, the `<metric value>` as
+`${secondsLeft}`, and the `<metric timestamp>` as `${now}`.
+
+That's all we need to do to send our metric to the OpenTelemetry Collector with
+the Carbon Receiver enabled.
+
+## Receive any metric with the OTel Collector
+
+To test this, initiate an OpenTelemetry Collector using the following
+configuration::
+
+```yaml
+receivers:
+  carbon:
+    endpoint: localhost:8080
+    transport: tcp
+    parser:
+      type: plaintext
+      config:
+
+exporters:
+  debug:
+    verbosity: detailed
+
+service:
+  pipelines:
+    metrics:
+      receivers: [carbon]
+      exporters: [debug]
+```
+
+For instance, if you've saved this file as `collector-config.yml`, execute the
+following command:
+
+```console
+$ ./otelcol --config collector-config.yml
+2023-11-24T12:52:51.340+0100	info	service@v0.89.0/telemetry.go:85	Setting up own telemetry...
+2023-11-24T12:52:51.341+0100	info	service@v0.89.0/telemetry.go:202	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
+2023-11-24T12:52:51.341+0100	info	exporter@v0.89.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
+2023-11-24T12:52:51.341+0100	info	service@v0.89.0/service.go:143	Starting otelcol-any-metric...	{"Version": "1.0.0", "NumCPU": 10}
+2023-11-24T12:52:51.341+0100	info	extensions/extensions.go:34	Starting extensions...
+2023-11-24T12:52:51.342+0100	info	service@v0.89.0/service.go:169	Everything is ready. Begin running and processing data.
+```
+
+{{% alert title="Note" color="secondary" %}}
+
+For testing, you can use the
+[OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib)
+distribution, which includes all available receivers. However, in a production
+setting, you can
+[construct your own Collector](/docs/collector/custom-collector/) using the
+OpenTelemetry Collector Builder
+([`ocb`](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder)).
+Here's a suggested configuration:
+
+```yaml
+dist:
+  name: otelcol-any-metric
+  description: Custom OpenTelemetry Collector for receiving any kind of metric
+  output_path: ./
+
+exporters:
+  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.89.0
+  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.89.0
+  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.89.0
+
+processors:
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor
+      v0.89.0
+
+receivers:
+  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.89.0
+  - gomod:
+      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/carbonreceiver
+      v0.89.0
+```
+
+{{% /alert %}}
+
+With the OpenTelemetry Collector operational, open a secondary shell and
+transmit your metric to it:
+
+```shell
+./ssl_check.sh opentelemetry.io | nc 127.0.0.1 8080
+```
+
+The
+[Debug Exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/debugexporter)
+will display the metric on the console for you:
+
+```log
+2023-11-24T12:54:51.369+0100	info	ResourceMetrics #0
+Resource SchemaURL:
+ScopeMetrics #0
+ScopeMetrics SchemaURL:
+InstrumentationScope
+Metric #0
+Descriptor:
+     -> Name: tls.server.not_after.time_left
+     -> Description:
+     -> Unit:
+     -> DataType: Gauge
+NumberDataPoints #0
+Data point attributes:
+     -> unit: Str(s)
+StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
+Timestamp: 2023-11-24 11:54:51 +0000 UTC
+Value: 4356471
+	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
+```
+
+That's it! You can implement the same technique with any other shell script you
+have that reports custom metrics.
+
+## Fine tuning with the Transform Processor
+
+The Carbon Receiver split the `<metric path>` using `;` as a delimiter to
+extract the metric name (first item) and data point attributes (all other
+items). In our example, this means that the metric name will be
+`tls.server.not_after.time_left` and there will be the data point attribute
+`unit: Str(s)`.
+
+While being straightforward, this approach does not allow you to set any other
+details, like a [resource](/docs/concepts/resources/), a metric description, or
+particularly, a metric unit.
+
+However, the
+[Transform Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor)
+can help you with this. By incorporating OpenTelemetry Transformation Language
+(OTTL) statements into your `collector-config.yml`, you can convert the data
+point attribute `unit` into the metric's unit:
+
+```yaml {hl_lines=["13-19",24]}
+receivers:
+  carbon:
+    endpoint: localhost:8080
+    transport: tcp
+    parser:
+      type: plaintext
+      config:
+
+exporters:
+  debug:
+    verbosity: detailed
+
+processors:
+  transform:
+    metric_statements:
+      - context: datapoint
+        statements:
+          - set(metric.unit, attributes["unit"])
+          - delete_key(attributes, "unit")
+service:
+  pipelines:
+    metrics:
+      receivers: [carbon]
+      processors: [transform]
+      exporters: [debug]
+```
+
+Execute the `ssl_check.sh` once more:
+
+```shell
+./ssl_check.sh opentelemetry.io | nc 127.0.0.1 8080
+```
+
+Now, the Debug Exporter will also incorporate the unit into the metric
+descriptor:
+
+```text {hl_lines=[10]}
+2023-11-24T12:54:51.369+0100	info	ResourceMetrics #0
+Resource SchemaURL:
+ScopeMetrics #0
+ScopeMetrics SchemaURL:
+InstrumentationScope
+Metric #0
+Descriptor:
+     -> Name: tls.server.not_after.time_left
+     -> Description:
+     -> Unit: s
+     -> DataType: Gauge
+NumberDataPoints #0
+Data point attributes:
+     -> unit: Str(s)
+StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
+Timestamp: 2023-11-24 11:54:51 +0000 UTC
+Value: 4356471
+	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
+```
+
+If you wish to associate your metric with a
+[service](/docs/specs/semconv/resource/#service) that you've instrumented using
+OpenTelemetry, you can initially add `service.name` and `service.namespace` to
+your shell script as data point attributes:
+
+```shell
+metricName="tls.server.not_after.time_left;unit=s;service.name=otel-webserver;service.namespace=opentelemetry.io"
+echo "${metricName} ${secondsLeft} ${now}"
+```
+
+Next, add another OTTL statement to create a resource from those data point
+attributes:
+
+```yaml
+processors:
+  transform:
+    metric_statements:
+      - context: datapoint
+        statements:
+          - set(metric.unit, attributes["unit"])
+          - set(resource.attributes["service.name"], attributes["service.name"])
+          - set(resource.attributes["service.namespace"],
+            attributes["service.namespace"])
+          - delete_key(attributes, "unit")
+          - delete_key(attributes, "service.name")
+          - delete_key(attributes, "service.namespace")
+```
+
+Run the `ssl_check.sh` once again:
+
+```shell
+./ssl_check.sh opentelemetry.io | nc 127.0.0.1 8080
+```
+
+Now, the Debug Exporter will also include the resource with attributes
+`service.name` and `service.namespace`:
+
+```text
+2023-11-24T14:49:03.806+0100	info	ResourceMetrics #0
+Resource SchemaURL:
+Resource attributes:
+     -> service.name: Str(otel-webserver)
+     -> service.namespace: Str(opentelemetry.io)
+ScopeMetrics #0
+ScopeMetrics SchemaURL:
+InstrumentationScope
+Metric #0
+Descriptor:
+     -> Name: tls.server.not_after.time_left
+     -> Description:
+     -> Unit: s
+     -> DataType: Gauge
+NumberDataPoints #0
+StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
+Timestamp: 2023-11-24 13:49:03 +0000 UTC
+Value: 4349619
+	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
+```
+
+The Transform Processor and OTTL offer a wide range of capabilities. Learn more
+from:
+
+- [OTTL README.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md)
+- [OTTL Me Why Transforming Telemetry in the OpenTelemetry Collector Just Got Better](https://www.youtube.com/watch?v=uVs0oUV72CE),
+  a talk by [Tyler Helmuth](https://github.com/TylerHelmuth) and
+  [Evan Bradley](https://github.com/evan-bradley)
+
+With this, you are ready to receive any custom metric with the OpenTelemetry
+Collector!
+
+## Bonus: Use OTLP!
+
+While the use of the Carbon Receiver and the Transform Processor is a dependable
+method to gather custom metrics, it may seem unconventional to use an external
+format to import metrics into OpenTelemetry, especially when the
+[OpenTelemetry Protocol](/docs/specs/otlp/) (OTLP) provides everything you need.
+
+As an alternative to using the Carbon Receiver, you can also transmit a custom
+metrics using
+[OTLP JSON](https://github.com/open-telemetry/opentelemetry-proto/tree/main/examples):
+
+```shell
+#!/bin/bash
+URL=${1}
+PORT=${2:-443}
+
+now=$(date +%s)
+notAfterString=$(echo q | openssl s_client -servername "${URL}" "${URL}:${PORT}" 2>/dev/null | openssl x509 -noout -enddate | awk -F"=" '{ print $2; }')
+if [[ "$(uname)" == "Darwin" ]] ; then
+  notAfter=$(date -j -f "%b %d %H:%M:%S %Y %Z" "${notAfterString}" +%s)
+else
+  notAfter=$(date -d "${notAfterString}" +%s)
+fi
+
+secondsLeft=$(($notAfter-$now))
+
+data="
+{
+    \"resourceMetrics\": [
+      {
+        \"resource\": {
+          \"attributes\": [
+            {
+              \"key\": \"service.name\",
+              \"value\": {
+                \"stringValue\": \"${URL}\"
+              }
+            }
+          ]
+        },
+        \"scopeMetrics\": [
+          {
+            \"metrics\": [
+              {
+                \"name\": \"tls.server.not_after.time_left\",
+                \"unit\": \"s\",
+                \"description\": \"\",
+                \"gauge\": {
+                  \"dataPoints\": [
+                    {
+                      \"asInt\": ${secondsLeft},
+                      \"timeUnixNano\": ${now}000000000
+                    }
+                  ]
+                }
+              }
+            ]
+          }
+        ]
+      }
+    ]
+  }
+"
+curl -X POST -H "Content-Type: application/json" -d "${data}" -i localhost:4318/v1/metrics
+```
+
+Activate the OTLP Receiver in your `collectors-config`:
+
+```yaml
+receivers:
+  otlp:
+    protocols:
+      http:
+      grpc:
+exporters:
+  debug:
+    verbosity: detailed
+service:
+  pipelines:
+    metrics:
+      receivers: [otlp]
+      exporters: [debug]
+```
+
+Execute your updated `ssl_check.sh`:
+
+```shell
+./ssl_check.sh opentelemetry.io
+```
+
+This time, your metric will be presented with the correct unit set and the
+resource reported as defined in your JSON:
+
+```text
+2023-11-24T15:28:51.212+0100	info	ResourceMetrics #0
+Resource SchemaURL:
+Resource attributes:
+     -> service.name: Str(opentelemetry.io)
+ScopeMetrics #0
+ScopeMetrics SchemaURL:
+InstrumentationScope
+Metric #0
+Descriptor:
+     -> Name: tls.server.not_after.time_left
+     -> Description:
+     -> Unit: s
+     -> DataType: Gauge
+NumberDataPoints #0
+StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
+Timestamp: 2023-11-24 14:28:51 +0000 UTC
+Value: 4347231
+	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
+```
+
+Working with JSON in shell scripts isn't particularly desirable, as this example
+clearly demonstrates! While there are methods for improvement, you may
+ultimately find it more efficient to use a language such as Python or Node.js,
+or incorporate metrics (with gauge support) into your preferred OTel CLI tool!
+
+## Summary
+
+In this post you learned how to use a _catch-all_ receiver like the Carbon
+Receiver to feed any metric into your OpenTelemetry Collector. Use this approach
+when none of the available receivers meet your needs and you don't want to write
+your won receiver in Go.
+
+You learned how to send your metrics to the OpenTelemetry Collector directly
+using OTLP and `curl`. Use this approach when you cannot modify the pipelines of
+your OpenTelemetry Collector. It will also become a valid alternative to
+_catch-all_ receivers with the availability of command line tools that export
+metrics via OTLP.
diff --git a/static/refcache.json b/static/refcache.json
index b745b0580dd4..e0d50e08adc9 100644
--- a/static/refcache.json
+++ b/static/refcache.json
@@ -2019,6 +2019,10 @@
     "StatusCode": 200,
     "LastSeen": "2023-10-26T12:18:58.23681+02:00"
   },
+  "https://github.com/TylerHelmuth": {
+    "StatusCode": 200,
+    "LastSeen": "2023-11-24T16:20:25.682993+01:00"
+  },
   "https://github.com/Workiva/opentelemetry-dart": {
     "StatusCode": 200,
     "LastSeen": "2023-06-30T08:34:38.397928-04:00"
@@ -3983,6 +3987,10 @@
     "StatusCode": 200,
     "LastSeen": "2023-06-29T16:11:25.544037-04:00"
   },
+  "https://graphite.readthedocs.io/en/stable/feeding-carbon.html#the-plaintext-protocol": {
+    "StatusCode": 200,
+    "LastSeen": "2023-11-24T16:20:23.626654+01:00"
+  },
   "https://graphiteapp.org/": {
     "StatusCode": 200,
     "LastSeen": "2023-06-29T16:03:54.321095-04:00"
@@ -6423,6 +6431,10 @@
     "StatusCode": 206,
     "LastSeen": "2023-10-16T20:09:53.898397+02:00"
   },
+  "https://www.openssl.org/docs/manmaster/man1/openssl-s_client.html": {
+    "StatusCode": 200,
+    "LastSeen": "2023-11-24T16:20:23.86666+01:00"
+  },
   "https://www.oracle.com/technical-resources/articles/javase/jmx.html": {
     "StatusCode": 200,
     "LastSeen": "2023-06-30T08:52:25.052421-04:00"