This is a basic telemetry example for exporting golang telemetry to some grafana cloud instance.
This example features:
- A simple echo golang http server
- A one-shot client program, that fires 5 requests and then exits
- An OTEL Collector, that accumulates all telemetry alongside the apps before pushing it into Grafana Cloud
- A Grafana Alloy collector, that accumulates telemetry alongside the apps before pushing it into Grafana Cloud
Golang OTEL library currently provides stable support for Traces and Metrics. Logging support is in beta. You can check the current status of the logging support here.
The example provides a internal/metrics, internal/tracing and internal/logs packages with some commentary regarding the configuration of Telemetry in OTEL. All telemetry is exported into local collector instances via push using OTLP protocol over gRPC.
Logging is done via log/slog, a unified structured logging interface added to the standard library in go1.21. As an example, fanout handler for log/slog is added, to showcase that OTEL log bridge can be used for export alongside other logging syncs.
A local instance OTEL Collector is used to not bother with telemetry export security directly in the applications. In production such a Collector would probably be privately available somewhere near the application, and would manage additional telemetry augmentation and processing, as well as security - auth, certs, etc.
With recent introduction of Grafana Alloy (a distribution of OTEL Collector by Grafana Labs), example configuration is also provided for it. In production, a particular choice between OTEL Collector and Grafana Alloy would depend on the specific needs of the application and the organization.
In this example, OTEL also adds host metrics to the exported metrics data, allowing for infra resource tracking.
To not lose any unexported telemetry before finishing, both apps have basic graceful shutdown logic implemented.
The config for OpenTelemetry Collector and Grafana Alloy is mostly generated by the Grafana Cloud.
Consult this link in grafana docs for opentelemetry collector config to get your version.
Consult this link in grafana docs for grafana alloy config to get your version.