Telegraf exporter and receiver #2951
Comments
This was referenced Mar 31, 2021
2 tasks
|
@tigrannajaryan this is a simple tracking issue, you may assign it to me. |
tigrannajaryan
pushed a commit
that referenced
this issue
May 4, 2021
This change adds an exporter for [InfluxData line protocol](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/), the native wire transfer protocol for [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) and [InfluxDB](https://www.influxdata.com/products/influxdb/). The conversion logic lives in [package `otel2influx`](https://github.com/influxdata/influxdb-observability/tree/main/otel2influx), and is also imported by the [proposed Telegraf OpenTelemetry input plugin](influxdata/telegraf#9077). The objective of this strategy is to create a canonical mapping, and an open-source implementation, of OpenTelemetry <-> Telegraf/InfluxDB translation. I am not proposing that `otel2influx`, in its current state, be the canonical mapping. Let's discuss that topic separate from this exporter. One shortcoming of this approach is that both `otel2influx` and `opentelemetry-collector` package their own generated protocol buffer objects. This means that every OTLP protobuf object is serialized and deserialized in this exporter. I propose that `opentelemetry-collector` instead depend on `opentelemetry-proto-go`, for the benefit of this exporter and other third-party packages. ```golang // r is a protobuf from otel2influx // td is a protobuf wrapper from otelcol var r otlpcollectortrace.ExportTraceServiceRequest if protoBytes, err := td.ToOtlpProtoBytes(); err != nil { return err } else if err = proto.Unmarshal(protoBytes, &r); err != nil { return err } ``` **Link to tracking Issue:** #2951 **Testing:** Little testing so far. The translation logic should be tested in [package `otel2influx`](https://github.com/influxdata/influxdb-observability/tree/main/otel2influx) (but is not currently). **Documentation:** README.md included in new exporter.
tigrannajaryan
pushed a commit
that referenced
this issue
May 11, 2021
This change adds a receiver for [InfluxData line protocol](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/), the native wire transfer protocol for [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) and [InfluxDB](https://www.influxdata.com/products/influxdb/). The conversion logic lives in [package `influx2otel`](https://github.com/influxdata/influxdb-observability/tree/main/influx2otel), and is also imported by the [proposed Telegraf OpenTelemetry output plugin](influxdata/telegraf#9228). The objective of this strategy is to create a canonical mapping, and an open-source implementation, of OpenTelemetry <-> Telegraf/InfluxDB translation. **Link to tracking Issue:** #2951 **Testing:** The translation logic is tested in [package `influx2otel`](https://github.com/influxdata/influxdb-observability/tree/main/influx2otel). **Documentation:** README.md included in new receiver.
alexperez52
referenced
this issue
in open-o11y/opentelemetry-collector-contrib
Aug 18, 2021
Prometheus remote write (PWR) exporter is supporting queued retry out of the box, but due to the strict requirements of PWR, the queue causes "out of order sample" errors. The queue won't be suitable for Prometheus unless it has a way to shard the data by timeseries. This is not going to break the existing users because if they have this feature enabled, they already can't use the PRW exporter correctly. Fixes #2949.
alexperez52
referenced
this issue
in open-o11y/opentelemetry-collector-contrib
Aug 18, 2021
…2974) * Enable queue for the Prometheus Remote Write Exporter internally This is a follow up to #2951. When we disable the queue completely, it causes the export to happen not in a consumer goroutine. Enable the queue internally that export gets its own goroutine not to block the entire collector pipeline. * Set a default queue size
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
I would like to use Telegraf agents and OpenTelemetry Collector agents together. I would like the translation to be consistent between implementations.
Describe the solution you'd like
OpenTelemetry Collector and Telegraf are both implemented in golang. Rather than let the two communities create their own mappings OTLP<->line-protocol, there should be one canonical mapping and one canonical golang implementation. In this way, improvements to otelcol benefit Telegraf, and visa versa.
A work-in-progress single package exists here.
With this single reference package, create the following:
Describe alternatives you've considered
Wrap otelcol in Telegraf execd input and execd output plugins. This is hard to package and configure.
Wrap Telegraf in an otelcol extension similar to the fluentbit extension. Hard to package and configure.
Fork portions of Telegraf into otelcol. This has already been done with windowsperfcountersreceiver. The result is more code, less attention to bug fixes, potential licensing issues later, etc.
Additional context
Telegraf is an open-source metrics pipeline agent, similar to the OpenTelemetry Collector. Telegraf has hundreds of input, output, and other plugins and is fairly popular today. Connecting these two tools would foster OpenTelemetry growth.
Telegraf is related to InfluxDB (a time series database; both are InfluxData products), but does not depend on InfluxDB.
Proposed Telegraf input plugin for OTLP/gRPC
The text was updated successfully, but these errors were encountered: