Support telegraf plugins in Collector

[mxiamxia]

Telegraf has many input plugins which have been adopted & tested in many monitoring products. In OTel collector, we're trying to develop some of similar receivers/plugins(hostmetrics, vmmetrics) which exist in Telegraf(procstat, system, win_perf_counters) and new plugins in OTel provides less kind of metrics than Telegraf. Do we have plan to integrate Telegraf plugins in OTel? I see some discussion in the Gitter mentioned it is a roadmap item but want to confirm that.

[bogdandrutu]

This should be added to the collector-contrib repo. Also you can get inspired from what Splunk/SignalFx did with telegraf or propose your own design:
https://docs.signalfx.com/en/latest/integrations/agent/monitors/telegraf-procstat.html
https://docs.signalfx.com/en/latest/integrations/agent/monitors/telegraf-win_perf_counters.html

[jrcamp]

IMO a Telegraf output plugin should be written that can send data to the collector in some format the collector supports. Having a "meta agent" that wraps other arbitrary collector has numerous usability and technical issues. By the time you fix those issues and the maintenance cost associated with it you could have written a native monitor.

[mxiamxia]

This should be added to the collector-contrib repo. Also you can get inspired from what Splunk/SignalFx did with telegraf or propose your own design:
https://docs.signalfx.com/en/latest/integrations/agent/monitors/telegraf-procstat.html
https://docs.signalfx.com/en/latest/integrations/agent/monitors/telegraf-win_perf_counters.html

Thanks, but I am still curious why we don't port some of essential plugins (procstat, system, etc) from Telegraf to collector-core repo instead of re-implementing it again(like vmmetrics) in the core. We'll add the non-essential ones we need to collector-contrib repo.

[jrcamp]

Thanks, but I am still curious why we don't port some of essential plugins (procstat, system, etc) from Telegraf to collector-core repo instead of re-implementing it again(like vmmetrics) in the core. We'll add the non-essential ones we need to collector-contrib repo.

The metric data model between OT and telegraf are a bit different. We want to have consistent metadata across all OT metrics. In many cases telegraf uses a library called gopsutil which we are using as well. We've used some library code from telegraf like windows performance counters. Why call into telegraf and have it wrap things in its own format when it just in turn calls gopsutil? We can call it directly.

@james-bebbington is the one actually implementing the host metrics so I'll let him correct me if I got any of the details wrong.

[james-bebbington]

Sorry I'm a bit late to this, but yes that's correct. From my point of view/understanding:

For the Host Metrics receiver, we are focusing on adding a core set of OS-indepedent (as much as possible) metrics to assess host health - I'm not sure if it will end up being as exhaustive as procstat. For now, if you want to obtain those metrics, one option is to write a Telegraf output plugin as @jrcamp suggested above, or another option is to make use of any equivalent prometheus exporters via the Collector's existing prometheus receiver. The idea has been floated to allow the Collector to be configured to manage running prometheus exporter plugins directly (out of process), but that idea is only in the initial thought / planning phase.

While I'm not directly porting plugins from Telegraf or similar apps for the reasons @jrcamp outlined above, in most cases we are using the same core code, e.g. gopsutil (with a few exceptions to avoid expensive methods of obtaining metrics such as WMI on Windows). I'm planning on creating a PR shortly that brings in some code from Telegraf (that wraps win) rather then re-implementing it in the Collector.

[jrcamp]

Closing as we've decided to implement native receivers. If a user needs to use telegraf they can run telegraf and configure their output plugins to write to the collector in a supported format.