Add input to collect http data using any parser

[davidbirdsong]

We have services that have implemented native influxdb data formats and expose them at scrapable endpoints. Instead of pushing these values directly to influxdb, we want telegraf to grab them which gives us all the flexibility that telegraf provides: prometheus scrape endpoint, kafka push, nsq, or push to influxdb.

Is there no way to scrape influx's native format?

[OrangeTux]

I don't know whether I understand you correct or not, but you want Telegraf call HTTP endpoints that return data formatted according to InfluxDB's line protocol?

You could use the exec input plugin.

[[inputs.exec]]
 commands = ["curl example.com"]
 data_format = "influx"

[davidbirdsong]

Yes.

I'm sure it sounds odd at first blush, but I'll point out that telegraf is useful beyond collection and forwarding of metrics. It's the best so far, IMO, at providing a metrics router that allows external tools to intermingle data that most services or libraries don't want or need to be concerned with. For instance, we use ansible to drop data into the telegraf include dirs that merges inventory data like machines groups, tags, state, and other k/v data into metrics as they flow through--sub in any other config management or service discovery for similar approaches.

I was originally looking to heka for it's general purpose message router to mix in arbitrary data, but telegraf reads and writes to so many of the systems we use and in a metrics format that makes the most sense.

Given this approach, it makes sense for services to choose the influx line format regardless of whether they choose to push or pull.

something like

[[inputs.httpprobe]]
 data_format = "influx"
  servers = [
    "http://my.service.com/_stats"
  ]

I'm happy to whip up a PR in the next week if it sounds like an acceptable idea.

[OrangeTux]

I don't quite understand this line:

it makes sense for services to choose the influx line format regardless of whether they choose to push or pull.

An input plugin fetches metrics from a source every once in a while. I think this is what you mean with 'pull'. The exec plug is an input plugin.

A service input plugin is a special type of input plugin which runs in the background. This type of plugin can be used by externel tools to push metrics to. An example is the TCP Listener. This service input plugin listens on a port for metrics. Would this plugin, or a new HTTP Listener plugin, solve your problem?

[davidbirdsong]

On Wed, Mar 9, 2016 at 1:32 AM Auke Willem Oosterhoff <
[email protected]> wrote:

I don't quite understand this line:

it makes sense for services to choose the influx line format regardless of
whether they choose to push or pull.

An input plugin fetches metrics from a source every once in a while. I
think this is what you mean with 'pull'. The exec plug is an input
plugin.

A service input plugin is a special type of input plugin which runs in the
background. This type of plugin can be used by externel tools to push
metrics to. An example is the TCP Listener
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/tcp_listener.
This service input plugin listens on a port for metrics. Whould this
plugin, or a new HTTP Listener plugin, solve your problem?

Sorry, in my case, 'service' means a server we at imgix authored and want
telegraf to scrape periodically. We may be the first to export influx line
metrics from our own code as an endpoint needing to be scraped but I think
others might want to do this too with their own services given an example.
I realize that scraping is more prometheus style, but telegraf seems great
for mixing both pull/push metrics exporting methods.

—
Reply to this email directly or view it on GitHub
#813 (comment)
.

[sparrc]

this seems fine to me, basically you want the httpjson plugin to be able to scrape any data format, correct?

You should be able to take the httpjson plugin, rename it something like http_scrape, and have it use the generic parsers like the exec plugin does (including json). We could then deprecate the httpjson plugin in favor of this new one.

[davidbirdsong]

Yeah, that sounds good. Something like http_scrape that a user's endpoint responds json as the default for backwards compatibility, but telegraf can set Accept-Encoding to specify something more specific like influx-line and maybe in the future something like protobuf.

[danielnelson]

I think we should just name this http instead of http_scrape, my reasoning is that scrape makes me think we are processing unstructured html and not primarily targeting APIs.

[danielnelson]

Added in #3546 thanks to @grange74