Standardize on tag representing the source host of the metrics

[danielnelson]

Feature Request

Proposal:

The host tag in Telegraf reports the hostname of the system running Telegraf, but when using a plugin that gathers from another service on the network, it can be helpful to know the hostname running the service.

Use source as the tag name and a domain name as the value. The name source mirrors Wavefront style (#4402). The host tag would be unchanged, which means some metrics would begin having both tags: host=example.org,source=example.org.

The new tag would, of course, be a convention only. Users would be free to add any value to this tag or tagexclude it completely. There wouldn't be any special syntax added to the telegraf.conf as was done with the host tag, and each plugin would be responsible for adding the appropriate value.

Current behavior:

Variety of tags are used, with a variety of value formats, here is a sample:

docker,engine_host=debian-stretch-docker
nginx,server=localhost,port=80
mongodb,hostname=127.0.0.1:27017
prometheus,url=http://example.org:9273/metrics

Desired behavior:

docker,source=debian-stretch-docker
nginx,source=localhost,port=80
mongodb,source=127.0.0.1,port=27017
prometheus,source=example.org,port=9273,resource=/metrics,scheme=http

Use case:

Unification of the tag name and format will allow lookup of metrics from a particular hostname.

[danielnelson]

cc @glinton @goller @phemmer @puckpuck @lpic10

[phemmer]

Been pondering this since the ping, and I'm still unsure about the name. I think standardizing on something is a good idea, as it is a very common occurrence that the metrics being recorded aren't about the host telegraf is running on. It's just the name source that has me a little hesitant. I just keep thinking of scenarios where telegraf is monitoring communication between 2 entities, one of them a source of the communication, one of them a destination. In this case, "source" might not represent where the metrics are being recorded from.
The only 2 terms that pop into mind are "origin", and "agent". Though I'm not too fond of either. "origin" kinda suffers the same problem as "source". The term "agent" is stolen from SNMP nomenclature, and isn't very self-explanatory.

[danielnelson]

I also think "agent" would cause confusion with the existing "host" tag, since we promote Telegraf as an agent. That said I'm not sure the proposed "source" and "host" are any better, and I could imagine either name being used to refer to the host running Telegraf. In many ways I would rather swap the meaning of agent/source with host, but of course it would be too disruptive so we won't be doing that.

I do slightly prefer "origin" over "source", but I don't think the difference is enough to justify differing from other tools.

[puckpuck]

I did some digging internally to find out why we have it called source.

The Wavefront platform treats host/source special versus other tags, and provides functionality around it. One of Wavefront's early customers, was interested in instrumenting entities beyond just hosts, and we certainly provided the functionality to do what they asked, however the semantics of calling the entity hosts could be perceived incorrectly.

After some discussions with the customer, the name source was proposed and adopted.

TBH we didn't consider origin at all, and though in some sense I can see it, when it comes to monitoring and metric data, sources seems more fitting. Data is often associated with having a source. (ie: the term data sources)

[danielnelson]

Let's use source as proposed, I will add some guidance to the CONTRIBUTING documentation. I don't plan to go through all the plugins, unfortunately I don't have the time, but I will try add them when I can.

[danielnelson]

One minor detail that I think we should provide guidance on. In the case that the source self reports its hostname, should we favor the host reported hostname or the connection hostname?

For example, let's say a plugin connects to https://10.1.2.3 and it returns a message {"hostname": "shodan", "value": 42}, should the plugin use the ip address "10.1.2.3" that was actually connected to or should it attempt to use the self reported hostname "shodan"?

[puckpuck]

self reported hostname should overrule. In this case shodan

[danielnelson]

A few more cases I've been thinking about:

• Should we include the source tag on local only metrics?

  cpu,host=xyzzy,source=xyzzy usage_idle=42
  

• Should we try to get a public hostname when connecting to a loopback address? Let's say you are connecting to a server with the address http://localhost:1234, would we convert that to source=localhost or special case loopback addresses and try the result of source=os.Hostname()?

My current thoughts are that we should add source on all metrics and use os.Hostname() for loopbacks.

[puckpuck]

Loopback can be described as: 127.0.0.1, localhost, ::1, and fe80::1%lo0 (ipv4 and ipv6)

I agree source should pull os.Hostname() if you use a loopback.

[danielnelson]

I think we can use IsLoopback to recognize all the forms.

[Hipska]

@danielnelson What is the state on this with the snmp input plugin? It currently still uses agent_address, should that be changed to source or should a user pick the sysName to be added as 'source'?

[powersj]

We are not currently planning a 2.0 release in the foreseeable future. New plugins should generally use the source tag to represent these types of values and it is up to the community and maintainers to ensure this occurs.

In terms of updating existing plugins, happy to see PRs with config options to switch the value, or for users to use the rename processor to produce a more consistent experience.

[Hipska]

Correct, so why closing this? I just fixed a few of those plugins not yet following this guideline. There are still others to be done.

[powersj]

You can open new issues as you put up PRs

[Hipska]

IMHO it makes more sense to let these PRs refer to this issue.