Datadog Output Plugin Appends "Value" to Stat Names

[feelobot]

[feelobot]

This started after the new telegraf release 0.10.0, updating to 0.10.2 did not fix

[sparrc]

This was supposed to be fixed by #604, are you 100% sure the new values coming into datadog still have "value" appended? I don't quite see how that's possible with the current code.

[feelobot]

yup those metrics were from yesterday, we switched to using dogstatsd and it works fine now

[sparrc]

So you were sending metrics into Telegraf statsd and then forwarding to datadog? Can you post your config please?

[sparrc]

@feelobot I suspect you may have unintentionally been running with the wrong version, the code in question was fixed in 0.10.2 and I can't reproduce.

[feelobot]

its just a standard statsd input, datadog output

# Telegraf configuration

# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared plugins.

# Even if a plugin has no configuration, it must be declared in here
# to be active. Declaring a plugin means just specifying the name
# as a section with no variables. To deactivate a plugin, comment
# out the name and any variables.

# Use 'telegraf -config telegraf.toml -test' to see what metrics a config
# file would generate.

# One rule that plugins conform to is wherever a connection string
# can be passed, the values '' and 'localhost' are treated specially.
# They indicate to the plugin to use their own builtin configuration to
# connect to the local system.

# NOTE: The configuration has a few required parameters. They are marked
# with 'required'. Be sure to edit those to make this configuration work.

# Tags can also be specified via a normal map, but only one form at a time:
[tags]
  region = "<%= @region %>"
  account = "<%= @account %>"
  chef_role = "<%= @chef_role %>"
  instance_id = "<%= @instance_id %>"
# Configuration for telegraf agent
[agent]
  # Default data collection interval for all plugins
  interval = "10s"
  # Rounds collection interval to 'interval'
  # ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  # Default data flushing interval for all outputs. You should not set this below
  # interval. Maximum flush_interval will be flush_interval + flush_jitter
  flush_interval = "10s"
  # Jitter the flush interval by a random amount. This is primarily to avoid
  # large write spikes for users running a large number of telegraf instances.
  # ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "5s"

  # Run telegraf in debug mode
  debug = false
  # Override default hostname, if empty use os.Hostname()
  hostname = ""


###############################################################################
#                                  OUTPUTS                                    #
###############################################################################

[outputs]

# Configuration for the AMQP server to send metrics to
[[outputs.influxdb]]
  # The full HTTP or UDP endpoint URL for your InfluxDB instance
  # Multiple urls can be specified for InfluxDB cluster support.
  # urls = ["udp://localhost:8089"] # UDP endpoint example
  urls = ["<%= @influx_host %>"] # required
  # The target database for metrics (telegraf will create it if not exists)
  database = "<%= @influx_db %>" # required
  # Precision of writes, valid values are n, u, ms, s, m, and h
  # note: using second precision greatly helps InfluxDB compression
  precision = "s"

  # Connection timeout (for the connection with InfluxDB), formatted as a string.
  # If not provided, will default to 0 (no timeout)
  # timeout = "5s"
  username = "<%= @influx_user %>"
  password = "<%= @influx_pass %>"
  # Set the user agent for HTTP POSTs (can be useful for log differentiation)
  # user_agent = "telegraf"
  # Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
  # udp_payload = 512

[[outputs.datadog]]
  # Datadog API key
  apikey = "<%= @datadog_api_key %>" # required.

###############################################################################
#                                  PLUGINS                                    #
###############################################################################

###############################################################################
#                              SERVICE PLUGINS                                #
###############################################################################
# Statsd Server
[[inputs.statsd]]
  # Address and port to host UDP listener on
  service_address = ":8125"
  # Delete gauges every interval (default=false)
  delete_gauges = false
  # Delete counters every interval (default=false)
  delete_counters = false
  # Delete sets every interval (default=false)
  delete_sets = false
  # Delete timings & histograms every interval (default=true)
  delete_timings = true
  # Percentiles to calculate for timing & histogram stats
  percentiles = [90]

  # templates = [
  #     "cpu.* measurement*"
  # ]

  # Number of UDP messages allowed to queue up, once filled,
  # the statsd server will start dropping packets
  allowed_pending_messages = 10000

  # Number of timing/histogram values to track per-measurement in the
  # calculation of percentiles. Raising this limit increases the accuracy
  # of percentiles but also increases the memory usage and cpu time.
  percentile_limit = 1000

[sparrc]

I tried with this config and couldn't get it to reproduce on master or 0.10.2:

[feelobot]

did you try using the binary inside http://get.influxdb.org/telegraf/telegraf-0.10.2-1_linux_amd64.tar.gz /usr/bin

[feelobot]

Also how is your datadog so dark and pretty

[sparrc]

heh I use the "Dark Reader" chrome extension with Inconsolata font :-)

https://chrome.google.com/webstore/detail/dark-reader/eimadpbcbfnmbkopoojfekhnkhdbieeh?hl=en

I was building 0.10.2 from source, I'll try the binary

[sparrc]

Tried with the 0.10.2 binary on a fresh VM and still the same result,

Telegraf output:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ ./usr/bin/telegraf -config ./telegraf.conf -input-filter statsd -output-filter datadog -debug
2016/02/09 21:25:16 Attempting connection to output: datadog
2016/02/09 21:25:16 Successfully connected to output: datadog
2016/02/09 21:25:16 Starting Telegraf (version 0.10.2)
2016/02/09 21:25:16 Loaded outputs: datadog
2016/02/09 21:25:16 Loaded inputs: statsd
2016/02/09 21:25:16 Tags enabled: host=vagrant-ubuntu-trusty-64
2016/02/09 21:25:16 Agent Config: Interval:10s, Debug:true, Quiet:false, Hostname:"vagrant-ubuntu-trusty-64", Flush Interval:10s
2016/02/09 21:25:20 Started the statsd service on :8125
2016/02/09 21:25:20 Gathered metrics, (10s interval), from 1 inputs in 360.937µs
2016/02/09 21:25:20 Statsd listener listening on:  [::]:8125
[...]
> issue.672.test,host=vagrant-ubuntu-trusty-64,metric_type=timing 90_percentile=50,count=1i,lower=50,mean=50,stddev=0,upper=50 1455053230003396413
2016/02/09 21:27:10 Gathered metrics, (10s interval), from 1 inputs in 84.268µs
2016/02/09 21:27:10 Wrote 1 metrics to output datadog in 372.712728ms

Statsd metric sent:

echo "issue.672.test:50|ms" | nc -u -w1 localhost 8125

[feelobot]

ill try to make a screen cast for you later on one of our machines, from the telegraf debug output everything looks fine, its only in datadog we see the metrics coming in all jacked up.