telegraf resources consumption is growing over time

[romfreiman]

Running a telegraf vs collectd comparison over time, with similar plugins enabled (process + statsd).
Seems that telegraf resources (both memory and cpu) keep on growing over time

Relevant telegraf.conf:

[global_tags]
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "10s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
debug = true
quiet = false
logfile = ""
hostname = ""
omit_hostname = false
[[outputs.influxdb]]
retention_policy = ""
write_consistency = "any"
timeout = "5s"
[[inputs.cpu]]
percpu = true
totalcpu = true
collect_cpu_time = false
[[inputs.disk]]
ignore_fs = ["tmpfs", "devtmpfs"]
[[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]
[[inputs.docker]]
timeout = "5s"
total = true
[[inputs.procstat]]
pattern="collectd"
[[inputs.procstat]]
pattern="telegraf"
[[inputs.procstat]]
pattern="consul"
[[inputs.procstat]]
pattern="strato-ntp-client"
[[inputs.procstat]]
pattern="/usr/bin/neutron-metadata-agent"
[[inputs.procstat]]
pattern="openvswitch"
[[inputs.procstat]]
pattern="/usr/bin/neutron-dhcp-agent"
[[inputs.procstat]]
pattern="/usr/bin/neutron-l3-agent"
[[inputs.procstat]]
pattern="/usr/bin/nova-compute"
[[inputs.procstat]]
pattern="/usr/bin/nova-cert"
[[inputs.procstat]]
pattern="/usr/bin/nova-consoleauth"
[[inputs.procstat]]
pattern="mancala.management.garbagecollector"
[[inputs.procstat]]
pattern="mancala.management.service"
[[inputs.procstat]]
pattern="mancalaServer.bin"
[[inputs.procstat]]
pattern="mancalaClient.bin"
[[inputs.procstat]]
pattern="/usr/bin/osmosis"
[[inputs.procstat]]
pattern="strato.rabbitmq.main"
[[inputs.procstat]]
pattern="strato.noded"
[[inputs.procstat]]
pattern="strato.snld"
[[inputs.procstat]]
pattern="strato.logdump"
[[inputs.procstat]]
pattern="strato.runtime.nrad"
[[inputs.procstat]]
pattern="mancala.objectstore.service.main"
[[inputs.procstat]]
pattern="mancala.management.externalstorage.main"
[[inputs.procstat]]
pattern="policy_store"
[[inputs.procstat]]
pattern="libvirtd"
[[inputs.procstat]]
pattern="lttng"
[[inputs.procstat]]
pattern="stratonet.agent"
[[inputs.procstat]]
pattern="filebeat"
[[inputs.procstat]]
pattern="nodedapi"
[[inputs.procstat]]
pattern="/usr/bin/neutron-server"
[[inputs.procstat]]
pattern="/usr/bin/nova-scheduler"
[[inputs.procstat]]
pattern="/usr/bin/nova-conductor"
[[inputs.procstat]]
pattern="/usr/bin/nova-api"
[[inputs.procstat]]
pattern="/usr/bin/nova-novncproxy"
[[inputs.procstat]]
pattern="strato-ntp-server"
[[inputs.procstat]]
pattern="bin/beam.smp"
[[inputs.procstat]]
pattern="melet-api"
[[inputs.procstat]]
pattern="virtual-nb"
[[inputs.procstat]]
pattern="virtual-dr"
[[inputs.procstat]]
pattern="virtual-maestro"
[[inputs.procstat]]
pattern="strato.oort.main"
[[inputs.procstat]]
pattern="/usr/bin/cinder-api"
[[inputs.procstat]]
pattern="/usr/bin/cinder-scheduler"
[[inputs.procstat]]
pattern="/usr/bin/cinder-volume"
[[inputs.procstat]]
pattern="keystone"
[[inputs.procstat]]
pattern="harbour"
[[inputs.procstat]]
pattern="strato.kuiper"
[[inputs.procstat]]
pattern="northbound"
[[inputs.procstat]]
pattern="strato.runtime.grad"
[[inputs.procstat]]
pattern="strato.gargantua"
[[inputs.procstat]]
pattern="strato.syncer.main"
[[inputs.procstat]]
pattern="clusterd"
[[inputs.procstat]]
pattern="mysqld"
[[inputs.procstat]]
pattern="influxd"
[[inputs.procstat]]
pattern="mancala.dr.service.main"
[[inputs.procstat]]
pattern="object"
[[inputs.procstat]]
pattern="ovsdb-server"
[[inputs.procstat]]
pattern="ovs-vswitchd"
[[inputs.procstat]]
pattern="qemu-kvm"
[[inputs.statsd]]
service_address = ":8125"

System info:

Telegraf v1.2.1 (git: release-1.2 3b6ffb3)
centos7.2,
Linux stratonode0.node.strato 3.10.0-327.10.1.el7.strato0004.46fa8ebfb529.x86_64 #1 SMP Sun Mar 27 15:13:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Working vs influxdb container (influx v1.2).

Any explanation? I really would like to start using telegraf, but want to be sure regrading its behavior.

[danielnelson]

This should be fixed in the upcoming version 1.3.0, it was caused by the procstat plugin. Can you check with the nightly build to be sure https://dl.influxdata.com/telegraf/nightlies/telegraf-nightly.x86_64.rpm?

[romfreiman]

Deployed. Ill update tomorrow.

[romfreiman]

Seems to be much better 👍

@danielnelson can you point me to the relevant fix please? Also, when 1.3.0 expected to be released?

[danielnelson]

Fixed in #2540, the 1.3.0 Milestone has the items still remaining for the next release.