Error writing string fields to stackdriver output

[ihsw]

Relevant telegraf.conf:

# # Configuration for Google Cloud Stackdriver to send metrics to
[[outputs.stackdriver]]
#   # GCP Project
  project = "$PROJECT_ID"
#
#   # The namespace for the metric descriptor
  namespace = "telegraf"

System info:

$ telegraf --version
Telegraf 1.9.2 (git: HEAD dda80799)

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.1 LTS
Release:	18.04
Codename:	bionic

[Include Telegraf version, operating system name, and other relevant details]

Steps to reproduce:

1. Uncomment outputs.stackdriver config
2. Ensure GCE VM has Monitoring Metric Writer IAM role

Expected behavior:

Logs are pushed to StackDriver

Actual behavior:

Error when starting Telegraf:

2019-01-18T20:58:11Z E! [agent] Error writing to output [stackdriver]: rpc error: code = InvalidArgument desc = Field metricDescriptor.valueType had an invalid value of "STRING": When creating metric custom.googleapis.com/telegraf/system/uptime_format: the value type is not supported for custom metrics.

Additional info:

Not sure what else to do to get this working.

[danielnelson]

As a temporary workaround, you could use the fielddrop metric filtering parameter to drop any string fields. This should clear up the error but may be tedious if you have many string fields.

[[outputs.stackdriver]]
  fielddrop = ["uptime_format"]

[ihsw]

@danielnelson that worked! Thank you so much.

[ihsw]

@danielnelson thank you for fixing this! Also, hopefully StackDriver's Metric pricing becomes more reasonable. Feeding Telegraf telemetry into StackDriver can be pricey per GCE instance.

One might consider using Telegraf+StackDriver with per-minute resolution rather than per-second, to significantly reduce costs.

[danielnelson]

It seems that Stackdriver supports 1m resolution max for custom metrics, so you will want to reduce the collection interval or aggregate the metrics before sending them to Stackdriver.

There are some other limitations I have found, which lead me to believe that it makes the most sense to send only certain metrics to Stackdriver while pushing the bulk of the data to InfluxDB. I just added this note to the documentation: https://github.com/influxdata/telegraf/tree/master/plugins/outputs/stackdriver#restrictions

[ihsw]

@danielnelson these are excellent solutions. I have also noticed StackDriver barfing errors when receiving data points out-of-order. The only remedy was to restart telegraf and be (un)happy with a gap in data.

The basistats aggregator looks excellent.