[[outputs.azure_monitor]] + [[inputs.rabbitmq]] = Bad Request from Azure

[raphaelquati]

Relevant telegraf.conf:

[agent]
  collection_jitter = "0s"
  debug = true
  flush_interval = "10s"
  flush_jitter = "0s"
  hostname = "telegraf-polling-service"
  interval = "30s"
  logfile = ""
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  omit_hostname = false
  precision = ""
  quiet = false
  round_interval = true

[[outputs.azure_monitor]]
  region = "eastus"
  resource_id = "/subscriptions/???/resourceGroups/????/providers/Microsoft.OperationalInsights/workspaces/?????"
  strings_as_dimensions = true
[[inputs.rabbitmq]]
  insecure_skip_verify = false
  name = "rabbitmq"
  password = "pass"
  url = "http://rabbitmq15672"
  username = "user"

System info:

Kubernetes 1.11
Telegraf version:1.8.3 (telegraf_s)
Using helm package from https://github.com/influxdata/tick-charts/tree/master/telegraf-s

Steps to reproduce:

1. Create a Service Principal
   az ad sp create-for-rbac --name TelegrafAKS --role="Monitoring Metrics Publisher" --scopes="/subscriptions/??/resourceGroups/??"

2. Setup environment variables (as described at https://github.com/influxdata/telegraf/blob/master/plugins/outputs/azure_monitor/README.md) or use helm package
   AZURE_CLIENT_ID
   AZURE_CLIENT_SECRET
   AZURE_TENANT_ID

3. Install helm package:
   helm install --name telegraf .\telegraf-s --namespace monitor -f .\values.yaml

values.yaml:

service:
  enabled: false

env:
- name: "AZURE_CLIENT_ID"
  value: ""
- name: "AZURE_CLIENT_SECRET"
  value: ""
- name: "AZURE_TENANT_ID"
  value: ""

config:
  agent:
    debug: true
  outputs:
    - azure_monitor:
        region: "eastus"
        resource_id: "/subscriptions/???/resourceGroups/????/providers/Microsoft.OperationalInsights/workspaces/????"
        strings_as_dimensions: true
  inputs:
    - rabbitmq:
        url: "http://rabbitmq:15672"
        name: "rabbitmq"
        username: "user"
        password: "pass"
        insecure_skip_verify: false

Expected behavior:

Run fine

Actual behavior:

Error

Additional info:

If I change the input from rabbitmq to cpu, it runs fine.
Changing strings_as_dimensions from true to false: nothing happens

Logs;

2018/11/29 18:48:05 I! Using config file: /etc/telegraf/telegraf.conf
2018-11-29T18:48:05Z D! Attempting connection to output: azure_monitor
2018-11-29T18:48:05Z D! Writing to Azure Monitor URL: https://eastus.monitoring.azure.com/subscriptions/????/resourceGroups/????/providers/Microsoft.OperationalInsights/workspaces/?????/metrics
2018-11-29T18:48:05Z D! Successfully connected to output: azure_monitor
2018-11-29T18:48:05Z I! Starting Telegraf 1.8.3
2018-11-29T18:48:05Z I! Loaded inputs: inputs.rabbitmq
2018-11-29T18:48:05Z I! Loaded aggregators:
2018-11-29T18:48:05Z I! Loaded processors:
2018-11-29T18:48:05Z I! Loaded outputs: azure_monitor
2018-11-29T18:48:05Z I! Tags enabled: host=telegraf-polling-service
2018-11-29T18:48:05Z I! Agent Config: Interval:30s, Quiet:false, Hostname:"telegraf-polling-service", Flush Interval:10s
2018-11-29T18:48:40Z D! Output [azure_monitor] buffer fullness: 0 / 10000 metrics.
2018-11-29T18:48:50Z D! Output [azure_monitor] buffer fullness: 0 / 10000 metrics.
2018-11-29T18:49:00Z D! Output [azure_monitor] buffer fullness: 341 / 10000 metrics.
2018-11-29T18:49:00Z E! Error writing to output [azure_monitor]: failed to write batch: [400] 400 Bad Request

[raphaelquati]

Related?

[danielnelson]

Thanks, can you run telegraf --input-filter rabbitmq --test and paste the output (replacing characters one to one where needed)?

[raphaelquati]

I've compiled the sources, and modified azure_monitor plugin to output the entire error reponse:

2018-11-30T00:36:11Z E! [agent] Error writing to output [azure_monitor]: failed to write batch: [400] 400 Bad Request {"error":{"code":"BadRequest","message":"'dimValues' and 'dimNames' should have matching lengths. DimValues.Count = 7, DimNames.Count = 8.\r\n"}}

I've captured the buffer sent to Azure:

{"time":"2018-11-29T22:14:00-02:00","data":{"baseData":{"metric":"messages_publish_in","namespace":"Telegraf/rabbitmq_exchange","dimNames":["auto_delete","durable","exchange","host","internal","type","url","vhost"],"series":[{"dimValues":["false","true","amq.direct","telegraf-polling-service","false","direct","http://rabbitmq:15672","/"],"min":0,"max":0,"sum":0,"count":1}]}}}

I think the error is URL or VHOST. I'll test.

[raphaelquati]

The error is not related to tags URL or VHOST.

In translate function (which transforms tags into dimensions):

if tag.Key == "" || tag.Value == "" {
    continue
}

This prevents empty dimName/dimValue on azure metric.
But on the function Write, it uses the first metric (wilth all dimNames), but next metrics can have less dimensions (empty strings). In RabbitMQ, this happens with "Exchange" tag.

[raphaelquati]

Is this related? #3882

[danielnelson]

Based on the output from that issue it is common to have an exchange with no name, so I think we should just pass it along as an empty string if we can.

[raphaelquati]

Azure Metric doesn't support empty string as dimValue.

[danielnelson]

Closed by #5083

[IvanovOleg]

@raphaelquati how did you manage to add an additional error message?

[raphaelquati]

@raphaelquati how did you manage to add an additional error message?

I've change the code to dump the vars in functions like (

telegraf/plugins/outputs/azure_monitor/azure_monitor.go

Line 246 in 048ce7e

func (a *AzureMonitor) Write(metrics []telegraf.Metric) error {

)