Failed to send to Kinesis with missing partition tag

[DimaRabkin]

Relevant telegraf.conf:

[[outputs.kinesis]]

  region = "us-east-1"
  streamname = "MY_STREAM"

  data_format = "json"

  tagexclude = ["database"]

  [outputs.kinesis.partition]
    method = "tag"
    key = "task-name"

  [outputs.kinesis.tagpass]
    database = ["task_stats"]

System info:

Telegraf 1.6

Steps to reproduce:

1. Use the attached telegraf.conf
2. Try to send a metric to the telegraf's input without the tag "task-name"

Expected behavior:

As explained in https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kinesis#tag
It should send the record to Kinesis and use an empty string with partition key

Actual behavior:

It throws exceptions:

018-10-21T08:37:40Z E! Wrote a 2 point batch to Kinesis in 17.697303ms.
2018-10-21T08:37:40Z E! Wrote a 1 point batch to Kinesis in 19.77042ms.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis : You have configured a Partition using tag task-name which does not exist.
2018-10-21T08:37:50Z E! kinesis: Unable to write to Kinesis : InvalidParameter: 19 validation error(s) found.
- minimum field size of 1, PutRecordsInput.Records[0].PartitionKey.
- minimum field size of 1, PutRecordsInput.Records[1].PartitionKey.
- minimum field size of 1, PutRecordsInput.Records[2].PartitionKey.
- minimum field size of 1, PutRecordsInput.Records[3].PartitionKey.
- minimum field size of 1, PutRecordsInput.Records[4].PartitionKey.

[glinton]

Thanks, I believe this is an issue with the implementation rather than simply a docs bug, but I may be wrong.

[glinton]

Seems like an easy fix, add return "" after this if block and remove the log entry.

[danielnelson]

We should remove this log message, since it is not an error. Documentation needs to be updated to mention the fallback to random uuid if configured. Also, we ought to use metric.GetTag(k.Partition.Key) now.

[marcosnils]

Seems like an easy fix, add return "" after this if block and remove the log entry.

I've been looking at this issue and the code is actually returning "" in this case. The thing is that you can't use empty partition keys in kinesis. What would be the desired output in this case?, generate a UUID for those metrics that don't contain the tag-name or just error out sending to kinesis?

[danielnelson]

Maybe we should have a default value option when using the tag method, since this is the only type that can fail:

[outputs.kinesis.partition]
  method = "tag"
  key = "host"
  default = "telegraf"

[marcosnils]

Sounds good. Shall we make the default option required?, or just set it to telegrapf if unspecified?

[danielnelson]

Will need to default to telegraf so older config files will continue to work.

[DimaRabkin]

@marcosnils Are you sure Kinesis doesn't support empty string as a Partition key?
If so, then the output should check the tag's value and use the default value (or "telegraf") when tag's value is empty string, no?

In this option, I think there should be a warning log

[marcosnils]

@marcosnils Are you sure Kinesis doesn't support empty string as a Partition key?

Yup. From the AWS documentation (https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html#Streams-PutRecord-request-PartitionKey)

Type: String

Length Constraints: Minimum length of 1. Maximum length of 256.

Required: Yes

If so, then the output should check the tag's value and use the default value (or "telegraf") when tag's value is empty string, no?

That's what I did in my PR.

In this option, I think there should be a warning log

Is this necessary if the behavior is expected and documented?. I've added the corresponding documentation in the PR.

[DimaRabkin]

@marcosnils 👍