Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Message Max bytes configuration #4537

Conversation

otherpirate
Copy link
Contributor

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

@danielnelson
Copy link
Contributor

What happens if the message is over the max, does err := k.producer.SendMessages(msgs) return an error?

@otherpirate
Copy link
Contributor Author

When message is bigger than MessageMaxBytes config this error occurs:
2018-08-09T15:47:24Z E! Error writing to output [kafka]: kafka: Failed to produce message to topic <topic>: kafka server: Message was too large, server rejected it to avoid allocation error.

It's the same error with default value(without configuration)

@danielnelson danielnelson added this to the 1.8.0 milestone Aug 9, 2018
@danielnelson
Copy link
Contributor

It seems like short term we ought to drop these messages, after logging, because otherwise the output plugin will just be stuck trying to send with no potential to send successfully. I don't think this needs to be done in the PR though.

Long term, I would like to add a max size limitation to the Serializer interface. This way we could attempt to split the messages, similar to how we can currently limit single lines in the influx serializer. We would have to consider what this means for each serializer and ultimately it still may impossible to write some metrics in the allotted space.

@glinton glinton merged commit 434fa8d into influxdata:kafka-perf-batch Aug 13, 2018
@danielnelson
Copy link
Contributor

cherry-picked to master 61513c6

rgitzel pushed a commit to rgitzel/telegraf that referenced this pull request Oct 17, 2018
otherpirate added a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019
otherpirate added a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019
dupondje pushed a commit to dupondje/telegraf that referenced this pull request Apr 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants