When log line from fluentd contains non-UTF-8 characters, the whole log line is dropped in Loki #5099
Comments
chaudum
added a commit
that referenced
this issue
Jan 12, 2022
When a log line in hash format contains non UTF-8 characters fluentd would drop the complete line because it failed to convert the line in key-value format. By forcing UTF-8 encoding and replacing non UTF-8 characters with empty strings the log line will not be dropped but only contain the valid UTF-8 characters. Fixes #5099 Signed-off-by: Christian Haudum <[email protected]>
3 tasks
cyriltovena
pushed a commit
that referenced
this issue
Jan 12, 2022
…utf-8 characters (#5107) * fluentd: Remove non utf-8 characters from log lines When a log line in hash format contains non UTF-8 characters fluentd would drop the complete line because it failed to convert the line in key-value format. By forcing UTF-8 encoding and replacing non UTF-8 characters with empty strings the log line will not be dropped but only contain the valid UTF-8 characters. Fixes #5099 Signed-off-by: Christian Haudum <[email protected]> * Bump fluent-plugin-grafana-loki to version 1.2.17 Signed-off-by: Christian Haudum <[email protected]> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <[email protected]> * fixup! fluentd: Remove non utf-8 characters from log lines Signed-off-by: Christian Haudum <[email protected]>
|
The customer has come back that this fix changed the error but did not resolve it fully: Deployed fluent-plugin-grafana-loki-1.2.17.gem with no success. Now we get a similar (not same) error message for specific logs: |
chaudum
added a commit
that referenced
this issue
Jan 17, 2022
This change fixes a bug that is caused by incorrectly trying to encode any values other than string. This bug was a regression introduced with #5107 and caused `NoMethodError` errors. Fixes #5099 Signed-off-by: Christian Haudum <[email protected]>
cyriltovena
pushed a commit
that referenced
this issue
Jan 17, 2022
* Fix encoding error in fluentd client This change fixes a bug that is caused by incorrectly trying to encode any values other than string. This bug was a regression introduced with #5107 and caused `NoMethodError` errors. Fixes #5099 Signed-off-by: Christian Haudum <[email protected]> * Bump fluent-plugin-grafana-loki to version 1.2.18 Signed-off-by: Christian Haudum <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
A customer is running into an issue where a Cisco device sends a log line that contains non UTF-8 characters into fluentd, when that is forwarded to Loki (using the fluentd-loki-output plugin), Loki drops the whole log line
Debugging steps
Customer debugging and troubleshooting involved the following steps:
1.
2
They came back with specific excerpts of the log lines that were causing trouble
3
Suggested the customer to use fluentd plugin to force valid encoding but they weren't comfortable with using it in production due to the last commit being over a year ago and not looking like an officially supported plugin which may cause future fluentd migration issues.
4
Finally, customer tried the following different configurations with fluent-plugin-record-modifier to address the problem:
But none of these worked always returning the same error messages regarding invalid chars by loki output plugin: „got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"“
Customer notes that the official fluent documention states:
Could this be a loki-output-plugin issue/
Expected behavior
If any non UTF-8 characters are seen, they should be replaced and the log line should be saved rather then having the whole log line get dropped.
Environment:
The text was updated successfully, but these errors were encountered: