Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_stackdriver: convert invalid utf-8 character to replacement string #9473

Open
shuaich opened this issue Oct 8, 2024 · 1 comment
Open

Comments

@shuaich
Copy link
Contributor

shuaich commented Oct 8, 2024

Is your feature request related to a problem? Please describe.

Google Cloud Logging API requires strings to be UTF-8 encoded[1]. Log entries with invalid UTF-8 characters fail to be ingested when partial_success is enabled, otherwise the whole WriteLogEntries request is dropped.

The out_google_cloud plugin in fluentd provides a feature that converts invalid UTF-8 characters to replacement strings[2].

I propose to add the same feature available in out_stackdriver plugin.

[1]https://cloud.google.com/logging/docs/reference/v2/rpc/google.logging.v2#google.logging.v2.LoggingServiceV2.WriteLogEntries
[2]https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/blob/master/lib/fluent/plugin/out_google_cloud.rb#L1873C9-L1873C24

Describe the solution you'd like

To add two configuration parameters: coerce_to_utf8 and non_utf8_replacement_string. The expected behaviour is:

  1. coerce_to_utf8 defaults to false, non_utf8_replacement_string defaults to a space character ' '.
  2. When coerce_to_utf8 is enabled, convert invalid UTF-8 characters to configured replacement string for all string fields in LogEntry
  3. When coerce_to_utf8 is disabled, skip the conversion

Describe alternatives you've considered

Additional context

@shuaich
Copy link
Contributor Author

shuaich commented Oct 8, 2024

Fluentbit strips away remaining string containing the invalid UTF-8 chracters, relevant issues are #2142 #5035

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant