[exporter/awss3] Raw Log Setting

[blakeromano]

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

In the Amazon Cloudwatch Exporter there is a configuration setting called raw_log which will take the OpenTelemetry format and significantly reduce the size of the JSON and do some processing to make logs easily queryable in CloudWatch logs. I'd love to see this be added to the S3 Exporter. When trying to then do an ingestion of the Logs in S3 into a service like OpenSearch the data in S3 is very hard to parse and process leading to pains when trying to query it. If the feature of raw_log was enabled for logs exported to S3 this would reduce the burden of engineers having to do transformations in Lambda, ETL jobs or OSI Pipelines.

Describe the solution you'd like

I would want the S3 Exporter to have a raw_log attribute defaulted to false that when enabled would emit the same logic the CloudWatch Exporter has to turn the logs into a unified struct before doing a PutObject to the JSON in S3. This should not impact anything on metrics or traces and should only impact logs.

Describe alternatives you've considered

There is ETL changes after the data is in S3 or some work that can be done via operators in the log receiver used that could be done in place of this however this solution would be much simpler do in the Exporter.

Additional context

The code the Cloudwatch Log Exporter does when raw_log is set to true lives here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awscloudwatchlogsexporter/exporter.go#L208-L239

[github-actions]

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski

%s See Adding Labels via Comments if you do not have permissions to add labels yourself.

[blakeromano]

I looked at the awss3 exporter code more and think this could be accomplished with a new marshaler. Something maybe like prettyJSON or something along those lines. Or logMarshaler maybe?

[atoulme]

You can do that using the encoding config field, https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awss3exporter/README.md?plain=1#L31

You can use of one of the encoding extensions under extensions/encoding or come up with your own.

[blakeromano]

@atoulme that seems to me that none of the encoding extensions have been added to the contrib distribution yet so unfortunately I don't think that will totally get me what I want. I am happy to contribute a new encoder however it doesn't quite seem that encoding is in any distribution yet.

[atoulme]

We can fix that, for sure. Do you want to file issues to make it so?

[blakeromano]

When you mean "fix that" are you referring to none of the encoding extensions in the contrib distribution or are you referring to adding a new encoder?

[atoulme]

Both I guess. That said I'm working on making the text_encoding encoder marshal logs so it just writes the body.

[atoulme]

Here, would you like to review #32696 ?

[blakeromano]

I have a starting PR here -> #32722 to add some logic into jsonlogencoding extension which I think is the best spot for this

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/awss3: @atoulme @pdelewski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[blakeromano]

This is fixed now 🚀