Log receivers - Improve translation to pdata, avoid emitting entire pdata.Logs for each log record.

[djaglowski]

Currently this translation is 1:1. Need to define and implement a more efficient translation.

[pmalek-sumo]

I'd be willing to work on this but I'll need some guidance on the desired direction of this.

My understanding is that there is a func Convert(obsLog *entry.Entry) pdata.Logs
which does the actual conversion.

If we'd like to somehow make this not 1:1 translation then ... we'd need to aggregate entry.Entrys and then using a particular interval send them further for processing? Or is there a different vision for this that I'm not seeing yet?

[djaglowski]

I think it makes sense to accumulate entry.Entry's into []entry.Entry and pass this to Convert([]entry.Entry), which will then batch and convert to a single pdata.Logs. There is an implementation of this here, which may be a good starting point.

• Accumulation of entry.Entry's into []entry.Entry could take place here.

• []entry.Entry should probably be released based when either a max number of entries is reached, or a timeout is reached.
  Perhaps in a subsequent PR, the max number and max time can be made configurable as part of BaseConfig.

• We can group entries together when entry.Entry.Resource fields are a perfect match.

• Are there any other ways in which entries should be batched? Maybe someone more familiar with pdata.Logs can weigh in?

[pmalek-sumo]

I've prepared something that might address this issue in https://github.com/SumoLogic/opentelemetry-collector-contrib/tree/issue-2330-improved-logs-translation

Compare across forks: main...SumoLogic:issue-2330-improved-logs-translation

@djaglowski If this is more or less the direction we'd like to this in I can add some tests, refactor it a little bit and submit a PR so that we can take it from there.

[djaglowski]

@pmalek-sumo This is more or less what I was expecting. Looks like a great start.

[pmalek-sumo]

I've created #2694 as a proposal for this issue.

Current implementation allows configuration via 2 knobs:

• flush_trigger_count - amount of entries aggregated so far in current batch. If this is exceeded then a flush of (all aggregated) log entries will be scheduled
• flush_interval - an interval at which flushes will be scheduled.

Waiting for your feedback.

[tigrannajaryan]

@pmalek-sumo @djaglowski do we expect more improvements or we consider done with this?

[pmalek-sumo]

I'd like to propose a variant of #2949 for consideration as design improvement.

The only concern I have about this not being accepted though is that it increases the cpu consumption by a few percent.
If there's will to take a look at it ( a rebased, cleaned up version of #2949) I'll prepare it then

[djaglowski]

@pmalek-sumo If you are ok with it, I would like to consider this issue done.

We can open a new issue to track further proposed changes.

cc @tigrannajaryan

[tigrannajaryan]

Sounds good. I am closing this issue. We can continue looking at #2949