This repository has been archived by the owner on May 25, 2022. It is now read-only.
Allow all operators to batch and pass entries through the pipeline #67
Labels
needs design
Requires a design proposal before implementation
See this issue for additional context.
The general idea is that we could save computation further down the line if we are able to keep related logs together.
Most input operators have an implied local context that could be used, but we need to consider carefully whether or not this really applies in some situations. What is the implied context for a
file_input
operator that is reading in all files in a directory? Are they necessarily related by resource, or just directory? If asyslog_input
operator is receiving from multiple systems, how do we handle batching?Most parsers can likely just iterate over the batch of entries.
This might require formalizing the concept of a resource at a level above entry:
The text was updated successfully, but these errors were encountered: