file_input: last log is not being flushed if multiline is used

[sumo-drosiek]

If using multiline detection with line_start_pattern for log input, last line is not being flushed at all. This is due to no control mechanism which check if more data is going to be appended to the file, so we always wait for more data.

Same behavior for no multiline and file ended with no new line

My proposal is to flush data if there was no more data appended to a file and/or the file has not been modified for some period of time. Another approach is to flush the last log if there is no file descriptors in system, which uses that file (except our). Time could be configurable in file_input.

---

test.log:

LOGSTART 1
LOGPART 1
LOGPART 1
LOGSTART 2
LOGPART 2
LOGSTART 3

config.yaml

receivers:
  filelog:
    start_at: beginning
    include:
      - ./open-receiver/test.log
    multiline:
      line_start_pattern: ^LOGSTART

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    logs:
      receivers:
        - filelog
      exporters:
        - logging

logging output:

2021-07-09T10:05:09.084+0200    DEBUG   loggingexporter/logging_exporter.go:141 ResourceLog #0
InstrumentationLibraryLogs #0
InstrumentationLibrary  
LogRecord #0
Timestamp: 2021-07-09 08:05:08.98527233 +0000 UTC
Severity: Undefined
ShortName: 
Body: LOGSTART 2
LOGPART 2
Attributes:
     -> file.name: STRING(test.log)
LogRecord #1
Timestamp: 2021-07-09 08:05:08.985249861 +0000 UTC
Severity: Undefined
ShortName: 
Body: LOGSTART 1
LOGPART 1
LOGPART 1
Attributes:
     -> file.name: STRING(test.log)

^C2021-07-09T10:05:25.293+0200  info    service/application.go:210      Received signal from OS {"signal": "interrupt"}