tail_files should persist initial offset as end of file

[andrewkroh]

At startup Filebeat initializes the registry with info about the file that it opens (but this does not contain an offset so it’s left as 0). This is the expected behavior for tail_files: false as we do not want to advance the offset until some data has been read. But for tail_files: true my personal expectation is that it would initially write the offset as the end of the file, instead of 0, so that if Filebeat exits it can resume reading from that point.

• Version: 1.3.1
• Operating System: affects all
• Steps to Reproduce:
  • Delete .filebeat registry file.
  • Write multiple lines of text to out.log.
  • Start filebeat with tail_file: true.
  • Stop filebeat.
  • Read the .filebeat registry. It will have an offset of 0 because no events were written to the file while Filebeat was reading.

Config:

filebeat:
  prospectors:
    - paths:
        - out.log
      input_type: log
      tail_files: true
      close_older: 1s

[andrewkroh]

beats/filebeat/crawler/prospector.go

Lines 402 to 405 in 162fb25

	// Launch the harvester
	h.SetOffset(oldState.offset)
	h.Start()
	p.registrar.Persist <- h.GetState()

If the Havester.Start() wasn't asynchronous or it initialized the state prior to starting the havester goroutine, then this would work as expected because h.GetState() would return a fully initialized input.FileState containing the offset.

[ruflin]

This should be solved together with #2613 by moving tail_files logic to the prospector.

Please note that in the 5.0 release the correct state is written as soon as the file is closed.

[ruflin]

Closing as this was solved in #2932 and should be available in 5.1. As 5.0 is GA and filebeat 5.0 is backward compatible, I suggest we do not try to backport this.