[receiver/filelog] Read lost files first in a poll cycle

[swiatekm]

Description:
We currently read lost files at the end of each poll cycle. As a result, we will always read lines from rotated files after reading them from newly created files, resulting in them being emitted in a different order than they were written. This usually doesn't matter, but it does when using the recombine operator. We ran into this when handling multipart logs generated by container runtimes in K8s.

This change moves handling lost files to the start of the poll cycle, directly after files for the cycle are detected and opened. I believe that it guarantees both durability and ordering if the rotation behaves reasonably, that is:

• files aren't written to after having been rotated
• rotated files don't match the pattern

@djaglowski please check me on this. My reasoning goes as follows:

• files read in a given cycle are determined in makeReaders, when we Open them to calculate Fingerprints
• a file is lost if it was present in the previous cycle, but not the current one
• even if a file is rotated during a cycle, it'll be considered lost in the next one, and ordering will be maintained

Link to tracking Issue: #12084

Testing:
Added a test verifying that we actually get log lines in the right order. This test fails without the other changes.

Documentation:
Added a section about log line ordering across rotations to design.md.

[swiatekm]

Hm, I can see the Unix and Windows implementations of the roller make different assumptions about when they're called. I wonder if we should change the API for the roller, or just switch the Windows roller to keep old readers and only close those. I'll do the former for now.

[djaglowski]

I think the way you've split up the interface makes sense. This looks good to me.

Please add a changelog entry. (note the new process)

[swiatekm]

I think the way you've split up the interface makes sense. This looks good to me.

Please add a changelog entry. (note the new process)

Done.

[djaglowski]

Thanks for for the contribution @swiatekm-sumo and for your patience rehashing the details with me.