Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/filelog] Log lines are reordered during file rotation #12084

Closed
swiatekm opened this issue Jul 5, 2022 · 1 comment
Closed

[receiver/filelog] Log lines are reordered during file rotation #12084

swiatekm opened this issue Jul 5, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@swiatekm
Copy link
Contributor

swiatekm commented Jul 5, 2022

Describe the bug
When a file is rotated, filelog receiver may read lines from the new file before exhausting the old one, leading to log lines being reordered. This is a problem especially when using recombine to stitch multipart or multiline logs back together.

Steps to reproduce
Set a large interval length (5s for example), then rotate a file close to the end of it.

What did you expect to see?
Logs in the order they were written (and consistent with their timestamps).

What did you see instead?
Newer logs from the new file before older logs from the rotated file.

What version did you use?
v0.52.0 - v0.54.0, no change in behaviour between versions.

What config did you use?

receivers:
  filelog:
    poll_interval: 5s
    include:
    - /var/log/containers/*/*/*.log
  exporters:
    logging:
  service:
    telemetry:
      logs:
        level: debug
    pipelines:
      containers:
        receivers: [filelog]
        processors: []
        exporters: [logging]

Environment
OS: BottlerocketOS 1.8.0

Additional context
This came up in K8s, where container runtimes will split large log lines into parts, and the log collection agent needs to stitch them back together.

A fix is WIP here: #11889

@swiatekm
Copy link
Contributor Author

swiatekm commented Jul 5, 2022

Fixed in #11889

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant