Fix handling of truncated files for Filestream input

[belimawr]

Proposed commit message

This commit fixes the handling of file truncation in the Filestream input. In some file truncation cases Filebeat would detect the truncation but it would not set the offset to zero, leading to the registry offset always increasing and causing the file to be re-ingested every time Filebeat was restarted.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Author's Checklist

How to test this PR locally

All scripts and configuration files are at the end of this section.

1. File is truncated while Filebeat is running

1. Create a log file and keep adding data to it. Make sure to close the file after each line is added. If the file is still open and being written to while it's truncated the OS will likely not reset the file size. Use loop-add-data.sh
2. Start Filebeat
3. Truncate the file: truncate -s 50 /tmp/logs/log.log`. This will truncate the file but leave a whole line to be ingested later on.
4. Ensure Filebeat detects the file truncation and starts reading from the beginning
5. Ensure Filebeat correctly update the offset in the registry to 50 once it finishes reading the file

2. File is truncated while Filebeat is stopped

1. Create a log file. Use gen-file.sh
2. Start Filebeat
3. Ensure the whole file is read
4. Stop Filebeat
5. Ensure the registry contain the correct offset: 1000.
6. Truncate the file: truncate -s 50 /tmp/logs/log.log`. This will truncate the file but leave a whole line to be ingested later on.
7. Start Filebeat
8. Ensure Filebeat detects the file truncation and starts reading from the beginning
9. Ensure Filebeat correctly update the offset in the registry to 50 once it finishes reading the file

Configuration file and scripts

loop-add-data.sh

#!/bin/bash
cd /tmp/logs

rm foo.log
# Each line is 50 bytes
i=0
while true
do
    # Each log line is 50 bytes
    printf "log line with some padding %10d ===========\n" $i >> /tmp/logs/foo.log
    let "i=i+1"
    sleep 1
done

gen-file.sh

#!/bin/bash
cd /tmp/logs

rm foo.log
# Each line is 50 bytes
for i in {1..20}
do
    # Each log line is 50 bytes
    printf "log line with some padding %10d ===========\n" $i >> /tmp/logs/foo.log
    #sleep 0.05
done
ls -laih /tmp/logs

filebeat.yml

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    prospector.scanner.check_interval: 30s
    paths:
      - /tmp/logs/foo.log*

output:
  file:
    enabled: true
    codec.json:
      pretty: false
    path: ${path.home}
    filename: "output"
    rotate_on_startup: true

queue.mem:
  flush:
    timeout: 1s
    min_events: 32

filebeat.registry.flush: 1s

logging:
  level: debug
  selectors:
    - file_watcher
    - input.filestream
    - input.harvester
  metrics:
    enabled: false

Related issues

• Closes Filestream sometimes does not update registry offset when file is truncated #38070

## Use cases
## Screenshots
## Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 144 min 8 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filestream-truncation-bug upstream/filestream-truncation-bug
git merge upstream/main
git push upstream filestream-truncation-bug

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 0978bba

Failed CI Steps

• :ubuntu: Go Integration Tests

History

• 💔 Build #4046 failed 7c2a1f0
• 💔 Build #4045 failed a71e1f6
• 💔 Build #4044 failed b732514
• 💔 Build #4020 failed 30814ec
• 💔 Build #3962 failed f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #2396 succeeded 7c2a1f0
• 💚 Build #2370 succeeded 30814ec
• 💚 Build #2312 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #1166 succeeded 7c2a1f0
• 💔 Build #1162 failed 43e82b4
• 💚 Build #1140 succeeded 30814ec
• 💚 Build #1082 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #1124 succeeded 7c2a1f0
• 💚 Build #1098 succeeded 30814ec
• 💚 Build #1040 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 0978bba

Failed CI Steps

• :python: Python Integration Tests

History

• 💚 Build #3614 succeeded 7c2a1f0
• 💚 Build #3588 succeeded 30814ec
• 💚 Build #3530 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #1559 succeeded 7c2a1f0
• 💚 Build #1533 succeeded 30814ec
• 💚 Build #1475 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #2403 succeeded 7c2a1f0
• 💚 Build #2377 succeeded 30814ec
• 💚 Build #2319 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #1121 succeeded 7c2a1f0
• 💚 Build #1095 succeeded 30814ec
• 💚 Build #1037 succeeded f1a1d69

cc @belimawr

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0978bba

History

• 💚 Build #2693 succeeded 7c2a1f0
• 💔 Build #2667 failed 30814ec
• 💔 Build #2609 failed f1a1d69

cc @belimawr

[rdner]

@belimawr Did you make sure all these tests are failing without the fix applied?

[belimawr]

Yes, I wrote them before start working on the fix. But it has been a while since I ran them, so I'll re-run them without the fix before merging just to be on the safe side.

[belimawr]

--- FAIL: TestFilestreamLiveFileTruncation (8.91s)
    filestream_truncation_test.go:110: expecting offset 500 got 10500 instead

--- FAIL: TestFilestreamOfflineFileTruncation (4.28s)
    filestream_truncation_test.go:150: expecting offset 250 got 750 instead

[rdner]

Given yes is the answer to my question above, everything looks good.

[belimawr]

I rebased onto main to fix the merge conflicts, force push.

[belimawr]

/test

[dliappis]

Left a small question, otherwise LGTM from the ingest-eng-prod PoV

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filestream-truncation-bug upstream/filestream-truncation-bug
git merge upstream/main
git push upstream filestream-truncation-bug