Filestream input locks files in Windows

[Sergi-GC]

• Version: v7.13.1 and v7.15.1
• Operating System: Windows
• Description:

When using the Filestream input in Windows, Filebeat locks the files it's monitoring and prevents other processes from renaming or moving the files.

This has been reproduced in Windows Server 2016, Windows 10 Enterprise and in Windows 10 Home environment.

Error in Windows Server 2016:

Error in Windows 10 Home:

This issue does not reproduce in MacOS.
And also does not happen with the Log input.

• Steps to reproduce:

Case 1: Commenting out the close inactive settings for both inputs

Settings:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\logs\this_file_is_not_locked.log

  clean_removed: true
  close_renamed: true
 #close_inactive: 1m
  scan_frequency: 10s

- type: filestream
  enabled: true
  paths:
    - C:\logs\this_file_is_locked.log

  clean_removed: true
  close.on_state_change.removed: true
  close.on_state_change.renamed: true
  #close.on_state_change.inactive: 1m
  prospector.scanner.check_interval: 10s

Results:

• The file this_file_is_not_locked can be renamed/deleted/moved any time since the moment Filebeat is started.
• The file this_file_is_locked cannot be renamed/deleted/moved since the moment Filebeat is started.

Case 2: Setting close inactive settings to 1min

Settings:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\logs\this_file_is_not_locked.log

  clean_removed: true
  close_renamed: true
  close_inactive: 1m
  scan_frequency: 10s

- type: filestream
  enabled: true
  paths:
    - C:\logs\this_file_is_locked.log
   
  clean_removed: true
  close.on_state_change.removed: true
  close.on_state_change.renamed: true
  close.on_state_change.inactive: 1m
  prospector.scanner.check_interval: 10s

Results:

• The file this_file_is_not_locked can be renamed/deleted/moved any time since the moment Filebeat is started.
• The file this_file_is_locked cannot be renamed/deleted/moved for about 1m since the moment Filebeat is started. After 1 min the file can be renamed/deleted/moved.

Case 3: Commenting out all the close/clean settings

Settings:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\logs\this_file_is_not_locked.log

  #clean_removed: true
  #close_renamed: true
  #close_inactive: 1m
  scan_frequency: 10s

- type: filestream
  enabled: true
  paths:
    - C:\logs\this_file_is_locked.log
   
  #clean_removed: true
  #close.on_state_change.removed: true
  #close.on_state_change.renamed: true
  #close.on_state_change.inactive: 1m
  prospector.scanner.check_interval: 10s

Results:

• The file this_file_is_not_locked can be renamed/deleted/moved any time since the moment Filebeat is started.

• The file this_file_is_locked cannot be renamed/deleted/moved for about 1m since the moment Filebeat is started.

• Logs from each scenario:
  filebeat_case_1.log
  filebeat_case_2.log
  filebeat_case_3.log

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[amolnater-qasource]

Hi @kvch
We have revalidated this issue on 8.1 BC-6 Kibana self-managed environment.
HOST: AWS-Windows 2016 Server

Build details:
BUILD: 50485
COMMIT: 4aaeda23aea9c3bf29698878c70a0107ea3c1659
Artifact Link: https://staging.elastic.co/8.1.0-6b89df21/summary-8.1.0.html

We had below observations:

Case 1: Commenting out the close inactive settings for both inputs

We were able to move, rename and delete both locked_file.log and unlocked_file.log.

Case 2: Setting close inactive settings to 1min

We were able to move, rename and delete both locked_file.log and unlocked_file.log.

Case 3: Commenting out all the close/clean settings

We were able to move, rename and delete both locked_file.log and unlocked_file.log.

Hence, this issue is not reproducible now.

For reference we have used below filebeat.yml and log files.
filebeat.zip
[Necessary changes were made as per different cases]
unlocked_file.log
locked_file.log

Please let us know if this is the required result and let us know if we are missing anything.

Thanks!!

[amolnater-qasource]

FYI @jlind23