Logstash doesn't discover the files if it does not have a permission on target folder.

[mashhurs]

Description

When installing Logstash through the package managers (like yum install logstash), running as a service and using file input plugin, Logstash may not able to discover the files. It is because Logstash doesn't have a permission to consume and delete permission to the target folder/path files. Sometimes it is an expected behaviour but for the most cases users expect Logstash to directly consume files once it is installed. We somehow need to document and educate them how to overcome.

To figure out if Logstash is discovering files, we can go to Logstash dir and run following command: sudo runuser -u logstash -- bin/ruby -e 'puts Dir.glob("/home/logstash/files/*")' (change the path). Or we need to enable the trace logs. Add following config in /etc/logstash/log4j2.properties

logger.fileinput.name = logstash.inputs.file
logger.fileinput.level = trace
logger.filewatch.name = filewatch
logger.filewatch.level = trace

Once we restart Logstash, in the plain log (/var/log/logstash/logstash-plain.log) we can observe the file watch discovery lines that tell us discouvery count is zero.

[2022-10-13T17:02:18,968][TRACE][filewatch.discoverer     ][main][...] discover_files {:count=>0}

Solution as it sounds, is to provide a permission. Eg.: sudo chown a+rwx /home/logstash/files, make sure to change the command for proper user.

[2022-10-13T17:02:18,968][TRACE][filewatch.discoverer     ][main][...] discover_files {:count=>3}

• Version: 8.4.3
• Operating System: Oracle 8.5
• Config File (if you have sensitive info, please remove it):
• Sample Data:
• Steps to Reproduce: fresh Logstash install and using file-input