Stops sending logs after harvester error

[glenjamin]

While slurping logs from a nomad install, we found that filebeat would stop sending logs, and tailing the log showed no errors.

Looking further up the logs we do see a single error line, but then the process keeps running and doesn't ship any logs.

Log sample:

2019-07-19T18:17:48.295Z        ERROR   log/input.go:484        Harvester could not be started on new file: /opt/nomad/alloc/ba9e1994-fd5e-8e64-55fb-4afa2e1a1894/alloc/logs/.asldjkasldjaslkdjsalkdjasdlkjasd.stdout.fifo, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Tried to open non regular file: "prw-------" .asldjkasldjaslkdjsalkdjasdlkjasd.stdout.fifo

The affected directory looks like this

-rw-r--r-- 1 root   root    13970 Jul 19 19:13 asdfghjklasdfghjklsdfghjk.stderr.0
prw------- 1 root   root        0 Jul 19 19:13 .asdfghjklasdfghjklsdfghjk.stderr.fifo|
-rw-r--r-- 1 root   root       30 Jul 19 19:12 asdfghjklasdfghjklsdfghjk.stdout.0
prw------- 1 root   root        0 Jul 19 19:12 .asdfghjklasdfghjklsdfghjk.stdout.fifo|

For confirmed bugs, please report:

• Version: 7.2.0
• Operating System: Ubuntu 18.04
• Discuss Forum URL: n/a
• Steps to Reproduce: have the glob match some named pipes

[danlsgiga]

Facing the same issue... this also leads to filebeat having a memory leak until the nomad job kills the task due to OOM.

However, I'm running filebeat 6.2.3.

[danlsgiga]

It seems to be not applying the glob matching rule describe here

[danlsgiga]

@glenjamin, FYI, I overcome this specific issue by setting exclude_files: ['\.fifo$'] in the filebeat prospector config.

Anyhow, I believe filebeat should avoid that, specifically if it says the * glob pattern does not include hidden files (dot files).

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue doesn't have a Team:<team> label.