Error if using sincedb_path => "/dev/null" on Darwin

[andsel]

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: any (but tested with 4.1.10)
• Operating System: Darwin
• Config File:

input {
  file {
    path => '/tmp/logstash*.log'
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
output {
  stdout {
    codec => 'rubydebug'
  }
}

• Sample Data: some file with data
• Steps to Reproduce:
  Start any version of Logstash bin/logstash -f pipeline.conf on MacOSX.

it produces this error:

[2020-08-14T13:14:30,048][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File start_position=>"beginning", path=>["/Users/cdarocha/support_cases/525435/test2/inputtest.json"], id=>"e9133d1073a9989bba8fda70f772d153d156466e27ed5e88e15aa5ca155bac8b", sincedb_path=>"/dev/null", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_5021ba8d-0bde-483c-9dee-3acbca06181c", enable_metric=>true, charset=>"UTF-8">, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false>
  Error: Permission denied - Permission denied
  Exception: Errno::EACCES
  Stack: org/jruby/RubyFile.java:1269:in `utime'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1133:in `block in touch'
org/jruby/RubyArray.java:1792:in `each'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:1130:in `touch'
/tmp/logstash-6.8.8/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.16/lib/filewatch/sincedb_collection.rb:22:in `initialize'
/tmp/logstash-6.8.8/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.16/lib/filewatch/observing_base.rb:62:in `build_watch_and_dependencies'
/tmp/logstash-6.8.8/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.16/lib/filewatch/observing_base.rb:56:in `initialize'
/tmp/logstash-6.8.8/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.16/lib/logstash/inputs/file.rb:341:in `start_processing'
/tmp/logstash-6.8.8/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.1.16/lib/logstash/inputs/file.rb:346:in `run'
/tmp/logstash-6.8.8/logstash-core/lib/logstash/pipeline.rb:426:in `inputworker'
/tmp/logstash-6.8.8/logstash-core/lib/logstash/pipeline.rb:420:in `block in start_input'

Trying to open an irb from the LS pack with bin/logstash -i irb and issue the command:

FileUtils.touch("/dev/null")

produces the same error. It seems related to the fact that the t = Time.now; File.utime(t,t, '/dev/null') produces:

Errno::EACCES: Permission denied - Permission denied
   utime at org/jruby/RubyFile.java:1267
  <main> at -e:1

So to check the file path for sincedb it could be done with:

File.open("/dev/null", "w") { puts 1}

[alexander-marquardt]

I can confirm that this bug is occurring on my Logstash 7.9.2 running on OSX. The error I am seeing is the same as that above.

Input file is:

input {
  file {
    path => "/Users/alexandermarquardt/Documents/elastic-stack/7.9.2/logstash-7.9.2/config/dynamic_test.csv"
    start_position => "beginning" 
    
    # The following line will ensure re-reading of input 
    # each time logstash executes.
    sincedb_path => "/dev/null" # remove this for production 
  }
}

Removing sincedb_path fixes the error.

[ugosan]

As a workaround you can point the sincedb file to a real file instead of /dev/null and tell it to clear it immediately, the effect is the same - the file is re-read everytime the pipeline reloads.

input {
  file {
    path => "/Users/ugosan/temp/input.txt"
    start_position => "beginning"
    #an actual file as sincedb...
    sincedb_path => "/Users/ugosan/temp/an_actual_file"
    #...that is always cleaned up:
    sincedb_clean_after => 0
  }
}

[alexander-marquardt]

Running as sudo also works. [EDIT: Don't ever do this in production! ]

[yaauie]

Running as sudo also works.

... and also gives Logstash root/superuser privileges, which is not advised in a production environment.

[alexander-marquardt]

Running as sudo also works.

... and also gives Logstash root/superuser privileges, which is not advised in a production environment.

100% agree - I think I meant that as an observation, rather than a recommendation ... but I can see how that could end up being interpreted incorrectly. Added an "edit" to the above.