file_completed_action delete not working for gzip files

[hkelley]

I didn't get a reply on the forum so I'm posting here.

When I read .gz log files, the files are not being deleted after logstash processing. When I read an uncompressed file it seems to delete correctly.

• Windows 10
• logstash-input-file (4.1.11)
• logstash-7.4.2

input { 
	file {		
		path => "C:/temp/CS_dump/logstash-pickup/part-*.gz"  
		sincedb_path => "NUL"
		codec => json
		mode => "read"
		file_completed_action => "delete"
	}
}

I can create and delete files in the folder so I don't see how it could be a permission issue. I don't see any errors in the logs related to a failed delete attempt.

[andsel]

@hkelly when you delete the uncompressed file, do you also use the * shell expansion or point to a single file?
Could you try with the latest 4.1.14 version of the plugin published 27/1/2019 because contains the bugfix #254 that solves a delete on multiple files?
Thanks for your feedback after the test, it's welcome

[hkelley]

When I delete the files myself, from the shell, either explicitly or by wildcard, the files are deleted.

I updated logstash-input-file 4.1.11 to 4.1.14 but I get the same result in logstash.

Is there a log file I can generate for more clues?

[andsel]

I've tried with this:

• created sample text file in /tmp with

echo "blabla1" > ls_file_test1.txt && gzip ls_file_test1.txt && echo "blabla2" > ls_file_test2.txt && gzip ls_file_test2.txt

• created this Logstash config file input_file_gzipped.conf:

input {
  file {
    path => '/tmp/ls_file_test*.gz'
    sincedb_path => "NUL"
    mode => "read"
    file_completed_action => "delete"
  }
}
output {
  stdout {
    codec => 'rubydebug'
  }
}

• launched with

bin/logstash -f input_file_gzipped.conf --debug

And gzipped files are correctly deleted. Tried both version 7.4.2and latestmaster`.

Probably the the user you use to run Logstash is different from the user you used to modify files in the path C:/temp/CS_dump/logstash-pickup/ and they have different permissions. Checkout your log to see if something is written about failed deletion, start Logstash with --debug CLI switch

[hkelley]

• I am creating the files and running Logstash as the same user. This should eliminate any chance of a permission deficit. I confirmed that the user can delete the files.

• Configuration is:
  input { file { path => "C:/temp/CS_dump/logstash-pickup/*.txt.gz" sincedb_path => "NUL" mode => "read" file_completed_action => "delete" } } output { stdout { codec => rubydebug } }

• Launched with
  \elastic\logstash-7.4.2\bin\logstash.bat -l c:\temp\CS_dump\logstash-logs -f "logstash-test.conf" --debug

• I do not see any any errors (though I'm not terribly familiar with logstash so I may be overlooking). Log attached.
  logstash-plain.log

[andsel]

I suspect that

logstash-input-file/lib/logstash/inputs/delete_completed_file_handler.rb

Lines 4 to 7 in 42e2ef2

	class DeleteCompletedFileHandler
	def handle(path)
	Pathname.new(path).unlink rescue nil
	end

unlink using a File.delete in JRuby create a twin handler to the same file, the other is kept by

logstash-input-file/lib/filewatch/watched_file.rb

Lines 18 to 19 in 42e2ef2

	@pathname = Pathname.new(pathname) # given arg pathname might be a string or a Pathname object
	@path = @pathname.to_path

so Windows doesn't delete and Unix yes. At the moment I don't have a Windows box to test this assumption

[ewilansky]

I ran into this issue too. Setting logstash to debug output showed this:
The "file_completed_log_path" setting must be provided when the "file_completed_action" is set to "log" or "log_and_delete".

Once I did as the message said, the file was deleted and the information appeared in the log. The file input documentation: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-file_completed_action, suggests that both are necessary, but doesn't say it outright.