Compressed files land in /tmp

[szalansky]

I have a strange feeling that compressed files land in /tmp directory. I realised that after one of my rake tasks had been interrupted due to lack of space on a disc.

Command I run (for a large number of images, size around 50 GB):

image_optim = ImageOptim.new(:pngout => false, :pngcrush => false,
:optipng => { :level => 7 }, :advpng => false, :jpegoptim => { :strip => "all" },
:jpegtran => false)
logger = Logger.new("#{Rails.root}/log/image_optim.log")

image_optim.optimize_images!(images) do |unoptimized, optimized|
logger.info("#{Time.now} optimised image: #{unoptimized}") if optimized
end

Is it possible to avoid such behaviour or dump files to /dev/null, or delete them during procesing?

[toy]

First thought is that intermediate temporary files are not removed as their objects are not garbage collected. Will check

[szalansky]

@toy I had a look at source and I think this method can called used in a block that is passed to optimize_images!. I gave it a try, but there is another file created as a replacement (with slightly different name).

motur20140131-8365-yujq3o-0.jpg - file I delete.
motur20140131-8365-6p1nm2-0.jpg - additional file

[szalansky]

@toy I decided to set TMPDIR as /dev/null before running script and it worked fine, but I'm aware that was a walkaround (maybe those temporary files are used in different stages of compression process?)

[toy]

What do you expect to get by setting TMPDIR to /dev/null?
Anyway it will not work as bad TMPDIR will be ignored: ruby:::lib/tmpdir.rb

[szalansky]

@toy I only want temp files not to gather somewhere and take disk space as I can't get rid of them in a more elegant way.

[toy]

Please try branch unlink-intermediate.

[szalansky]

@toy I just had a look at it and I will test that soon and get back to you. Thank you for your effort!

[szalansky]

@toy testing it at the moment. I think it removes the files selectively, because total numbers of files in directory gradually grows:

rszalanski@raf-macbook /tmp/image_optim $ find . -type f | wc -l                                                                                                                                                                 [ruby-2.1.0]
find: ./list_thumbnail-2F-640x36020140205-82716-1f45lff-0.jpg.lock: No such file or directory
    3544

Memory consumption also grows:

Important think: at once I compress smaller chunk of images (2000 or 3000).

[toy]

I've wrote a small script to check memory consumption and temp file leakage. Using it on current master 493bba3 shows lots of still existing tempfiles before GC, but zero afterwards. But for unlink-intermediate 7ebc2db it shows zero also before GC.

[szalansky]

@toy Ok, I finally get reasonable numbers and $TMPDIR looks clean (files are added and deleted). Would you like me to make more tests? I think this change you made is kinda nice improvement.

[toy]

What did you do to get to reasonable numbers? Did you change something in image_optim? Would be nice if you can test more to get to conclusive result. Otherwise I am going to merge it.

[toy]

@szalansky Did you manage to test more? Can this issue be considered resolved?

[szalansky]

@toy I'm sorry, but I didn't get notification about your previous comment. Yes, this issue is resolved. Thank you for your effort and time.

[toy]

@szalansky Strange, probably I should always @mention person I'm writing to. Thanks for checking