Trying to find how to do multiple write operations with a loaded image

[quentin-bettoum]

I've been playing around with Image and Vix and this is what I'm trying to do:

1. Load an image
2. Resize it
3. Output it in different formats with some options

I guess it would be more optimized in this order since the conversion would be done on a downsized version of the input image.

It seems there are different ways to do this, but most of the time I have a Failed to write VipsImage error when trying to do multiple write operations to different files, it will write the first one and fail for all other attempts.

I obviously lack experience about file handling in general. I'm reading docs and articles about Elixir files handling, File.stream, etc.

Here is a module I've made for my tests, I've added some comments to indicate where it succeeds and where it fails.
I've tried using both Image and Vix methods. It seems using Image.write!/3 or Vix.Vips.Image.write_to_file/2 gives the same result.

I would appreciate any help to understand what are the reasons of the errors.

defmodule MyTest do
  alias Vix.Vips.Image, as: Vimage

  @img_path "image.jpg"

  defp convert_image_in_multiple_formats(img) do
    Image.write!(img, "image-thumb.webp", quality: 30, strip_metadata: true)
    Image.write!(img, "image-thumb.avif", quality: 30, strip_metadata: true, effort: 1)
    Image.write!(img, "image-thumb.jpg", quality: 30, strip_metadata: true)

    # Vimage.write_to_file(img, "image-thumb.webp[Q=30,strip]")
    # Vimage.write_to_file(img, "image-thumb.avif[Q=30,strip,effort=0]")
    # Vimage.write_to_file(img, "image-thumb.jpg[Q=30,strip]")
  end

  ## Image methods
  # ERROR
  def thumbnail_from_path() do
    @img_path
    |> Image.thumbnail!("1920x1080")
    |> convert_image_in_multiple_formats()
  end

  # ERROR
  def open_and_thumbnail() do
    @img_path 
    |> Image.open!()
    |> Image.thumbnail!("1920x1080")
    |> convert_image_in_multiple_formats()
  end

  # OK
  def open_and_thumbnail_random_access() do
    @img_path 
    |> Image.open!([access: :random])
    |> Image.thumbnail!("1920x1080")
    |> convert_image_in_multiple_formats()
  end

  ## Vix methods
  # ERROR
  def vix_thumbnail_from_path() do
    @img_path
    |> Vix.Vips.Operation.thumbnail!(1920, height: 1080)
    |> convert_image_in_multiple_formats()
  end

  # ERROR
  def vix_thumbnail_from_buffer() do
    @img_path 
    |> File.read!()
    |> Vix.Vips.Operation.thumbnail_buffer!(1920, height: 1080)
    |> convert_image_in_multiple_formats()
  end

  # OK
  def new_from_file() do
    {:ok, %Vimage{} = img} = Vimage.new_from_file(@img_path)

    img
    |> Vix.Vips.Operation.thumbnail_image!(1920, height: 1080)
    |> convert_image_in_multiple_formats()
  end

  # OK
  def new_from_buffer() do
    {:ok, %Vimage{} = img} = @img_path |> File.read!() |> Vimage.new_from_buffer()

    img
    |> Vix.Vips.Operation.thumbnail_image!(1920, height: 1080)
    |> convert_image_in_multiple_formats()
  end
end

[kipcole9]

This is probably my fault. TLDR; use Image.open(path, access: :random) if you're going to output an image into different formats. I default in Image to access: :sequential because it optimises for the use case of open, transform, write. libvips itself defaults to access: :random and perhaps I should be too. I can see you are doing the same above in one of your test cases.

As for the order of operations, libvips is demand driven and it will only process pixels it needs to when it needs to. In fact it won't really do anything until to render the image (as you do for writing, composing and so on).

There is one "special case" which is Image.thumbnail/2 when the argument is a pathname (not a Vix.Vips.Image.t). In this case, libvips is higher optimised for file opening, block resizing, detail resizing and output. So basically Image.thumbnail with a path argument can perform better than Image.thumbnail with a Vix.Vips.Image.t. In Vix, as you have seen, this resolve to the two calls Operation.thumbnail versus Operation.thumbnail_buffer. Where you can, use Image.thumbnail with a path argument.

Image really is just a wrapper around Vix with a focus more on developer ergonomics and packaging some more common imaging "recipes" for easy of use and more idiomatic Elixir style. So you should see the same behaviour from Image and Vix since one just wraps the other.

[kipcole9]

In short, your example (which I copied below) is probably the most efficient for this. But it would need to be benchmarked against using the pathname form of Image.thumbnail because of its optimisations:

  def open_and_thumbnail_random_access() do
    @img_path 
    |> Image.open!(access: :random)
    |> Image.thumbnail!("1920x1080")
    |> convert_image_in_multiple_formats()
  end

I'll fire up benchee and see what it says.

[kipcole9]

I added a simple benchmark with the following results that suggests the approach you are using (and which I documented again in the message above) is probably the fastest way to do what you're after. As I understand it, the most expensive operation is decoding the image and therefore not too surprising if limiting the number of times an image is opened is a good thing.

Name                                    ips        average  deviation         median         99th %
Image.open |> Image.thumbnail          9.06      110.37 ms     ±2.55%      109.82 ms      121.92 ms
Image.thumbnail(path)                  6.24      160.22 ms     ±1.07%      159.92 ms      165.08 ms

Comparison:
Image.open |> Image.thumbnail          9.06
Image.thumbnail(path)                  6.24 - 1.45x slower +49.84 ms

Memory usage statistics:

Name                             Memory usage
Image.open |> Image.thumbnail        10.57 KB
Image.thumbnail(path)                19.08 KB - 1.80x memory usage +8.51 KB

[kipcole9]

I will change the default to access: :random in the next version of Image. The approach I'm taking is definitely not helping developer ergonomics. Last note - this is nothing to do with Elixir or Erlang file handling, this is all related to how libvips optimises file handling for performance and memory.

[quentin-bettoum]

Thanks a lot for taking the time to answer. It is a lot clearer now.
I've also come across these pages which helped me understand a little bit better how sequential and random access are working in libvips:

• How libvips opens files
• vips_image_new_from_file

I had planned to investigate the shrink-on-load option.
From what I've read there, it should only be used to get to 2x larger than the desired output size to avoid aliasing.
In my case, since I don't know in advance if the input image will be large enough, I would have to be able to check for the size programmatically before opening the image to see if I should apply a shrink parameter (and of how much) or not.

I was wondering why Image.thumbnail with a path as input which uses vips-thumbnail doesn't have a shrink parameter? Is it done automatically in this operation?

I've reused your benchmark and changed a few things, especially the input file by using a larger file such as the Hong-Kong one.
I've also tried using the shrink option on the Image.open to see the difference.

image_path = Path.expand("test/support/images/Hong-Kong-2015-07-1998.jpg") # 8356x4700
dims = "500x200" # x2 = 1000x400
# 8356x4700 / 8 = 1044x587

open_and_thumbnail =
  fn shrink ->
    image =
      image_path
      |> Image.open!(shrink: shrink)
      |> Image.thumbnail!(dims)

    Image.write!(image, :memory, suffix: ".jpg", quality: 40, strip_metadata: true)
    Image.write!(image, :memory, suffix: ".avif", effort: 1, quality: 40, strip_metadata: true)
    Image.write!(image, :memory, suffix: ".webp", quality: 30, strip_metadata: true)
  end

Benchee.run(
  %{
    "Image.open |> Image.thumbnail (no shrink)" => fn -> open_and_thumbnail.(1) end,
    "Image.open |> Image.thumbnail (shrink of 2)" => fn -> open_and_thumbnail.(2) end,
    "Image.open |> Image.thumbnail (shrink of 4)" => fn -> open_and_thumbnail.(4) end,
    "Image.open |> Image.thumbnail (shrink of 8)" => fn -> open_and_thumbnail.(8) end,
    "Image.thumbnail(path)" => fn ->
      Image.thumbnail!(image_path, dims) |> Image.write!(:memory, suffix: ".jpg", quality: 40, strip_metadata: true)
      Image.thumbnail!(image_path, dims) |> Image.write!(:memory, suffix: ".avif", effort: 1, quality: 40, strip_metadata: true)
      Image.thumbnail!(image_path, dims) |> Image.write!(:memory, suffix: ".webp", quality: 30, strip_metadata: true)
    end
    },
  time: 20,
  memory_time: 2
)

On a large file (which tend to be common these days with smartphones oftenly creating files of several megabytes), and without shrink, it seems that the Image.thumbnail with the path input is favored.
However, if you can apply an optimal shrink on the Image.open (which in case of the Hong Kong file is 8 for an output of 500x200), then it becomes much faster.

Image.open |> Image.thumbnail (shrink of 8)         16.86       59.32 ms     ±5.08%       59.27 ms       67.93 ms
Image.open |> Image.thumbnail (shrink of 4)          2.37      422.28 ms     ±9.14%      416.99 ms      506.17 ms
Image.thumbnail(path)                                1.61      622.58 ms     ±1.92%      619.72 ms      686.25 ms
Image.open |> Image.thumbnail (shrink of 2)          1.20      830.67 ms     ±9.22%      843.32 ms      982.09 ms
Image.open |> Image.thumbnail (no shrink)            0.42     2404.69 ms     ±6.72%     2360.68 ms     2638.81 ms

Comparison:
Image.open |> Image.thumbnail (shrink of 8)         16.86
Image.open |> Image.thumbnail (shrink of 4)          2.37 - 7.12x slower +362.96 ms
Image.thumbnail(path)                                1.61 - 10.50x slower +563.26 ms
Image.open |> Image.thumbnail (shrink of 2)          1.20 - 14.00x slower +771.35 ms
Image.open |> Image.thumbnail (no shrink)            0.42 - 40.54x slower +2345.37 ms

Memory usage statistics:

Name                                           Memory usage
Image.open |> Image.thumbnail (shrink of 8)        31.66 KB
Image.open |> Image.thumbnail (shrink of 4)        31.66 KB - 1.00x memory usage +0 KB
Image.thumbnail(path)                              28.99 KB - 0.92x memory usage -2.66406 KB
Image.open |> Image.thumbnail (shrink of 2)        31.66 KB - 1.00x memory usage +0 KB
Image.open |> Image.thumbnail (no shrink)          31.66 KB - 1.00x memory usage +0 KB

[kipcole9]

Wow, this is great work @quentin-bettoum, really appreciate it and your results are close to what I expected earlier.

I was wondering why Image.thumbnail with a path as input which uses vips-thumbnail doesn't have a shrink parameter? Is it done automatically in this operation?

Correct, Operation.thumbnail/2 does a block shrink automatically. From the docs:

Make a thumbnail from a file. Shrinking is done in three stages: using any shrink-on-load features available in the file import library, using a block shrink, and using a lanczos3 shrink. At least the final 200% is done with lanczos3. The output should be high quality, and the operation should be quick.

That is why, in general, using Image.thumbnail/2 with a path as the image source should, for many use cases, be faster. But as you note, it doesn't open the image in access: :random mode which might be a good option to add and I'll open a discussion on the libvips repo.

Lastly, see how little memory is used? Its one of the really great things about the on-demand streaming nature of libvips.

[kipcole9]

Hmmm, I wonder if there is a way to estimate what the optimal shrink-on-load value might be for a given image and target thumbnail size?

[quentin-bettoum]

Lastly, see how little memory is used?

Yes, and I was thinking it might not be relevant, because if I use libvips in CLI it will indicate a memory high-water mark that is often much bigger than that (but still reasonable).

A few exemples with the Hong-Kong photo:

# Using thumbnail to output a 500x200
vips thumbnail Hong-Kong-2015-07-1998.jpg hong-kong.webp[Q=30,strip] 500 --height=200
memory: high-water mark 8,53 MB

Without thumbnailing:

vips webpsave Hong-Kong-2015-07-1998.jpg hong-kong.webp
memory: high-water mark 45,90 MB
# Its interesting to see that the `copy` will take a little bit more memory to do the same thing as `webpsave` above.
vips copy Hong-Kong-2015-07-1998.jpg hong-kong.webp
memory: high-water mark 47,43 MB

I've done these tests with some smaller images than the Hong-Kong one, and the memory high-water mark drops significantly.
From my tests, I found that the memory consumption is affected by the resolution of the input file and the wanted resolution of the output.
The format of the output (tested with webp, avif, and jpg) doesn't seem to have any effect on memory consumption though.

In the Benchee doc about Measuring Memory Consumption, it is said:

The measurement is limited to the process that Benchee executes your provided code in - i.e. other processes (like worker pools)/the whole BEAM isn't taken into account.

So I'm guessing that the external libvips processes aren't taken into account either.

Hmmm, I wonder if there is a way to estimate what the optimal shrink-on-load value might be for a given image and target thumbnail size?

That's what I will look for now, I will report back if I find a solution.

[kipcole9]

So I'm guessing that the external libvips processes aren't taken into account either.

Oh, indeed, very good point.

As for memory high water mark, you can get that same output with Vix/libvips by:

export VIPS_LEAK=true

It will output to stdout when you exit the application (iex, mix or whatever) cleanly.