Extract RGB pixels and blurhash

[stiang]

This might be trivial for people who understand image formats better than me, but is there currently a way to use this library to extract the RGB pixels from an image, so that they are suitable for use with BlurHash?

Here is the example code from the blurhash-elixir package:

# Pixel data supplied in RGB order, with 3 bytes per pixels.
pixels = [255, 43, 20, 11, 0, 155, ...]

hash = BlurHash.encode(pixels, 30, 30, 4, 3)

I’m looking for a way to get the pixels in the above code from an image opened with Image.open/2.

[kipcole9]

Since you need the pixel data as a list I think the following is the most straight forward:

{:oh, image} = Image.open("./test/support/images/example.webp")
{:ok, binary} = Vix.Vips.Image.write_to_binary(image)
pixel_list = :binary.bin_to_list(binary)
[100, 177, 247, 100, 177, 247, 100, 177, 247, 98, 176, 245, 95, 172, 242, 94,
 171, 241, 94, 171, 241, 89, 171, 242, 81, 171, 245, 76, 171, 246, 76, 171, 246,
 76, 171, 246, 76, 171, 246, 76, 171, 246, 76, 171, 246, 76, 171, 246, 78, 172,
 ...]

A couple of observations:

1. The blur hash implementation you referenced appears to expect 4 bytes per pixel, so you may need to Image.add_alpha/1 to the image. add_alpha/1 will only add an alpha band if there isn't one already so its safe to use on any image.
2. The blur hash implementation has a heavy reliance on Enum.fetch!/2 on lists. This isn't a very idiomatic thing to do and will be very slow on large images so resizing the image first will likely be important. Something like Image.thumbnail("path/to/image.jpg", longest_siide) would be the fastest.
3. The code above assumes the image is a sRGB image, 8 bits per band. If there is any situation where that is not the case you will need to convert the image first. See Image.to_colorspace/2 and Image.cast/2.

Implementing blurhash has been on my to-do list for a while so I'll take a look at a more idiomatic implementation in Image. There are some opportunties for improved performance like operating on binary data not lists, converting the image to linear sRGB natively and so on.

[stiang]

Thank you, this is better than what I came up with myself while I waited (and after I discovered get_pixel):

Enum.map(0..(height - 1), fn y ->
  Enum.map(0..(width - 1), fn x ->
    {:ok, pixel} = Image.get_pixel(thumb, x, y)
    pixel ++ [0]
  end)
  |> List.flatten()
end)
|> List.flatten()

It sort of works, but a native (and faster) implementation would be very welcome. I spent a good amount of time struggling with "out of bounds" errors because I tried returning just the 3 bytes that the example code suggests is needed, it wasn’t until I studied the source that I discovered it wanted 4. I figured it had something to do with alpha, so I crudely hard-coded it to 0 just to see what happened, and surprisingly it worked.

I am adding the blurhash in an asynchronous job that works with a tiny thumbnail in a fixed image format (without an alpha band), and performance isn’t really an issue at this point, but I would still really like to see blurhash supported natively in Image (especially encoding).

Image is quickly becoming the solution for anything image related in Elixir, and I love it. Thank you for creating it!

[kipcole9]

Thanks for the feedback and the kind words, much appreciated. And I'm glad you're at least up and running. It might take me a couple of days but I will work on an implementation leveraging the good work already done in the lib blurhash-elixir lib. I'll let you know when it's ready for a test drive.

Some other placeholder techniques I've been looking at you might be interested in:

• Thumbhash
• Squip
• Primitive (Image to SVG)

[stiang]

That sounds great, looking forward to it! I found another Elixir blurhash library that might also be worth a look when implementing this.

Those other techniques look very interesting. Thumbhash, especially, seems better than blurhash in almost every way, but my goal in the current project is to be able to use the blurhash with expo-image in a React Native app, since that image component supports showing a blurhash placeholder by providing it as a simple prop.

So I hope your implementation will at least support blurhash as one of the options, but I’m pretty sure I would also generate a thumbhash and store that too for each image, if Image were to support it - they only take up a few bytes each, and I could then switch to thumbhash when/if expo-image adds support (or I could roll my own image component).

[kipcole9]

The rinpath/blurhash library looks a good implementation to build upon. I think you should be able to use it directly with:

{:oh, image} = Image.open("./test/support/images/example.webp")
{:ok, binary} = Vix.Vips.Image.write_to_binary(image)
Blurhash.encode(binary, Image.width(image), Image.height(image), components_x, components_y)

[kipcole9]

I've pushed a commit that implements Image.Blurhash.encode/2 using the code from rinpatch_blurhash that you referenced. I didn't plan to vendor the code, but it needed some work to compile cleanly and it hasn't seen activity for a while so it seems prudent to do so.

Would you consider giving this a try? Just configure image with {:image, github: "kipcole9/image"}. It's a reasonably large download because of the test images but I think it should be manageable.

[kipcole9]

I've added Image.Blurhash.decode/3. I think this is good to release now, as long as your testing confirms it works in your use case. Let me know?

[stiang]

Hello again, I’m not sure if this will be seen by anyone, but I have come across a problem with encode/2 that I don’t know how to solve. For some (PNG) images I get the following error: Image must have 3 bands. Found 4 bands in the image. I can see in the source code that there is an explicit check that the image has exactly three bands.

These are images uploaded from a React Native app to my Elixir API, so there shouldn’t be anything special about them.

Does this have to do with transparency - is there an extra band for that information? And is there some way to remove the "least important" band without fundamentally changing the image?

[kipcole9]

The 4th band in a png images is the alpha band that controls transparency. It's quite possible that simply discarding the transparency layer is acceptable for a blur hash. Would you mind a bit of an experiment?

Try Image.Blurhash.encode(image[0..2]) and see if the results are acceptable to you? If we reach the conclusion that removing the alpha band doesn't produce a materially worse result then I can force that to happen. If it does produce a materially worse blur hash then I'll have to think how we might deal with that.

[kipcole9]

There are two other experiments you might consider trying, which are to flatten the image with different background colours. Note that blur hash, by definition, does not support alpha bands so the image has to be flattened in some fashion.

The two things I'd suggest trying:

image
|> Image.flatten()
|> Image.Blurhash.encode()

and

image
|> Image.flatten(background_color: :white)
|> Image.Blurhash.encode()

[stiang]

Yes, I actually discovered that Image.flatten/2 solves the issue before I saw your reply.

I am perfectly happy with this solution - as you say, the blurhash spec doesn’t support transparency, so it has to be removed somehow. Maybe it’s best to leave the flattening to the user, so that we are free to choose the best background color for each case (ie. sometimes black is better, other times white)?

Do you still want me to perform the experiment in your first comment? I’d be happy to if it’s useful to you, but I suppose the result should be similar to Image.flatten/2 using the same (default) background color as image[0..2] would produce?

Thanks for the quick and thorough reply!

[kipcole9]

If you've got a path forward, no experimentation required. As you say, this pre-processing step should live outside blur hash. I'll make the documentation a bit clearer.

[kipcole9]

Image version 0.44.0 is now published with the following changelog entry:

Enhancements

• Adds Image.Blurhash.encode/2 and Image.Blurhash.decode/1 to encode and decode blurhashes. Based upon a fork of rinpatch_blurhash. Thanks to @stiang for the suggestion. Thanks very much to @rinpatch for the implementation.