PNG Image Support

[clloyd]

In order for use The Grid as an image source/picker in TagManager we need the ability to add .png files to the grid.

A standard contributor image is a transparent png (https://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2014/3/13/1394733751535/ZoeWilliams.png)

[paperboyo]

Hello, @Reettaphant, @kenoir,

Since I don’t know what would be best place to store some pseudo-research, I will keep it here unless told otherwise. Also, I don’t really know much about this, I may be wrong all the time. I definitely confuse terminology (alpha and transparency are not the same).

PNGs come in different flavours and have to be treated and served differently because of that. We have to be able to identify these flavours using some image characteristics. We may even need to pass some of that to frontends for resizer needs (through CAPI?! 🙀).

Here is a preliminary, not exhaustive list of things we will find useful (and which should be indexable/searchable):

• TrueColor
  • 8- or 16-bit per component
  • with or without alpha
• Paletted
  • with 1-bit alpha (there is just one colour in the pallete with alpha=0) or with 8-bit alpha (with pixel(s) in the palette with values of alpha 0-255)
  • having different no. of colours≥2 (some of which may have different amounts of transparency)

Below I will try to see how GraphicsMagick 1.3.23 and ImageMagick 6.9.3-7 could help us here using identify -verbose (similarly to #1281 (comment)).

	IM	GM
TrueColor 8bpc alpha	Type: TrueColorAlpha Depth: 8-bit	Type: true color with transparency Depth: 8 bits-per-pixel component
TrueColor 16bpc alpha	Type: TrueColorAlpha Depth: 16-bit	Type: true color with transparency Depth: 16 bits-per-pixel component
TrueColor 8bpc no alpha	Type: TrueColor Depth: 8-bit	Type: true color Depth: 8 bits-per-pixel component
Paletted 256 colors with 1-bit alpha	Type: PaletteAlpha Colors: 255	Type: true color with transparency No colors info ❗
Paletted 256 colors with 8-bit alpha	Type: PaletteAlpha Colors: 256	Type: true color with transparency No colors info ❗
Paletted 64 colors with 8-bit alpha	Type: PaletteAlpha Colors: 64	Type: true color with transparency No colors info ❗

Observations/links:

• IM better than GM
• one possible alternative to IM & GM, pngcheck -v can’t distinguish between Paletted 256 colors with 1-bit alpha and Paletted 256 colors with 8-bit alpha, unless we decipher chunk tRNS at offset 0x00643, length [not 256]: [not 256] transparency entries as meaning 8-bit alpha (?)
• both IM & GM spit out image size as Geometry: [width]x[height][+x+x]
• IM does not preserve XMP metadata when converting PNG to JPG
• all IM’s image types: convert -list type; all GM’s image types: http://www.graphicsmagick.org/api/types.html#imagetype
• scary 😱 PNG manual: http://www.libpng.org/pub/png/book/toc.html
• preventing pngquant from removing ICC colour profile (and all other metadata): https://github.com/pornel/pngquant/blob/master/rwpng.c#L303 (all above images have AdobeRGB profile embedded, but last two - couldn’t find any implementation of pngquant, inc. Photoshop (sic!), that storess ICC info in its output. If I read it correctly, spec allows that. Also, we may need to be aware that PNG can store colur space characteristics independently of ICC (http://www.libpng.org/pub/png/book/chapter10.html)
• one way to flatten a transparency in PNG using a specified colour: convert f45cf7f14e43297ecbc16a87c14ecc6166864c16_png8_64_trans.png -background skyblue -alpha remove -alpha off f45cf7f14e43297ecbc16a87c14ecc6166864c16_png8_64_matte.jpg (http://imagemagick.org/Usage/masking/#alpha_background)
• more to definitely come, most unfortunately

Also: super-sorry for huge 🌵!

Regards
Mateusz

[paperboyo]

Hello,

Just food for thought if we decide that we want to allow a “crisp graphic with no transparency” PNG “type” in Grid in addition to allowing “this should really be a JPG” and “cutout” types.

Here is an image that would be a good candidate and exhibits characteristics specific to images of this class. The original in Grid currently is a JPG of 360KB (hash: d52ee9abe87d3168aba1d2cc1d41915a18927e58). Educating authors to supply similar images in PNG8 without transparency would allow for Grid to identify them easily as “graphic” type. But it will never be foolproof.

This JPG with our current resizer settings produces unnecessary artifacts which is but one argument for being able to create a special pipeline for this type of images:

Here is this same image converted to PNG24 (90KB):

Same image is ~45KB as 256-colour PNG8 and still looks good as a 15-colour PNG8 of ~17KB when decided automatically by pngquant (could be used for final to-master conversion, should be compiled not to remove ICC profiles):
pngquant.exe --quality=45-90 [JPG_converted_losslessly_to_PNG24].png :

There are two distinct problems if we decide to create a pipeline for similar graphic-like crisp imagery:

1. identifying candidates for this treatment either programmatically or by user’s concious choices (or both)
2. ensuring that master’s do not bloat downstream

Ad. 1. If there is no transparency and the filetype is PNG8 we may quite safely assume that image was created with exactly this type in mind. We could introduce a filesize check if the master is too big and then push such image through “this should really be a JPG” pipeline instead (e.g. for photographs erroneously saved as PNG8).
Much bigger problem is how to identify such images when they are JPGs and PNG24. We may want to give users a choice or a special lane for upload (e.g. for Visuals), but this comes with its own set of problems (least of all - UX).
One way to identify them programmatically would be to decide on a cut-off for the no. of unique colours in an image (for JPGs & PNG24s).

gm identify -format %k "[original JPG].jpg" outputs 1606 as unique no. of colours for a JPG original of d52ee9abe87d3168aba1d2cc1d41915a18927e58 (probably because of JPG artifacts). No photographic source would ever return that low a number. There must be other ways we could use to make identification more robust but that’s waay above my understanding or Google-fu.

Ad 2. There are two ways - either provide a compressed master (pngquant used similarly to above, probably) or pass on enough info to the frontend’s resizer to perform a compression on it’s own. My tests indicate that the second option would need to pass no. of output colours (15 here, which we wouldn’t have known without pngquant anyway) to be used for resizer’s colorquant parameter (otherwise resizer returns unncessarily big image of 255 colours when asked for PNG8). Situation may change soon as our resizer is close to getting an imaging pipeline overhaul. So a decision on that may better wait.

Introducing this third type/class of images would mean that we would need to introduce a mechanism to pass information to frontends (beyond of relying simply on two distinct image extensions of JPG & PNG).

I hope there is at least some use to the above.

Regards
Mateusz

[paperboyo]

Hello,

Just testing cropping of TrueColor PNGs with alpha on TEST! @Reettaphant: Absolutely exceptional work! Wow! 16bpc works like a charm (master 8bpc), pngquant for crops!

Grid's 317px crop (L) and original converted to sRGB from ProPhotRGB, resized (Bicubic Sharper) to 317px, saved as PNG24 in Photoshop, run through ImageAlpha (pngquant GUI) and ImageOptim (~4KB reduction) (R):

[L: 73KB; R: 72KB; amazing match! we may at some point revisit either our resizing kernels or sharpening or both, but that's not PNG-specific]

Couple of loosely connected points:

• super-minor: is bit depth reduction at the end or beginning of imaging operations?
• thumbnails need baking into a background we decide on properly (or if we want it to be changeable we may look at having them as pngquanted PNGs*)
• thumbnail has a jpg extension (minor)
• here the original is 39MB, imgops preview image is 2.9MB, preview view of fullframe crop is 3.8MB (it is a fullsize crop, actually) - this is not sustainable, definitely not Jonny-sustainable
• we may look into pngquant keeping sRGB profile (or, rather, profiles in general)
• we should think about normalising metadata e.g. colour model for JPGs and PNGs
• we may want to look at XMP
• we may want to actually try to get some life some time

*pngquanted PNG thumb made manually from 316px crop:

22KB
Grid's usual JPG thumb:

12KB

Regards
Mateusz

[Reettaphant]

@paperboyo Glad to hear it works!

In response to your queries and comments:

• Depth reduction is done at the end of the operations, does this make sense?
• Yes, thumbnail backgrounds are something we might look into later
• Thumbnails are actually jpegs, graphicsmagick generates jpegs automatically
• So does this mean we are going to have to run pngquant in imageops as well
• I've added the rest to my list

[paperboyo]

Depth reduction is done at the end of the operations, does this make sense?

Absolutely, may mean some slight speed reduction, but is more correct imaging-wise...

We have recently reduced the size of our thumbs considerably (#1621, among others), PNGs aren't nearly as prevalent as JPGs - all this makes me think that we may decide it's more flexible to have them as pngquanted PNGs. But how this is affecting our setup - I wouldn't know. So if hard/impossible - there is a way to burn them into a background of our choice properly - I just forgot the GM command. And Slack has forgotten it too...

[paperboyo]

Hello,

Testing cropping of paletted PNGs with alpha... All is well. There is one small thing. The master crops (that do not go through pngquant) are bloated compared to original (if it’s optimised). They can be losslessly optimised.

Example 5c8055055377ded723c241eb118b50f8c5dd9441:
Original: 137KB
Master fullframe: 198KB

We may not care about it (since there is a resizer later on anyway). But we also may :-)

Running masters through pngquant reduces the weight, but removes colour profile and introduces unnecessary changes to the pixel structure, since it’s a lossy process.

I have not found a way to optimise them through GM only. Here are some alternative libraries (all output pixel-to-pixel identical to the original):

pngout [original].png /kiCCP [output].png 139KB (5,63 seconds) version: 13 Feb 2015

OptiPNG hasn’t produced substantial savings on the test file.

PNGCrush hasn’t produced substantial savings on the test file.

AdvPNG hasn’t produced substantial savings on the test file.

CryoPNG hasn’t produced substantial savings on the test file.

This is not proper test, as there is only one file (can’t upload from home). It’s entirely possible that on some other files the results would differ... And 5,6s for a file that small is a significant hit. Will keep looking...

Regards
Mateusz

[paperboyo]

Adjusting arguments to
pngout [original].png /kiCCP /s2 [output].png
gives 144KB in just 0,8 second.

[paperboyo]

This is now (mostly) working, TagManager can start integrating Grid’s output and Grid itself*, @clloyd. Let me know if I can help in any way :-)

*I think it would be very helpful, if an upload through TagManager’s instance of modal Grid could draw the metadata from TagManager itself, for instance.

[clloyd]

@paperboyo Yep! Offered it to anyone willing to take a stab at the TagManager codebase, if no one bites, then seems like a good flaw Friday project for myself. Lets continue discussion in guardian/tagmanager#233

[kenoir]

Done.