Additional palette sorting algorithm

[andrews05]

This adds a new palette sorting algorithm that attempts to minimise entropy by an approximate solution to the Traveling Salesman Problem. The algorithm comes from "An efficient Re-indexing algorithm for color-mapped images" by Battiato et al (https://ieeexplore.ieee.org/document/1344033).
It's fast and effective and works in addition to the luma sort (which remains the single most effective sort). In order to keep lower presets fast though, I've only enabled this for o3 and higher.

Results on a set of 190 indexed images at -o5:
18,932,727 bytes - master
18,578,306 bytes - PR
18,559,863 bytes - PR + #509
(These images may be particularly suited to alternative sorting methods - the gains here are not necessarily what should be expected on average)

Note I looked into the 120 different palette sorting methods from TruePNG, as mentioned in #74 (and seen in action in the Zopfli KrzYmod fork). They're... largely ineffective. The combination of all 120 methods are outperformed by just the existing luma sort plus this new one. That's not to say there's nothing further to be gained from them, but trying to brute force all the combinations definitely seems like a bad idea. There are other algorithms I hope to explore in future...

@ace-dent Thought this might interest you

UPDATE: I realised a quick tweak to alpha values in the luma sort can provide a great improvement on images with transparency. The following numbers were taken with PR #509 as base.
-o2:
19,065,549 bytes - base (luma sort)
18,949,747 bytes - modified luma sort

-o5:
18,922,165 bytes - base (luma sort)
18,559,863 bytes - new sorting algorithm + luma sort
18,544,813 bytes - new sorting algorithm + modified luma sort

[ace-dent]

@ andrews05 - great addition!!
I did some work around palette sorting & filters a while back.

I discovered a very simple heuristic that is cheap to check and quite effective:

• Many indexed images have filter none, so 0 is written into the image stream for every row.
• It makes sense to check which contiguous runs of color are adjacent to the start and end of rows; to make the most common edge color the first palette entry 0, giving the longest runs of zero in the image stream.
• This minimizes the entropy of the filter code appearing in the stream. Other colors may then be sorted according to other methods.

E.g.

PS - I have a bunch of 'palette order sensitive' images I used for testing. I could share a zip with you?

[ace-dent]

Palette order sensitive images.zip

[ace-dent]

And historic discussion here:
https://encode.su/threads/3879-Optimizing-palette-order-for-indexed-png-images

[andrews05]

@ace-dent Nice, thanks for that! I ran some tests on your files:

270,856 original
270,749 master: -Zo6 --force --np (baseline, no change to palette)
267,823 master: -Zo6 --force
262,821 PR: -Zo6 --force
262,571 PR: -Zo6

I'll see if I can incorporate your heuristic and do some further tests sometime...

[edit] I've implemented your suggestion and see small but generally positive changes 🙂

262,349 PR: -Zo6

And the set of files from my first post is now down to 18,539,101 bytes.

[ace-dent]

@andrews05 thanks for implementing this! Wow :-D
Few comments:

1. I couldn't tell from the code, but are you look at both left and right edges? Obviously it is the consecutive runs of 0 we are trying to maximize.
2. I'm presuming palette sorting is done very early, before we know what filters are used (as is normally the case)? As you are aware, there are complex interactions between filters and palette order. If the line filters were known, this might effect the 'golden' edge color; i.e. if the majority of rows were filtered 2, we might place the promoted edge color to that palette position... or even do on a row-by-row basis rather than taking a modal average... but gets complex! :-/
3. Is this edge colour enforced with both luma and Battiato algorithms?
4. I haven't looked at how this interacts when you also have a tRNS table. Typically we order palettes to minimize size of tRNS chunk. At worst, I guess adding an opaque color to the start... adds 5 bytes? Fortunately real world graphics (e.g. icons) will have transparency at the image edges.
5. Finally: disclaimer! I only tested this with a small set of 'synthetic' images, where I kinda knew this would work. Please be sure it doesn't hurt compression!! In your testing, do you record images that are less compressed, in addition to those that benefit?

Cheers.

[ace-dent]

I did a quick survey of what other apps do. Have you considered testing chroma (/hue) sort? I certainly have test images that benefit from this. It seems reasonable to do (1) frequency – of some form, (2) chroma, (3) luma and (4) a more advanced statistical model (Battiato)... unless the 'Battiato' method is consistently equal or more performant than 1-3.

pngrewrite (& imageworsener)

Sort the palette to put transparent colors first, then sort by how frequently the color is used. Sorting by frequency will often help the png filters make the image more compressible. It also makes it easier for people to see which colors aren't used much and allow them to manually reduce the color palette.

OptiPNG
None, from my reading of the source code... I was sure it did some sorting !? Just removes unused palette entries.

Pngoptimizer
Tries Frequency and Luminance

Pngcrush
Weird but interesting!…

Implement palette-building (from ImageMagick-6.7.0 or later, minus the "PNG8" part) -- actually ImageMagick puts the transparent colors first, then the semitransparent colors, and finally the opaque colors, and does not sort colors by frequency of use but just adds them to the palette/colormap as it encounters them, so it might be improved.

PNGPalTest
Generates many options (inc. ascending / descending sort and calcs based on different color spaces!):

• Account for alpha first or ignore.
• Frequency / Luminance / Color
• Nearest neighbour (and weighted).

[ace-dent]

You may also be interested in this paper. The modified Zeng method (2016) seems to surpass Battiato (2004). Zeng was adopted by Google for WebM (source).

[andrews05]

1. Yup, see line 235:

   oxipng/src/reduction/palette.rs

   Lines 233 to 236 in f7ee78d

   	for line in png.scan_lines(false) {
   	counts[line.data[0] as usize] += 1;
   	counts[line.data[line.data.len() - 1] as usize] += 1;
   	}

2. Yeah, the alpha optimisation was easy to integrate with the filters since it can freely vary by line, but I don't know if that's really possible with the palette order. I thought the Battiato method was well suited for Sub (1) so I tried promoting the edge colour to that position but it was very much a regression on average.
3. Currently just Battiato - ran out of time to test it on luma too, but I will get there :)
4. Yeah, tRNS is interesting with indexed. Worst case you could place a single transparent colour last in a palette of 256 entries, leaving 255 redundant tRNS bytes in between. The Battiato sort doesn't consider the values of the colours, but the luma sort does ensure transparent entries come first. And oxipng does take into account the size of these chunks when evaluating which transformation is best.
5. I generally don't look too much at individual files - I tested on about 4 different sets of images and the edge colour selection did provide an average improvement on all of them. Any change is pretty much guaranteed to regress some file though 😅

Regarding other methods you mention: In my (albeit limited) experience, frequency (popularity) sort has been largely ineffective. I mean, it might look good compared to unsorted (or order of appearance) but pales in comparison to the luma sort. The zopflipng KrzyMod fork tries 120 combinations of these options, which were borrowed from TruePNG:

11. --palette_priorities=[types]

   palette priorities to try:
   p: popularity
   r: RGB
   y: Y'UV
   l: L*a*b*
   m: MSB
   By default, if this argument is not given, all strategies are tried.

12. --palette_directions=[types]

   palette directions to try:
   a: ascending
   d: descending
   By default, if this argument is not given, all strategies are tried.

13. --palette_transparencies=[types]

   palette transparencies to try:
   i: ignore
   s: sort
   f: first
   By default, if this argument is not given, all strategies are tried.

14. --palette_orders=[types]

   palette orders to try:
   p: none
   g: global
   d: distance
   w: distance, weighted by popularity
   n: distance, weighted by neighbor popularity
   By default, if this argument is not given, all strategies are tried.

As I mentioned in my first post, the results were actually disappointing. That said, I would like to narrow these down and find out if there are one or two specific combinations that show modest improvement when added to the luma and battiato sorts. (edit: either ydsn or ydsd seems best)

Finally, yes I do plan to try the mzeng method next! The reason I went with Battiato first was because 1. it has a much lower complexity (M²logM, vs M³ for mzeng), and 2. a later paper indicated it actually performed better for PNGs (most of the papers are testing with JPEG-LS). There's also a couple of others I hope to try, but they all take time to implement.

[andrews05]

I'm putting this on hold until #516 can be merged, as the sorting will be easier when we know we're always working in 8-bit.

[ace-dent]

2. I'm presuming palette sorting is done very early. ... If the line filters were known, this might effect the 'golden' edge color; i.e. if the majority of rows were filtered 2, we might place the promoted edge color to that palette position...

@andrews05 - Just thought: if the filters command line parameter is specified (and accessible?) then we know for filters 0 to 4 the row filter value and matching palette index to prefer.
E.g. if we are forcing a trial with -f4, we should try to put the most frequent edge color to index 4.

[andrews05]

@ace-dent I improved the co-occurrence matrix construction - your test files are now down to 261,712 bytes.

I experimented a little with altering the position for a specific filter but it did not give positive results. (Possibly because the evaluator always runs with fixed filters: None & Bigrams)

[ace-dent]

Re. filters: makes sense. Perhaps also diminishing returns -vs- added code complexity...
I'm excited to try all these enhancements in the next release! :-D

[shssoichiro]

I'm putting this on hold until #516 can be merged, as the sorting will be easier when we know we're always working in 8-bit.

Now that #516 is merged, are there any changes that need to be made to this MR?

[andrews05]

Yeah, I already sorted that out so this is good to go 🙂

[ace-dent]

@andrews05 - was Mzeng useful? Curious about your findings :-)

Update: I have also found images with very repeating patterns (so, e.g. filter=1 on all rows, etc.), where my edge color heuristic may fail. Please may you check for wins -vs- losses? https://github.com/ace-dent/8x8.me/tree/main/previews

[andrews05]

Oh yeah, I did experiment briefly with mzeng. On its own it averaged slightly better than battiato but when combined with the luma sort it averaged slightly worse. This was just on one set of files though (and the code was nowhere near production ready).
I do still hope to pursue this further sometime, I've just had other priorities.

I've just tried your "previews" folder - putting the popular edge colour first in the palette still comes out better than not doing that.

[andrews05]

Gah, there was a dumb bug I introduced in the clap update where fast mode was being disabled by default and reduced performance at level 2. I only found it while investigating a failing test here. All fixed now.