"sub-pixel" glyphs?

[piranna]

Instead of just only use lower block glyph, try to detect actual image shape and use glyphs that better adapt to it, like diagonals and so.

[niutech]

Do you mean derasterize?

[piranna]

Yes! Sort of :-P Probably that can be used, yes :-)

The proposed option of using sixels at #11 could be another alternative too :-)

[fathyb]

Definitely a good idea! Someone shared https://github.com/matrach/img2unicode which also produces pretty good results.

[piranna]

Definitely a good idea! Someone shared matrach/img2unicode which also produces pretty good results.

This is more alike I was asking about, but also taking in consideration triangular shape glyhps and diagonals and so, so they can provide a better adjustment. I think there was a project that was using Machine Learning to get the Unicode character that better fits the actual image...

[matrach]

I think there was a project that was using Machine Learning to get the Unicode character that better fits the actual image...

Hi all.

My proof of concept tool (https://github.com/matrach/img2unicode) uses some ML -- optimization with (Approximate) Nearest Neighbors to be exact. For each chunk of an image (i.e. 16x32 px), the tool basically selects a glyph (from a prerendered dataset) that optimizes both:

• perceptual similarity (implemented as a pixel-by-pixel Euclidean distance between the blurred glyph template and the chunk),
• visually matching edges (as you can see in the Obama example).

The need to use ML arose from the need to support arbitrary Unicode glyphs. This is not easily portable while maintaining the rendering quality, because there is a lot of variability between rendering by different terminal backends (e.g., libvte, kitty, etc.) and fonts.

The triangular shapes were added to Unicode 13 (Legacy Computing Symbols) in 2020, and at the time of writing of the tool, they were rarely supported by terminal rendering backends. (And probably are still not supported on Ubuntu 20.04 LTS' default terminal.)

If you haven't already, I'd suggest looking at tools like https://github.com/dankamongmen/notcurses that can transparently select image rendering with sixels, Kitty's Terminal graphics protocol, or fallback to standard blocks.

[piranna]

Thank you for the detailed explain :-D

[fathyb]

I considered notcurses, but it does not seem optimized for real-time rendering at 60fps unfortunately. For example, a good SIXEL implementation for us would need adaptative quantization (only run it on a few frames and re-use the palette on others) as it's the most expensive step.

I've been experimenting with this sub-pixel idea, and ended up with this pretty fast algorithm using a mask and a table lookup (<1ms per frame, seems to be auto-vectorize well on arm64):

carbonyl/src/output/quad.rs

Lines 3 to 41 in 50c2547

	pub fn binarize(a: Color, b: Color, c: Color, d: Color) -> (&'static str, Color, Color) {
	// Step 1: grayscale
	let lum = Color::new(0.299, 0.587, 0.114);
	let (x, y, z, w) = (
	a.cast::<f32>().dot(lum),
	b.cast::<f32>().dot(lum),
	c.cast::<f32>().dot(lum),
	d.cast::<f32>().dot(lum),
	);
	// Step 2: luminance middlepoint
	let min = x.min(y).min(z).min(w);
	let max = x.max(y).max(z).max(w);
	let mid = min + (max - min) / 2.0;
	// Step 3: table lookup using binary threshold mask
	TABLE[((x > mid) as usize) << 0
	| ((y > mid) as usize) << 1
	| ((z > mid) as usize) << 2
	| ((w > mid) as usize) << 3](a, b, c, d)
	}
	const TABLE: [fn(Color, Color, Color, Color) -> (&'static str, Color, Color); 16] = [
	|x, y, z, w| (" ", x.avg_with(y).avg_with(z).avg_with(w), Color::black()),
	|x, y, z, w| ("▖", x.avg_with(y).avg_with(z), w),
	|x, y, z, w| ("▗", x.avg_with(y).avg_with(w), z),
	|x, y, z, w| ("▄", x.avg_with(y), z.avg_with(w)),
	|x, y, z, w| ("▝", x.avg_with(z).avg_with(w), y),
	|x, y, z, w| ("▞", x.avg_with(z), y.avg_with(w)),
	|x, y, z, w| ("▐", x.avg_with(w), y.avg_with(z)),
	|x, y, z, w| ("▟", x, y.avg_with(z).avg_with(w)),
	|x, y, z, w| ("▘", y.avg_with(z).avg_with(w), x),
	|x, y, z, w| ("▌", y.avg_with(z), x.avg_with(w)),
	|x, y, z, w| ("▚", y.avg_with(w), x.avg_with(z)),
	|x, y, z, w| ("▙", y, x.avg_with(z).avg_with(w)),
	|x, y, z, w| ("▄", x.avg_with(y), z.avg_with(w)),
	|x, y, z, w| ("▛", z, x.avg_with(y).avg_with(w)),
	|x, y, z, w| ("▜", w, x.avg_with(y).avg_with(z)),
	|x, y, z, w| ("█", Color::black(), x.avg_with(y).avg_with(z).avg_with(w)),
	];

Results are interesting, note the building edges (just doing images for now):

Current	Quadrant binarization

[piranna]

Quadrant binarization looks pretty good! :-) I was personally thinking about the triangular glyphs, but your algorythm using quadrants works pretty well, and doubt we can get something better without seeing some artifacts due to not having homogeneous "pixels", good work :-)

[patsie75]

Not sure if it's useful, but I have made an image viewer in the awk scripting language which uses the unicode quadrant blocks, making use of some self made up color mixer. If it's useful then feel free to use it. if it's not, then that's fine.
The awk script (and some explanation in the README) can be found here: https://github.com/patsie75/awk-hires

[niutech]

@patsie75 Your code is licensed under GPL-3.0 and Carbonyl is Modified BSD. I'm not sure they are 100% compatible: "The GPL would allow you distribute the interface within another work only if that whole larger work were licensed under the GPL."

[piranna]

No, they are not compatible, but since they are in different languages (awk vs rust), I'm not sure if the translation would be covered by the GPL license...

[matrach]

carbonyl/src/output/quad.rs

Lines 3 to 41 in 50c2547

	pub fn binarize(a: Color, b: Color, c: Color, d: Color) -> (&'static str, Color, Color) {
	// Step 1: grayscale
	let lum = Color::new(0.299, 0.587, 0.114);
	let (x, y, z, w) = (
	a.cast::<f32>().dot(lum),
	b.cast::<f32>().dot(lum),
	c.cast::<f32>().dot(lum),
	d.cast::<f32>().dot(lum),
	);
	// Step 2: luminance middlepoint
	let min = x.min(y).min(z).min(w);
	let max = x.max(y).max(z).max(w);
	let mid = min + (max - min) / 2.0;
	// Step 3: table lookup using binary threshold mask
	TABLE[((x > mid) as usize) << 0
	| ((y > mid) as usize) << 1
	| ((z > mid) as usize) << 2
	| ((w > mid) as usize) << 3](a, b, c, d)
	}
	const TABLE: [fn(Color, Color, Color, Color) -> (&'static str, Color, Color); 16] = [
	|x, y, z, w| (" ", x.avg_with(y).avg_with(z).avg_with(w), Color::black()),
	|x, y, z, w| ("▖", x.avg_with(y).avg_with(z), w),
	|x, y, z, w| ("▗", x.avg_with(y).avg_with(w), z),
	|x, y, z, w| ("▄", x.avg_with(y), z.avg_with(w)),
	|x, y, z, w| ("▝", x.avg_with(z).avg_with(w), y),
	|x, y, z, w| ("▞", x.avg_with(z), y.avg_with(w)),
	|x, y, z, w| ("▐", x.avg_with(w), y.avg_with(z)),
	|x, y, z, w| ("▟", x, y.avg_with(z).avg_with(w)),
	|x, y, z, w| ("▘", y.avg_with(z).avg_with(w), x),
	|x, y, z, w| ("▌", y.avg_with(z), x.avg_with(w)),
	|x, y, z, w| ("▚", y.avg_with(w), x.avg_with(z)),
	|x, y, z, w| ("▙", y, x.avg_with(z).avg_with(w)),
	|x, y, z, w| ("▄", x.avg_with(y), z.avg_with(w)),
	|x, y, z, w| ("▛", z, x.avg_with(y).avg_with(w)),
	|x, y, z, w| ("▜", w, x.avg_with(y).avg_with(z)),
	|x, y, z, w| ("█", Color::black(), x.avg_with(y).avg_with(z).avg_with(w)),
	];

Note that thresholding by luma is probably suboptimal, since if you have a block such that a top half is red and a bottom half is green (with matching luminescence), then this algorithm would use a full block instead of ▀.
Please also note that the averaging of the color channels will give good results in a color space that is designed to do so.
Moreover, half of the quadrants are redundant because the background and foreground are reversed.

The img2unicode algorithm for block glyphs (glyphs rendered with consistent brightness without grayscale) is easily vectorized since it is just a matrix (tensor, really) multiplication.
https://github.com/matrach/img2unicode/blob/1f98475ee5d1e97ab712ead0bc31eab87127ec44/img2unicode/dual.py#L169

Thanks to this topic, I've finally described how the optimization for block-shapes works 😄 : https://github.com/matrach/img2unicode/blob/master/README.md#details-of-fastgenericdualoptimizer
It should be possible to even implement this method as a GPU-backed shader.

By the way, a while ago I also had prepared a selection of images for the evaluation of such methods. You can find them here: https://github.com/matrach/img2unicode-demos/tree/master/images

[fathyb]

I just merged this feature to master, I also added a --bitmap and a --zoom flag to get this:

$ docker run -ti -e COLORTERM=24bit fathyb/carbonyl:next https://youtube.fr --zoom=300 --bitmap

CleanShot.2023-02-14.at.05.44.26.mp4

Note that thresholding by luma is probably suboptimal, since if you have a block such that a top half is red and a bottom half is green (with matching luminescence), then this algorithm would use a full block instead of ▀.

I did notice that and fixed it by always using two blocks:

carbonyl/src/output/quad.rs

Lines 21 to 39 in 6747498

	// Step 3: average colors based on binary mask
	match FourBits::new(a > mid, b > mid, c > mid, d > mid) {
	B0000 => ("▄", x.avg_with(y), z.avg_with(w)),
	B0001 => ("▖", x.avg_with(y).avg_with(z), w),
	B0010 => ("▗", x.avg_with(y).avg_with(w), z),
	B0011 => ("▄", x.avg_with(y), z.avg_with(w)),
	B0100 => ("▝", x.avg_with(z).avg_with(w), y),
	B0101 => ("▞", x.avg_with(z), y.avg_with(w)),
	B0110 => ("▐", x.avg_with(w), y.avg_with(z)),
	B0111 => ("▘", y.avg_with(z).avg_with(w), x),
	B1000 => ("▘", y.avg_with(z).avg_with(w), x),
	B1001 => ("▌", y.avg_with(z), x.avg_with(w)),
	B1010 => ("▚", y.avg_with(w), x.avg_with(z)),
	B1011 => ("▝", x.avg_with(z).avg_with(w), y),
	B1100 => ("▄", x.avg_with(y), z.avg_with(w)),
	B1101 => ("▗", x.avg_with(y).avg_with(w), z),
	B1110 => ("▖", x.avg_with(y).avg_with(z), w),
	B1111 => ("▄", x.avg_with(y), z.avg_with(w)),
	}

Agreed on luma being suboptimal, but it's pretty fast to compute. I'm going to take a deeper look at your description, not very well versed in math 😅 But feel free to open a PR, that's definitely something I want.