[RFC] Sixel support

[ctrlcctrlv]

Sixels are an ancient protocol for supporting graphics in the terminal, similar, but inferior to, Kitty's native graphics protocol.

The major way sixels are inferior is that there is no alpha and the color palette is limited, unlike Kitty's support for full RGBA.

Be that as it may, due to their age, many other terminals support them, which becomes a self-fulfilling prophecy, as people are more likely to add sixel support than Kitty graphics support.

So, I wonder what the right play is here. Implement sixels? Not hard to do, just translate the commands into Kitty graphics commands internally.

Or, given our market leader position, ignore sixels, and continue to ask other VTEs to implement our protocol?

[kovidgoyal]

On Sun, Apr 05, 2020 at 11:58:55AM -0700, Fredrick Brennan wrote: [Sixels](https://en.wikipedia.org/wiki/Sixel) are an ancient protocol for supporting graphics in the terminal, similar, but inferior to, Kitty's native graphics protocol. The major way sixels are inferior is that there is no alpha and the color palette is limited, unlike Kitty's support for full RGBA. Be that as it may, due to their age, many other terminals support them, which becomes a self-fulfilling prophecy, as people are more likely to add sixel support than Kitty graphics support. So, I wonder what the right play is here. Implement sixels? Not hard to do, just translate the commands into Kitty graphics commands internally.

sixels have other problems, no z-index, much harder to implement, no support for efficient data transmission when the emulator and program have a shared filesystem/memory. See the original issue for graphics support in kitty (it is linked to from the graphics protocol page) for a discussion.

Or, given our market leader position, ignore sixels, and continue to ask other VTEs to implement our protocol?

As I noted on the original issue, I have no interest personally in implementing an inferior graphics protocol in kitty. I know of no program that I would use that supports sixels but not kitty's graphics protocol. That said, I wont refuse a patch to implement it on top of the existing graphics support, provided I was reasonabl confident the person implementing it would stick around to maintain it as well. Although I suspect it wont be quite trivial, as the semantics of how images drawn with sixels work with scrolling, screen boundaries, cursor position and text output are probably different.

[kovidgoyal]

For reference: #33

[kovidgoyal]

Since no one else has comments, closing.

[ctrlcctrlv]

Sixel's are once again on my mind. They're supported in a lot of programs, including mpv (Cf. #3673, saitoha/libsixel README.md).

I'd appreciate this being reopened, but even if it's not, I'm going to spend tonight integrating libsixel as an optional build dependency into Kitty. It's a C library with a Python binding. I think I can do it in a few hours.

[kovidgoyal]

The hard part isnt parsing the escape codes, which libsixel will do for you, its adding support for the very different semantics that sixels based images have wrt to things like scrolling, text overwriting the image, what happens when the terminal resizes, etc.

If I were you I would instead spend this energy on adding support for the kitty graphics protocol to mpv :)

[ctrlcctrlv]

I have the basics of Sixel support working.

I'm using xterm -ti vt340 as the source of truth for what should happen in various situations. I feel like xterm is already a logical source of truth anyway.

I posted the code to try to answer some of @dankamongmen's questions. No, sixel images don't blow away Kitty graphics data. They work together, they're implemented through grman_handle_command!

[kovidgoyal]

Good progress, now try the following things:

1. Sending an image larger than the screen
2. Moving the cursor back and printing some text over the image

See if what happens matches xterm.

[ctrlcctrlv]

Test case 1

I fixed this by switching from grman_handle_command to screen_handle_graphics_command.

Test case 2

I'm still working on this one, as you can see it's quite different. Any ideas?

I generated this test case by appending this to the .sixel file…

\x1b[10Ahello\x0a

[ctrlcctrlv]

Actually after I adjusted the z-index to -1, it's just a matter of fixing the placement of "hello". I think that the white background of the cells in xterm should be optional, but if it's really important to you I can fix that too.

New test case 2

[kovidgoyal]

Not sure I am following about the white background. I think the sixel semantics are that both the foreground and background colors are displayed for text printed over an image. @dankamongmen can correct me if I am wrong. For you I think that means the z-index needs to be low enough that the image is displayed below the background layer (IIRC this was a feature of the graphics protocol you requested :)

It is important to have the cursor positioned at the same location post drawing a sixel as xterm does it.

[ctrlcctrlv]

Got it. -2 is correct.

I've discovered that xterm considers a Sixel sequence as having an implied newline, even if the file is noeol. If the sequence is followed by a newline, however, that's not an extra newline.

[ctrlcctrlv]

I'm making good progress with this. I'm also now the lead maintainer of libsixel.

I have tested a variety of programs, and so far they all work equal to xterm except w3m. The issue with w3m is that it expects drawing sixels over top of text to erase the text. Here's an image comparison of w3m -sixel 'https://commons.wikimedia.org/wiki/Category:Atlantic_City_High_School'

xterm

kitty

I'm thinking how to fix this. Looking through screen.h for a function I can call to clear a square of cells before I draw a sixel image on it.

[kovidgoyal]

I dont think there are any pre-built functions for this, but you can add one in screen.c similar to screen_erase_in_display. It can serve as the basis for implementing DECERA

[valderman]

Did anything ever come out of this? An upcoming sixel patch is mentioned in #3737, but as far as I can tell that's where the trail ends.

[ctrlcctrlv]

@valderman yes it did. But Kitty is a very security important application. After I took over the maintainership of libsixel I unfortunately decided it cannot support the security demands of Kitty, it is too insecure internally. I need to write a Rust library or something. But there is a patch above that should still compile which uses libsixel if you are curious, the screenshots weren't faked or anything...

[valderman]

@ctrlcctrlv ah, I see. Is there anything planned for this new security-conscious sixel library that one could help out with, or is it more a vague vision of the future at this point?

[ctrlcctrlv]

@valderman I, or someone else, needs to write one in Rust with a compatible C ABI to the normal libsixel. kitty can then link to that. Cf. #2701 (comment)

[voronoipotato]

I didn't even know there was a security focus for kitty, I just use it because it's fast enough, has themes, and supports ligatures. in my journey to find a pretty fast terminal that supports both ligatures and sixels, I've settled on wezterm.

[ctrlcctrlv]

If it's not @kovidgoyal's focus it's at least mine for my patches, all I can say :)

[Cuteistfox]

im forking it @ctrlcctrlv could you submit apatch after i fork(in a few minuts

[Cuteistfox]

see https://github.com/Meow-purr/Cat

[Kreijstal]

so, how is sixels progress going? are there any emulators besides xterm that support sixels?

[ErrorNoInternet]

are there any emulators besides xterm that support sixels?

https://arewesixelyet.com/

[artcancrotp]

KiTTY's graphics protocol is superior to sixel, but then again, Betamax was superior to VHS, and we all know how that ended. sixel support please. Not just in KiTTY but everywhere.

[Kreijstal]

KiTTY's graphics protocol is superior to sixel, but then again, Betamax was superior to VHS, and we all know how that ended. sixel support please. Not just in KiTTY but everywhere.

this

[RobertWHurst]

Except that VHS was the superior format. Most people confuse the version of betamax used for professional mastering with the consumer version. They weren't even compatible. The consumer version of betamax was pitiful in comparison to vhs, which is why it failed.

Technology Connections did a series on this https://youtu.be/FyKRubB5N60?si=ysNxgfb-og4qtx6E

[artcancrotp]

Except that VHS was the superior format. Most people confuse the version of betamax used for professional mastering with the consumer version. They weren't even compatible. The consumer version of betamax was pitiful in comparison to vhs, which is why it failed.

You're wrong about that, but it also doesn't matter. At this point it's all over but the shoutin' -- Sixel has won. The ecosystem around Sixel continues to expand rapidly -- heck, it's even getting merged into Windows Terminal as we speak. Tools to generate Sixel graphics, people using it for data visualization, and many terminals supporting it -- there is no doubt that we are headed into an age where supporting something other than Sixel for terminal bitmap graphics is like supporting something other than ANSI for color and cursor control.

Don't get me wrong, KiTTY graphics is excellent but if the superior technology always won we'd all be using Amigas right now.

[SolitudeSF]

Sixel has won

nuh uh

[artcancrotp]

nuh uh

Thank you for that insightful, fact filled, and thought provoking response.

It's an open source project and Kovid can do whatever he wants with it, of course. But it is now the current year and writing a terminal that supports some other graphics protocol instead of Sixel is like writing a terminal that supports (for example) Wyse 50 terminal sequences instead of ANSI.

I switched from KiTTY to a different terminal program for exactly this reason. I'm not alone.

[SolitudeSF]

Cool analogy. Too bad it doesn't make sense.

[artcancrotp]

Cool analogy. Too bad it doesn't make sense.

That's because it all happened before you were born. Every terminal had a different language and we needed cumbersome abstraction layers to build software that could deal with the differences. Today, every terminal supports ANSI and you can count on the ESC [ codes being available everywhere. Everyone converged on the language originally conceived by DEC in 1978 for the VT100. Other terminal control languages are not just abandoned but practically forgotten.

Learn from technology history. So far all you've contributed to the conversation is rude quips. We are now in the same situation with terminal raster graphics where we were with basic terminal control codes in the 1980s. But time marches on and convergence happens. The world is converging around Sixel whether you like it or not.

[erwin]

Might be relevant

Window Terminal adds Sixel support

microsoft/terminal#17421

[artcancrotp]

Window Terminal adds Sixel support

There's a very spirited discussion that has been going on for a long time over there. It's a pretty sad state of affairs that Microsoft (MICROSOFT!) is acknowledging and adopting the de facto standard (sixel) while KiTTY continues soldiering forth with its own incompatible format.

I stopped using KiTTY because it has no Sixel roadmap and the world is quickly standardizing on Sixel.