Extension Kitty graphics protocol to support fonts

[ctrlcctrlv]

This issue continues discussion in #2145—

As I see it, we have two problems to figure out:

• the window metrics, that is, the shape of the cells;
• and, the font itself.

We've basically worked out the font itself. It will be SFNT, only some certain PUA codepoints.

The metrics though are an open question. If we just change up the symbol_map, it won't look same on all terminals. Thoughts @kovidgoyal ?

[kovidgoyal]

I'm not sure what the problem is?? The cell metrics are anyway set by
the main window font, which is not going to change. Characters from the
newly specified font will simply have to be fit within existing cells,
just as for fallback and symbol_map fonts.

[ctrlcctrlv]

I'm not sure it's very useful if metrics are not changeable, so I guess I am proposing to let applications change main window font as well.

Game developers need to know the metrics, at least within a certain threshold. Problem is, fonts can vary too much; from thin fonts like Sudo to wide ones like Press Start 2P.

[ctrlcctrlv]

The easiest way to do this is to enforce—

• one window = one process.

You might not be willing to do that though. I'm not convinced yet that it's impossible to make it per-window. It'll just take some work, which I'll try to do if it's needed.

[kovidgoyal]

I dont get the use case for changing the entire window font in a game?? As I
understand the initial proposal the idea is to be able to use fonts to
display small sprites easily without resorting to using actual
graphics??

[ctrlcctrlv]

Well for the simple Snake case, it looks much better with square glyphs than not.

This is doubly true for mazes.

[kovidgoyal]

Why do the cells have to be square for this? You can use two cells to get a square sprite. Your sprite font would then need to split up each sprite into two or alternately simply use a space after the code point (for PUA characters followed by spaces, kitty will auto-create a ligature of the character+spaces) so you can have square PUA glyphs.

[ctrlcctrlv]

If I use two cells, then if the user has a square (or squarish) font, it will look very strange.

[kovidgoyal]

I doubt there are squarish fonts, and an application always knows the
size of the cell via TIOCGSWINSZ which in kitty reports the number of
cells and the pixel size of the window content area. So it can decide
whether to use one or two cells or whatever number of cells gives it the
geometry it is looking for.

[kovidgoyal]

Oh and you can see that in action with

kitty +icat --print-window-size

[msokalski]

From my perspective, only having ability to define font with its own character cell aspect ratio would make this extension worth using. Below is my game example, in square character cells.

[ctrlcctrlv]

@msokalski Unfortunately, with the current implementation, symbol fonts, even if they are all the glyphs on a line, do not have their own metrics (cell shape).

Observe:

That's:

symbol_map U+0071 Fira Code Retina

So this must be dealt with one way or another. Needing to cram all glyphs into the cell shape of the master cell is an implementation hurdle that's difficult to cross.

[ctrlcctrlv]

Note that I don't think we should make this line based. That's way too difficult and will lead to bugs.

We just ought to offer a metrics command, so applications can change the cell shape of the entire window to meet their needs.

[kovidgoyal]

There is already a mechanism to change cell shapes, simply use a PUA
code point followed by as many spaces as you like. The glyph will be
rendered into that rectangle as an automatically created ligature. This
is how nerd fonts/powerline fonts for example render in kitty.

[ctrlcctrlv]

What if you want a cell shape of a width not expressable as width*n, where n‍∈ℕ*?

For example, suppose a user is using a cell ratio of 1.5:1, h:w. So, 2‍w makes our cell 1.5:2. 3‍w makes our cell 1.5:3. There's no way to get 1.5:1.5, 1.5 is not a possible n: we cannot therefore make a perfect square.

w is unlikely to be > h, but it can happen also.

A true metrics command would say, "a cell is now h by w", with the understanding that these values represent not pixels but rather something like CSS, a pixel times a certain device-specific multiplier to support HiDPI.

[kovidgoyal]

This is a terminal we are talking about, if you really want pixel
perfect graphics the way to do it is is to use the graphics protocol.

Trying to render perfect graphics by indirection through a font is
not appropriate.

[ctrlcctrlv]

If I can find a way to make it per-window without affecting performance...? Still no? I'm willing to spend some time refactoring if that's what it takes. 😋

As someone who works on fonts, I think metrics are a vital part of a font, even a monospaced one, and am not sure I want to contribute a way to set outlines without metrics...

I think our API should be like this:

Font slots

When loading a font there should be a flag, let's call it KITTY_MAINFONT, and if set, the font becomes the "main" font.

Really, instead of flags, it's better to think of them as slots. So you have a KITTY_MAINFONT (defined by font), n KITTY_SYMBOLFONT's (defined by running process symbol_map option), then n extra fonts, KITTY_APPFONT's. (for ease of implementation, should be constant, and guaranteed never to be decreased).

So, applications can use the API to put a font in any of the slots, and to add/remove symbol_maps.

This would allow an application wanting square fonts to load into KITTY_MAINFONT the font Press Start 2P (in my example above).

Then it can load into KITTY_APPFONT slot one a font with snake head, body, etc glyphs.

I am willing to contribute example code of course if we get this past the drawing board.

All C stuff (for now …)

• struct Font { type: FontType, char* data, bool data_is_path }
• enum FontType { KITTY_MAINFONT, KITTY_SYMBOLFONT, KITTY_APPFONT }
• bool add_font_to_slot(Font *font, int slot)
• bool del_font(int n)
• bool add_symbol_map(char* symbols, int slot) // error if slot not KITTY_SYMBOLFONT
• bool add_font_features(char* features, int slot)
• bool clear_font_features(int slot)
• bool clear_symbol_map(int slot)
• (int, int) get_preferred_metrics(int slot) // if font in slot were to be mainfont, metrics would become

[kovidgoyal]

I wont object to per window without affecting performance, but you will
have to basically re-write all of kitty's rendering and layout code
which all assumes that an OS window consists of identically sized cells.

I really dont think that level of effort is worth it, but if you want to
put it in, I wont stop you.

[ctrlcctrlv]

I just realized something: this will fix a problem I noticed before, that if I have two windows open in the same process, resizing one resizes the other, even though I hadn't intended to do that.

Presumably there are a lot of small bugs like that that will be solved as well

I'm not promising to succeed, only to try :-)

[kovidgoyal]

On Tue, Jan 07, 2020 at 06:17:10PM -0800, Fredrick Brennan wrote: I just realized something: this will fix a problem I noticed before, that if I have two windows open in the same process, resizing one resizes the other, even though I hadn't intended to do that.

Resizing the fonts in a window only changes fonts in other windows in the top level OS Window, not all top level windows in the same process. Unless you use the default font size shortcuts which do it for all windows. See change_font_size in https://sw.kovidgoyal.net/kitty/conf.html

Presumably there are a lot of small bugs like that that will be solved as well

There are no font size related bugs that I am aware of, beyond the limitation that you must use the same fonts size for all windows in a single top level OS Window.

[ctrlcctrlv]

Oh, are you calling tabs windows??

[kovidgoyal]

There are three things:

1. OS Window
   this contains
2. Tabs
   this contains
3. kitty windows

When you change the font size for any kitty window all kitty windows in
that OS Window will perforce change their font size, as will the tab bar.

However, windows in other OS Windows in the same process will not be
affected.

[ctrlcctrlv]

I understand the problem better now, thank you. I didn't understand that you meant, tabs. Please forgive me as I'm new to contributing to this project, and don't know all of Kitty's features, only the ones I use.

How about a different solution? :

• Instead of drawing the tabs by writing to the GPU cell grid, we draw tabs similar to how Konsole, et cetera does.
  
• Remember which fonts are in use by every tab
• On tab switched, change fonts

Tab names will be a single FreeType/HB buffer, written to the screen all at once. Is it important to you that I make it look exactly like it does now by default?

This would allow us to have different tabs with different font size as well, like Konsole.

[kovidgoyal]

kitty has no GUI library capable of drawing GUI controls and is never
going to link to one. So the tab bar has to be drawn using cells. And
the tab bar is not a problem anyway. It can always be drawn using the
default font/cell size. The problem is that multiple windows make up a
tab, and each of those windows are currently assumed to use the same
font/cell size. Changing that assumption is going to be a huge amount of
very fiddly work. If you have a different cell size per window, it
means:

1. you have to store font groups per window rather than per OS Window
   and change all code that uses them. This includes rendering, OpenGL,
   various other stuff.

2. OS Windows no longer have an overall cell size, a whole bunch of code
   relies on that

3. the code to layout windows inside a tab assumes a fixed cell size for
   all windows

4. Probably other things that I dont remember now.

I really dont think this amount of work/disruption is worth it just to
display pixel perfect images, for which there is already a perfectly
good solution.

[ctrlcctrlv]

(I work on FontForge, which has its own GUI toolkit, which I have also worked on. Actually, just incidentally, my most recent commit to FontForge dealt with drawing a tab bar, and tabs, and close button, and handling drawing of all that, and making it so tabs could be reordered et cetera. I wasn't recommending we start linking to Qt, the OpenGL API is more than enough.)

I don't really understand why that much has to change, though. We can just pretend that there is an overall cell size...basically, we change the font for all kitty windows in an OS window, but we keep a record of which fonts map to which tab, so when the user changes it, it still looks seemless, like each tab has its own fonts, when in reality we're switching them out every time

[kovidgoyal]

On Tue, Jan 07, 2020 at 07:55:12PM -0800, Fredrick Brennan wrote: (I work on FontForge, which has its own GUI toolkit, which I have also worked on. Actually, just incidentally, my most recent commit to FontForge dealt with drawing a tab bar, and tabs, and close button, and handling drawing of all that, and making it so tabs could be reordered et cetera. I wasn't recommending we start linking to Qt, the OpenGL API is more than enough.) I don't really understand why that much has to change, though. We can just pretend that there is an overall cell size...basically, we change the font for all kitty windows in an OS window, but we keep a record of which fonts map to which tab, so when the user changes it, it still looks seemless, like each tab has its own fonts, when in reality we're switching them out every time

Again, the tab bar is not the issue. The windows are.

[ctrlcctrlv]

Yeah, but the windows can't be seen until the user clicks the tab bar, so if we just switch up the fonts it doesn't matter—right?

[kovidgoyal]

On Tue, Jan 07, 2020 at 08:19:44PM -0800, Fredrick Brennan wrote: Yeah, but the windows can't be seen until the user clicks the tab bar, so if we just switch up the fonts it doesn't matter—right?

You can have more than one window visible per tab see the screenshot here: https://sw.kovidgoyal.net/kitty/ or run kitty -o enabled_layouts=tall and press ctrl+shift+enter a few times And see https://sw.kovidgoyal.net/kitty/#tabs-and-windows for details.

[ctrlcctrlv]

Oh, hmm, okay that's a big problem :-\

You're right it would be a lot of work.

Let's put a pin in this for now; my time can be better spent other ways, as I'm sure yours can be as well. :-)

[kovidgoyal]

Actually I think I am going to close this, as changing main font is not realistic IMO. I am willing to merge a protocol that allows applications to add entries to symbol_map but that's about the limit of what I think is doable in a reasonable way.

[ctrlcctrlv]

Please reopen this given the scourge of nerd fonts. https://gist.github.com/ctrlcctrlv/4ae789dabaea774548da5f00a6bf16e9

[kovidgoyal]

I agree Nerd fonts are pretty awful, I dont use them myself (only a few separators for powerline, drawn natively by kitty), but that doesnt really change my opinion on this issue. Having per window fonts is still too much work.

[ctrlcctrlv]

I don't think so. We already require fontconfig, and I know how to do it.

[ctrlcctrlv]

(We wouldn't have per-window fonts fullstop, we'd have per-window fallback fonts in my new conception of this idea.)

[kovidgoyal]

I'm confused then. Are you saying you dont care about metrics anymore. If so, as I said, when I closed this issue I am happy to consider a protocol that allows applications to specify additions to symbol_map.

[ctrlcctrlv]

Oh, right, I forgot that. Yes, no longer care about metrics. I'd rather get rid of nerd fonts.

[ctrlcctrlv]

(I mean, I still care, but not more than I care about deprecating nerd fonts lol.)

[ctrlcctrlv]

This will have to be a graphics extension, right? Send a font along with a symbol_map and in that Kitty window the replacement occurs?

[kovidgoyal]

It doesnt have anything to do with graphics. You have several
considerations:

1. You want to specify a per window mapping of unicode code points to
   glyphs from some font

2. This implies having a per window symbol_map, which is not too hard to
   do.

3. Since terminal programs can potentially be running on a different
   machine from the terminal itself, they would need to transmit the actual
   font data in the escape code alongwith a list of code-point to glyph
   maps. kitty would then need to load the font and use it for the
   specified code points.

There are several design considerations:

a) do we allow only one such mapping to be active per window. If not
then how many should be allowed and then there needs to be a way to
identify and remove mappings.

b) Do we bother with side channel transmission (filesystem/shared
memory) of font data as the graphics protocol does or just always
have the application transmit it via escape code?

c) do we have a way to activate/deactive the mappings? For example,
a full screen editor program will want to use mappings while active but
when suspended/running another program,
will want to de-active them so that they dont interfere.

d) Other things that dont come to mind right now :)

This will require careful thought/design so I suggest you opena
discussion for it and flesh out the semantics.

[ctrlcctrlv]

Will do. Thanks Kovid :)

And yes I only said it was graphics protocol related because the way I want to do it it will be in the code. No reason to write another way to transfer data from terminal application to users, can just use graphics proto