Incorrect emoji widths with conhost and openconsole

[CHerSun]

Windows Terminal version

1.20.24050601

Windows build number

10.0.19045.0

Other Software

Far manager v3 - easy to test by creating files with needed names and opening Far in different terminals, but the behavior is not specific to Far.

Steps to reproduce

print string / create file with emoji in it:

test1 ♻.txt
test2 🧩.txt
test3 ⭐.txt

In Windows Terminal

Windows Terminal + pwsh:

Plz, disregard the ^C. Note:

• ♻ size is like half-width or so - .txt is printed inside it
• 🧩 size looks to be correct
• ⭐ size looks to be correct, but note that the cursor is rendered between x and t, when it's actually at the end (-1 symbol cursor position rendering). Also note that black&white version is rendered, not colored one, like for other emojis.

with Far:

Everything looks good, but ♻, which is half-width or so and b&w image for star.

In conhost

Far using default Windows 10 console:

It's not very clear from the screenshot, but it's easy to repeat my test using latest stable Far manager v3 and creating files with the names above.

Notes for screenshot:

• ♻ looks to be rendered correctly (considering no emoji support)
• 🧩 looks to be rendered correctly, but the cursor shown over test2 file is 1 symbol shorter
• ⭐ vertical border is moved 1 symbol to the left

OpenConsole

Far in openconsole directly:

Same as with WT - ♻ symbol looks to be half-width or so.

NOTE: I've tested mainly with Consolas font, but I've tried a few other, like Fira Code, IBM Plex, Cascadia Mono - behavior seems to be consistent between fonts.
NOTE: to fast-insert emojis - Win+; and start typing recycle for ♻, puzzle for 🧩, star for ⭐.

Expected Behavior

Consistent spacing and rendering of emojis.

Actual Behavior

Inconsistent spacing, cursor positioning, etc.

[lhecker]

I apologize for the late response. The behavior is unfortunately intentional, because for most glyphs the widths are determined by the Unicode standard and not by the font. The standard is this one: https://www.unicode.org/reports/tr11/
If the font contains glyphs that are wider than they should be, they'll show up like this (overlapped).

Your glyphs are:

• ♻ https://codepoints.net/U+267b
• 🧩 https://codepoints.net/U+1f9e9
• ⭐ https://codepoints.net/U+2b50

Check out the "East Asian Width" for each of these. The first one is "neutral", which basically means "1 column", and the other two are "wide" (= 2 columns). We use the official text renderers from Windows (GDI and Direct2D) in their default configuration. The reason they behave differently is because GDI has Segoe UI Symbol (= black and white glyphs) higher up in the font fallback priority list than Segoe UI Emoji (= colored glyphs), and for Direct2D it's the other way around. Unfortunately neither font adheres to UAX #11.

[CHerSun]

@lhecker thank you for clarification. I understand there are technical issues. But from end-user point of view - that's a bug for sure.

Check out the "East Asian Width" for each of these. The first one is "neutral", which basically means "1 column", and the other two are "wide" (= 2 columns).

That might be a good reason for 2 different behaviors, but does not explain 3. All 3 symbols behave differently (1 overlaps, 1 renders normally, 1 shifts the cursor).

If that cannot be fixed in openconsole directly due to reliance on some other product (renderer in this case) - I'd really appreciate if the team takes ownership of the problem and opens required tickets, as dev team is the only entity who really knows what's happening underneath. User like me cannot chase that, unfortunately.

[lhecker]

There's more to this - let me respond to your issue description one by one:

Windows Terminal + pwsh:
♻ size is like half-width or so - .txt is printed inside it

The half-width is intentional due to UAX #11. The reason why ".txt" is printed inside it is: "This is the same treatment given by iTerm2 and Terminal.app on macOS (one of which I believe was the first terminal emulator to support emoji(?).)" (See #16852)
In other words, terminal applications expect us to render emojis with an overlap like that, because the first terminals that supported emojis did so.

However, if you expected the ♻ glyph (recycling symbol) to show up as an emoji then you should print ♻️ instead (recycling emoji). The difference between the two is that ♻ is U+267B and is called an "unqualified emoji". Unqualified means that it's unspecified how the glyph gets rendered. Firefox for instance draws them black/white inside monospace text and colored inside proportional text. ♻️ on the other hand is U+267B U+FE0F, which is a "fully-qualified emoji" thanks to the U+FE0F variation selector. This means it always gets rendered as something that's 2 cells wide and is colored if the renderer supports color (conhost and GDI don't).

If you want narrow emojis to show up as wide emojis you should always use U+FE0F. You can find a list of all emojis with their U+FE0F counterparts here: https://unicode.org/Public/emoji/latest/emoji-test.txt
Support for U+FE0F was still somewhat buggy until recently. That was fixed by #16916.

⭐ size looks to be correct, but note that the cursor is rendered between x and t, when it's actually at the end (-1 symbol cursor position rendering).

Most emojis don't work inside the pwsh prompt line. I'm not entirely sure why this one in particular is broken, but these problems are tracked here: PowerShell/PSReadLine#1329
I've been hearing whispers that PSReadLine will add support for grapheme clusters in the near future too, which would solve all of these issues all at once.

Also note that black&white version is rendered, not colored one, like for other emojis.

It seems that this is a bug in DirectWrite on Windows 10. We could try and fix that!
It also occurs with GDI in conhost, but I'm not entirely sure whether we can fix that easily.

Far using default Windows 10 console: [...]

The Windows 10 console predates significant parts of our emoji support and most of the bug fixes. Our request to backport them has been denied unfortunately, so we can't do anything about that. We do ship Windows Terminal on Windows 10 however which works correctly (as per iTerm 2's and Terminal.app's emoji behavior).

Far in openconsole directly: [...]

FYI OpenConsole is basically the latest conhost version. That's why it behaves like Windows Terminal. conhost on the latest Windows 11 version will also behave like that. But it still uses the GDI renderer so that's why it's black & white.

---

The only thing that's not intentional is how ⭐ looks like a tiny star in Windows Terminal on Windows 10 and in conhost in general.

Regarding Windows Terminal, I suspect that the font fallback table in the Windows 10 version of DirectWrite simply didn't include that codepoint in the list. Fixing that is non-trivial because the font fallback list is gigantic. We could patch that one codepoint inside our font-mapping code, but I'd not particularly like adding such bodgy compatibility code. I'd be open to do that though if we decide that we do want to see ⭐ fixed.

Regarding conhost, custom font fallback with Uniscribe works like this: https://learn.microsoft.com/en-us/windows/win32/intl/using-font-fallback#assign-a-fallback-font
However, we'd have to be extra careful because applications may actually rely on the fallback to Segoe UI Symbol and so it would either need a setting, or we'd have to make it so that we detect fully-qualified emojis automatically and only use Segoe UI Emoji for those that are. We currently don't have a Unicode database that includes this information, because we're trying to reduce the binary size. We could rely on the ICU database for this though, but even then, it'd be a very difficult fix.

[lhecker]

We've discussed this internally earlier this week and we've decided that we don't want to fix the ⭐ issue at the moment, unless more people are interested in seeing this fixed. This is mostly because fixing this is likely to be non-trivial right now.

The other issues are - for better or worse - unfortunately working as intended, as explained above. As such I'll be closing this issue for now. If you feel like anything I said is incorrect, please let me know!