createTerminal does not set the code point of the windows process to utf8, even though it expects the terminal to write utf8

[DavyLandman]

Does this issue occur when all extensions are disabled?: No (but I did disable all the other ones)

• VS Code Version: 1.83.1
• OS Version: Windows 11

Steps to Reproduce:

1. make a batch file (saved as utf8). (please note that I just made it a batch file for easier reproduction, in reality it's a full terminal).

echo off

echo "Code page before: "
chcp

echo "Failing unicode: 🚀"

echo "Now chaching codepage to UTF8"
chcp 65001
echo "Working unicode: 🚀"
timeout 25

2. add a command that executes the following code in an extension:

vscode.window.createTerminal({
    name: "Test",
    shellPath: "c:\\users\\davy\\test.bat"
});

3. see if fail to properly print utf8 stuff:
   

As I understand it:

1. VS Code & xterm on Windows seem to fall back to expecting utf8 (it even sets up LANG: en_US.UTF-8)
2. windows processes by default are run in the codepage of the user configuration, which can be CP437 or CP1252.
3. a process that is writing to it's output stream should respect the code page it's started with.
4. therefore it starts outputting bytes in it's code page.
5. once we manually switch to the expected codepage by vs code, suddenly utf8 bytes are passed along as utf8 bytes, and the terminal is happy to display emojis.

The actual bigger case is a java program that uses jline to make a REPL. As java picks up the code page from the windows kernel call, it sets up System.out with CP437 encoding. Later jline picks up the en_US.UTF-8 but also picks up the CP437 encoding. Now stuff gets messy.

My proposal would be at the very least to have an option for createTerminal to change the code page (instead of only setting en_US.UTF-8). And I'm not sure, but doing this by default might also change some of the encoding issues I see around emoji's in windows terminals.

[edit]
I've been doing some further reading, these discussions might be related:

• Terminal should force pseudoconsole host into UTF-8 codepage by default terminal#1802
• Setting to choose codepage of new terminal session for each profile.  terminal#15678

[Tyriar]

This issue seems to originate from an upstream component and is not something VSCode can fix by making a change to our code. As such, I will go ahead and close it. Please try to reproduce in Windows Terminal and report to microsoft/terminal with repro steps.

If you cannot reproduce in Windows Terminal, chances are it's fixed in a later version of Windows. You can read more about how we consume conpty here. Additionally, soon we plan on shipping conpty v2 with VS Code which should bring our behavior in line with Windows Terminal. Thanks for understanding.

/label upstream terminal-conpty windows

[DavyLandman]

Hi @Tyriar we actually ran into this for a custom terminal we created in VS Code (so just starting process and hooking it directly to the input and output streams)

I only used basic cmd in this example as a small test case. I linked terminal issues to show they had a similar problem. It seems there is a codepage argument for conhost, that should align VS Code expectations to how it's setting it up (it seems the fix would be to start conhost with conhost --codepage 65001 argument?)