distribute conhost fixes with Windows 10

[mintty]

Windows Terminal version

conhost of Windows 10

Windows build number

Windows 10

Other Software

No response

Steps to reproduce

Run a terminal application (not Windows Terminal), let's say Git bash from Git-for-Windows.
Run a Windows shell (like wsl.exe, cmd or powershell).
Output (cat or type) a text file with ANSI escape sequences for bold, reverse, and both.

Expected Behavior

Proper rendering of character attributes, also in combination with colour attributes, true colour etc.

Actual Behavior

Attribute rendering is broken because escape sequences are mangled.
Bold is forced to render white.

Related issues

#12806
#2837
#1817

Note

This bug was fixed in conhost for Windows 11. A fixed conhost.exe should also be updated for Windows 10.

[DHowett]

Hey there! We're intending on distributing updates to ConPTY through a NuGet package instead. The idea is that this will allow terminal emulator authors (and other folks who need it, of course) to lock to specific versions and fully vet compatibility with them rather than having to have a huge support matrix based on OS version. It also gets us out of the business of backporting which, at this point, is prohibitively expensive for Windows 10.

Is that something you'd be interested in, and able to take a dependency on? If not, I'd love to learn why not.

[mintty]

Is that the same as winget?
If you do that, I can include advice to users how to update themselves, but I cannot take any "dependency" on something outside the cygwin domain.

Actually it's hard to understand that including a bugfix update in one of the Windows updates shipped anyway raises extreme costs. A suitable drop-in replacement for conhost would apparently be OpenConsole from the Windows Terminal project.

[avih]

A suitable drop-in replacement for conhost would apparently be OpenConsole from the Windows Terminal project.

If you mean that you may include it with the mintty distribution (either in the same package or additional hosted download), then I don't think it's enough, because AFAIK mintty would need to load conpty.dll which matches this openconsole.exe, but conpty.dll is not shipped with the windows terminal releases (I think it's statically linked or some such - but still needs openconsole.exe AFAIK).

However, it is available at the windows terminal CI builds.

[mintty]

I tested OpenConsole as a drop-in replacement for System32/conhost.exe without touching console.dll (there is no conpty.dll) on Windows 10 and that fixes the attribute bugs for me.

[avih]

Right. I thought this was in the context of the mintty application, which would need to load conpty.dll in order to use openconsole.exe and its updated features, AFAIK.

openconsole.exe is indeed a stand-alone replacement for conhost.exe, and you can run it from anywhere to open a new console window with the updated features (and it doesn't need conpty.dll).

However, AFAIK it's not generally intended to be executed by the user, but does happen to support it and even supports argument as command to execute inside the window.

But unlike cmd.exe, openconsole (and probably conhost too) also support CLI args to "link" the console application to the kernel etc. It's basically an internal windows thing, which also happens to work stand-alone.

[DHowett]

Is that the same as winget?

No. NuGet is a package manager for C/C++/C# codebases.

The plan is that we would distribute conpty.dll and OpenConsole.exe for a number of architectures.
Anybody with an existing dependency on kernel32!CreatePseudoConsole could move over to conpty.dll!CreatePseudoConsole, and they can pin whichever version works best and has been most tested with their application.

In effect, it's taking the OS component that was ConPTY and making it a redistributable.

If you do that, I can include advice to users how to update themselves, but I cannot take any "dependency" on something outside the cygwin domain.

I'm curious about this. kernel32.dll lies outside Cygwin, but it's absolutely being loaded. Can you elaborate a bit on what taking a dependency looks like?

it's hard to understand that including a bugfix update in one of the Windows updates shipped anyway raises extreme costs

As a policy, Windows does not typically include features in servicing updates¹. The branches from which Windows 10 updates are built are intended only for bug fixes with a "strong" "business justification". Bug fixes are also "contained" for a while--that is, shipped with both code paths--so that if we break something catastrophically, we can roll it back.

Now, it would be all fine (and I could almost make a case for doing it!) if conhost were checked into Windows as a binary built out of this repository. Check in OpenConsole.exe and call it a day! Alas, it's ingested as a source drop.

Since we shipped Windows 10 20H1 (the last baseline version that's currently supported) the delta for just conhost is sitting at (as of today, cb48bab):

$ git diff --stat 74e68447d375e43a5f7bd45deddd6c5bf4bd1e5e origin/main -- ":(top,exclude)src/cascadia/" ":(top,exclude)src/winconpty/" ":(top,exclude)src/tools/MonarchPeasantSample/" ":(top,exclude)src/tools/MonarchPeasantPackage/" ":(top,exclude)src/tools/ansi-color/" ":(top,exclude)src/tools/ColorTool/" "src"

1000 files changed, 115079 insertions(+), 68605 deletions(-)

That includes features and bug fixes, and bug fixes to features.

It isn't in anybody's best interest (even yours!) for somebody to pick out just the bug fixes from that mess, backport them, contain them safely so they can be disabled, and then vet each one of them individually turned on and turned off. It's just not feasible at this point with the staffing the team has, the size of the delta, and the concomitant risk to Windows 10.

So: shipping a redistributable gets us the same guarantees as a binary drop of conhost without the risk to the OS or the individual bugfix backporting. It also means that we can release updates (and more bug fixes!) within a month, and that all of your consumers are using the same version of the console subsystem. You no longer have to field bug reports that differ based on OS version, nor reply that "all [they] can do to fix this bug is to upgrade Windows."

--

If you mean that you may include it with the mintty distribution (either in the same package or additional hosted download), then I don't think it's enough, because AFAIK mintty would need to load conpty.dll which matches this openconsole.exe, but conpty.dll is not shipped [...]

The central thesis here is that we would, in fact, ship conpty.dll 🙂

Footnotes

1. it is complicated now with Windows 11. The central thrust of this message still holds true. ↩

[j4james]

@DHowett I don't think @mintty's problem is necessarily with conpty - I believe their complaint is specifically about the version of conhost that is found in the system32 folder. Because even if you have the latest version of conpty (as we do in Windows Terminal), that doesn't help when you try running something like cmd.exe from within a wsl shell. At that point you're going to get redirected to the inbox conhost in system32, and on Windows 10 that's fairly buggy.

And I think for mintty users that may be a common scenario. They'll start in something like wsl (using the wsltty version of mintty, so not actually involving conpty at all), but from there they might need to run cmd.exe or some other win32 console app, and that's when things will go wrong.

At least that's my understanding of the issue. If you read the mintty FAQ Tips page, you'll note that their solution to the problem is literally telling users to replace the system32 conhost with a more up to date copy of OpenConsole. Ideally that's not something you want users to be doing.

But I understand why Windows 10 isn't likely to get an upgrade. I just wanted to explain why a NuGet distribution of conpty probably won't help in this case.

[avih]

Hey there! We're intending on distributing updates to ConPTY through a NuGet package instead.

Is there some timeline plan about when it's expected to happen?

Until that happens, or if the user doesn't install it via nuget, I'm guessing the unofficial, unsupported, and possibly not-license-following method is to get a copy of conpty.dll and openconsole.exe from the windows terminal project (build it manually or from a CI build), place them where the terminal emulator expects, and then the emulator would need to manually load this conpty.dll and start using it?

I've not used NuGet, so pardon the n00b-ness.

I presume the user would need to use a/the nuget client to globally install "conpty" package or some such, or optionally some app may do it for them? (e.g. a terminal emulator app would suggest to install the latest conpty)

After that, there would exist both the native conpty interface (e.g. a buggy one on windows 10), and the updated packaged conpty in parallel?

And after that, how would a terminal emulator author choose which conpty to use? (rough high level outline would suffice)

I presume such emulator author shouldn't care about the backend used by the chosen conpty, be it conhost (main or packaged) or openconsole or whatever else the backend name is in the package?

If the terminal app doesn't try to choose a specific conpty and just uses the API "normally", can a user somehow make that terminal app use the updated packaged conpty instead of the native one? (without manually replacing files in c:/windows/)

Once a terminal app uses one of the existing conpty interfaces, I presume that typically any console-app child process of that terminal would keep using the same conpty package which the terminal chose?

Would it typically also use the same conpty when a new console window is created? (e.g. if the current shell at the terminal is cmd.exe, then after start cmd)

[DHowett]

@j4james that is an excellent observation, thank you.

@avih the idea is that the developer uses NuGet, and bundles a specific version of the libraries they depend on. ConPTY becomes just one of those libraries. The user would not change the terminal’s backend just like they would not change which encryption library their browser uses.

[mintty]

So nuget is "yet another package manager"... Why? Can't you make a package for winget as that's available by default.

I do not use or want to use CreatePseudoConsole or "preload" a preferred console library (which may not be possible anyway as discussed by j4james). The scenario is different:
The user interactively invokes a Windows console program, like cmd or powershell or wsl. This pulls in conhost.exe under-the-hood, out of control of the terminal. Then whether you type file from cmd or cat file from wsl, if the file contains e.g. bold ANSI attributes their display is broken. Maybe it's possible to replace conhost.exe by OpenConsole.exe per process, using the obscure Windows shadow file system (no idea how that would be done). Otherwise the only solution is static replacement in the system, another questionable workaround from product management point of view. And yes, I am familiar and understand industrial-scale release management problems, but a conhost update is really not a new feature, just a fix.

As a side-note, speaking of wsl, it is hardly understandable why it enforces hook-in of the conhost layer. It would not be needed at all between a pty-based terminal (also e.g. a remote one) and the WSL side which is not interested in conhost stuff either. (If you run e.g. cmd within WSL, you can still pull in conhost then but that's not really the major use case.)
This comes to the request for a conhost-free mode of running the wsl.exe access program, a long-missing and urgently appropriate feature discussed elsewhere as "passthrough mode".