conpty: fall back to conhost if OpenConsole is missing

[DHowett]

This pull request is in support of WTU.

I initially added support for a new flag, PSEUDOCONSOLE_UNDOCKED_PREFER_INBOX_CONHOST, which I liked because it was more explicit. You can see that approach in commit cae3289.

Consider this an open discussion. Should we automatically fall back? Or should we allow a developer to opt for fallback?

Automatic fallback

Pros

• It's easier on the consumer
• We can eventually expand it to support $ARCH/openconsole.exe

Cons

• Packaging the project wrong will result in a working-but-somewhat-broken experience (old conhost)
• Implicit behavior may be bad

Manual selection

Pros

• Consumer gets explicit control

Cons

• Consumer must explicitly opt in. In the case of WTU, that means pushing a configuration down into TerminalConnection all the way from TerminalApp to say "use the inbox one, dummy"

[miniksa]

Oh. Well. When you originally told me about that flag in Teams, I thought you meant it was an #ifdef flag, not a bit flag to be passed the whole way down the stack.

I'd much rather not pipe that the entire way through the stack if we can help it...

However, I'm getting more scared of the implicit thing using the "somewhat broken" conhost and it being hard to tell which one it is connected to when a customer reports an error. Also, I'm not looking forward to a potential cycle when default app is put in, but that's likely to happen either way, I think.

[DHowett]

I think the cycle with defaultapp will be broken by the fact that the second OpenConsole will not spawn a new conpty using this API, but instead stand up pipes and put them into a terminal.

This change is only to support the case where we do not package OpenConsole with WTU right now.

One way to ameliorate the concern about picking up a 'broken' conhost is to add a step to the msixbundle check script that validates the presence of OpenConsole.exe. That will stop our WT builds from failing weirdly, but it may not help the eventual conpty nuget/framework package.

[miniksa]

I think the cycle with defaultapp will be broken by the fact that the second OpenConsole will not spawn a new conpty using this API, but instead stand up pipes and put them into a terminal.

This change is only to support the case where we do not package OpenConsole with WTU right now.

One way to ameliorate the concern about picking up a 'broken' conhost is to add a step to the msixbundle check script that validates the presence of OpenConsole.exe. That will stop our WT builds from failing weirdly, but it may not help the eventual conpty nuget/framework package.

I like the idea of a check in the bundle. And when we get to a nuget/framework package, we're gonna have this problem no matter what and need to have some sort of logging or exposure through the UI of what versions it is to diagnose issues. So I'm OK with that can being kicked down the road.

[DHowett]

Added validation in 90fb290

[miniksa]

Good enough. If we have reason to change our minds later, so be it. I'm happy with this.

[zadjii-msft]

This is simple and elegant. I'm happy with this design - if people want opt in to using openconsole as the conpty host, then they've got to add an openconsole.exe to their package. That makes sense to me. The added build rule is also a good example for other developers who might want to do the same thing.