Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IME UI does not follow the cursor in Windows Terminal #459

Closed
driver1998 opened this issue May 7, 2019 · 6 comments · Fixed by #1919
Closed

IME UI does not follow the cursor in Windows Terminal #459

driver1998 opened this issue May 7, 2019 · 6 comments · Fixed by #1919
Assignees
Labels
Area-Interaction Interacting with the vintage console window (as opposed to driving via API or hooks) Area-Output Related to output processing (inserting text into buffer, retrieving buffer text, etc.) Area-Rendering Text rendering, emoji, complex glyph & font-fallback issues Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Needs-Tag-Fix Doesn't match tag requirements Product-Terminal The new Windows Terminal. Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release. Work-Item It's being tracked by an actual work item internally. (to be removed soon)
Milestone

Comments

@driver1998
Copy link

This bug-tracker is monitored by Windows Console development team and other technical types. We like detail!

If you have a feature request, please post to the UserVoice.

Important: When reporting BSODs or security issues, DO NOT attach memory dumps, logs, or traces to Github issues. Instead, send dumps/traces to [email protected], referencing this GitHub issue.

Please use this form and describe your issue, concisely but precisely, with as much detail as possible

  • Your Windows build number: (Type ver at a Windows Command Prompt)
    10.0.18890.1000

  • What you're doing and what's happening: (Copy & paste specific commands and their output, or include screen shots)
    Build & run the new Windows Terminal, and type something using IME (I am using Microsoft Pinyin IME)

  • What's wrong / what should be happening instead:
    The IME UI appears in the upper left corner of the screen, not following the current cursor.
    TIM截图20190507122302

@zadjii-msft
Copy link
Member

This is something we're working on currently!

@zadjii-msft zadjii-msft added Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Work-Item It's being tracked by an actual work item internally. (to be removed soon) labels May 7, 2019
@zadjii-msft zadjii-msft added this to the Windows Terminal v1.0 milestone May 7, 2019
@DHowett-MSFT
Copy link
Contributor

@philnach's so kindly volunteered to look at it. I may even assign it to him! 😄

@philnach
Copy link
Member

philnach commented May 9, 2019

@driver1998, thanks! @DHowett-MSFT, send it my way.

@ghost ghost added the Needs-Tag-Fix Doesn't match tag requirements label May 17, 2019
@miniksa miniksa added Area-Interaction Interacting with the vintage console window (as opposed to driving via API or hooks) Area-Output Related to output processing (inserting text into buffer, retrieving buffer text, etc.) Area-Rendering Text rendering, emoji, complex glyph & font-fallback issues Product-Terminal The new Windows Terminal. and removed Mass-Chaos labels May 17, 2019
@ghost ghost removed the Needs-Tag-Fix Doesn't match tag requirements label May 18, 2019
@davidwin
Copy link
Contributor

So this is the same issue that makes the IME UI with a green arrow appear when opening the emoji panel? In #1557 I wasn't expecting that UI at all.

@DHowett-MSFT
Copy link
Contributor

It sure is! The toolbar with the green arrow is actually a fallback UI for when the application doesn't explicitly say "Here's an input handler!"

@ghost ghost added the In-PR This issue has a related PR label Jul 10, 2019
DHowett-MSFT pushed a commit that referenced this issue Nov 22, 2019
TerminalControl doesn't use any of the built in text input and edit
controls provided by XAML for text input, which means TermianlControl
needs to communicate with the Text Services Framework (TSF) in order to
provide Input Method Editor (IME) support.  Just like the rest of
Terminal we get to take advantage of newer APIs (Windows.UI.Text.Core)
namespace to provide support vs. the old TSF 1.0.

Windows.UI.Text.Core handles communication between a text edit control
and the text services primarily through a CoreTextEditContext object.

This change introduces a new UserControl TSFInputControl which is a
custom EditControl similar to the CustomEditControl sample[1].

TSFInputControl is similar (overlay with IME text) to how old console
(conimeinfo) handled IME. 

# Details
TSFInputControl is a Windows.UI.Xaml.Controls.UserControl

TSFInputControl contains a Canvas control for absolution positioning a
TextBlock control within its containing control (TerminalControl).

The TextBlock control is used for displaying candidate text from the
IME.  When the user makes a choice in the IME the TextBlock is cleared
and the text is written to the Terminal buffer like normal text.

TSFInputControl creates an instance of the CoreTextEditContext and
attaches appropriate event handlers to CoreTextEditContext in order to
interact with the IME.

A good write-up on how to interact with CoreTextEditContext can be found
here[2].

## Text Updates
Text updates from the IME come in on the TextUpdating event handler,
text updates are stored in an internal buffer (_inputBuffer).

## Completed Text
Once a user selects a text in the IME, the CompositionCompleted handler
is invoked.  The input buffer (_inputBuffer) is written to the Terminal
buffer, _inputBuffer is cleared and Canvas and TextBlock controls are
hidden until the user starts a composition session again.

## Positioning
Telling the IME where to properly position itself was the hardest part
of this change.  The IME expects to know it's location in screen
coordinates as supposed to client coordinates.  This is pretty easy if
you are a pure UWP, but since we are hosted inside a XAMLIsland the
client to screen coordinate translation is a little harder.  

### Calculating Screen Coordinates
1. Obtaining the Window position in Screen coordinates.
2. Determining the Client coordinate of the cursor.
3. Converting the Client coordinate of the cursor to Screen coordinates.
4. Offsetting the X and Y coordinate of the cursor by the position of
   the TerminalControl within the window (tabs if present, margins, etc..).
5. Applying any scale factor of the display.

Once we have the right position in screen coordinates, this is supplied
in the LayoutBounds of the CoreTextLayoutRequestedEventArgs which lets
the IME know where to position itself on the Screen.

## Font Information/Cursor/Writing to Terminal
3 events were added to the TSFInputControl to create a loosely-coupled
implementation between the TerminalControl and the TSFInputControl.
These events are used for obtaining Font information from the
TerminalControl, getting the Cursor position and writing to the terminal
buffer.

## Known Issues

- Width of TextBlock is hardcoded to 200 pixels and most likely should
  adjust to the available width of the current input line on the console
  (#3640)
- Entering text in the middle of an existing set of text has TextBlock
  render under existing text. Current Console behavior here isn't good
  experience either (writes over text)
- Text input at edges of window is clipped versus wrapping around to
  next line.  This isn't any worse than the original command line, but
  Terminal should be better (#3657)

## Future Considerations
Ideally, we'd be able to interact with the console buffer directly and
replace characters as the user types. 

## Validation
General steps to try functionality
- Open Console
- Switch to Simplified Chinese (Shortcut: Windows+Spacebar)
- Switch to Chinese mode on language bar

Scenarios validated:
- As user types unformatted candidates appear on command line and IME
  renders in correct position under unformatted characters.
- User can dismiss IME and text doesn't appear on command line 
- Switch back to English mode, functions like normal
- New tab has proper behavior
- Switching between tabs has proper behavior
- Switching away from Terminal Window with IME present causes IME to
  disappear

[1]: https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CustomEditControl
[2]: https://docs.microsoft.com/en-us/windows/uwp/design/input/custom-text-input

Closes #459
Closes #2213
Closes #3641
@ghost ghost added Needs-Tag-Fix Doesn't match tag requirements Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release. and removed In-PR This issue has a related PR labels Nov 22, 2019
@ghost
Copy link

ghost commented Nov 26, 2019

🎉This issue was addressed in #1919, which has now been successfully released as Windows Terminal Preview v0.7.3291.0.:tada:

Handy links:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Interaction Interacting with the vintage console window (as opposed to driving via API or hooks) Area-Output Related to output processing (inserting text into buffer, retrieving buffer text, etc.) Area-Rendering Text rendering, emoji, complex glyph & font-fallback issues Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. Needs-Tag-Fix Doesn't match tag requirements Product-Terminal The new Windows Terminal. Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release. Work-Item It's being tracked by an actual work item internally. (to be removed soon)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants