Accessibility: Signaling Model #2447

carlos-zamora · 2019-08-15T22:08:21Z

Description of the new feature/enhancement

UIA has signal events (link). ConHost supported them. WindowUiaProvider was an integral part to that.

Windows Terminal currently doesn't support them. We have some architecture in place to do so though.

Proposed technical implementation details (optional)

WindowUiaProvider already has the groundwork for that. It might be that we don't even need a WindowUiaProvider and just have XAML do some of it.

For reference, the signaling model for conhost was as follows:

<ConHost Event (i.e.: new text)>
--> Signal WindowUiaProvider
--> Signal ScreenInfoUiaProvider
--> Raise UiaAutomationEvent to Client

The client then responds to these events with more actions (i.e.: recreate object, etc...)

EDIT: The functionality for signaling includes...

Selection Events (selection changed) PR
Text Buffer Events (new output)
Scroll Events

I may possible need...

Resize events
Cursor Events

codeofdusk · 2019-09-30T17:49:51Z

Blocking nvaccess/nvda#10305.

## Summary of the Pull Request We used to return multiple text ranges to represent one selection. We only support one selection at a time, so we should only return one range. Additionally, I moved all TriggerSelection() calls to the renderer from Terminal to TermControl for consistency. This ensures we only call it _once_ when we make a change to our selection state. ## References #2447 - helps polish Signaling for Selection #4465 - This is more apparent as the problem holding back Signaling for Selection ## PR Checklist * [x] Closes #4452 Tested using Accessibility Insights.

- When performing chunk selection, the expansion now occurs at the time of the selection, not the rendering of the selection - `GetSelectionRects()` was moved to the `TextBuffer` and is now shared between ConHost and Windows Terminal - Some of the selection variables were renamed for clarity - Selection COORDs are now in the Text Buffer coordinate space - Fixes an issue with Shift+Click after performing a Multi-Click Selection ## References This also contributes to... - #4509: UIA Box Selection - #2447: UIA Signaling for Selection - #1354: UIA support for Wide Glyphs Now that the expansion occurs at before render-time, the selection anchors are an accurate representation of what is selected. We just need to move `GetText` to the `TextBuffer`. Then we can have those three issues just rely on code from the text buffer. This also means ConHost gets some of this stuff for free 😀 ### TextBuffer - `GetTextRects` is the abstracted form of `GetSelectionRects` - `_ExpandTextRow` is still needed to handle wide glyphs properly ### Terminal - Rename... - `_boxSelection` --> `_blockSelection` for consistency with ConHost - `_selectionAnchor` --> `_selectionStart` for consistency with UIA - `_endSelectionPosition` --> `_selectionEnd` for consistency with UIA - Selection anchors are in Text Buffer coordinates now - Really rely on `SetSelectionEnd` to accomplish appropriate chunk selection and shift+click actions ## Validation Steps Performed - Shift+Click - Multi-Click --> Shift+Click - Chunk Selection at... - top of buffer - bottom of buffer - random region in scrollback Closes #4465 Closes #4547

carlos-zamora · 2020-03-06T17:25:56Z

@codeofdusk @michaelDCurran @LeonarddeR
CC @DHowett-MSFT

I've attached a release x64 build of dev/cazamor/acc/sig/output. I'll be working off of that branch to get automation events working for output. The changes are pretty naive but simple: whenever we need to render new content to the screen, fire a TextChanged automation event.

For something as simple as echo hello, I can see that some degenerate ranges are created at the end of the written command, then at the end of the prompt after the command has been executed. However, it seems like a range connecting both points is never actually created and read.

Are you expecting a different Automation Event? Do I need to configure my TermControlAutomationPeer any further, maybe? Let me know what's going on from your end :)

NOTE: you'll have to run WindowsTerminal.exe here, NOT OpenConsole.exe. ConHost has not been switched over to the new signaling model yet, whereas Windows Terminal has.

WT_signal_output.zip

codeofdusk · 2020-03-06T17:38:48Z

@codeofdusk @michaelDCurran @LeonarddeR
CC @DHowett-MSFT

I've attached a release x64 build of dev/cazamor/acc/sig/output. I'll be working off of that branch to get automation events working for output. The changes are pretty naive but simple: whenever we need to render new content to the screen, fire a TextChanged automation event.

For something as simple as echo hello, I can see that some degenerate ranges are created at the end of the written command, then at the end of the prompt after the command has been executed. However, it seems like a range connecting both points is never actually created and read.

Are you expecting a different Automation Event? Do I need to configure my TermControlAutomationPeer any further, maybe? Let me know what's going on from your end :)

NOTE: you'll have to run WindowsTerminal.exe here, NOT OpenConsole.exe. ConHost has not been switched over to the new signaling model yet, whereas Windows Terminal has.

WT_signal_output.zip

Thanks @carlos-zamora This looks sufficient for implementing terminal support in NVDA. Things are mostly working (see nvaccess/nvda#10784), except we're getting doubled characters when typing (but I think I know where the issue is, and it's on our side). Are there plans to have this merged soon?

carlos-zamora · 2020-03-06T17:40:19Z

@codeofdusk @michaelDCurran @LeonarddeR
CC @DHowett-MSFT
I've attached a release x64 build of dev/cazamor/acc/sig/output. I'll be working off of that branch to get automation events working for output. The changes are pretty naive but simple: whenever we need to render new content to the screen, fire a TextChanged automation event.
For something as simple as echo hello, I can see that some degenerate ranges are created at the end of the written command, then at the end of the prompt after the command has been executed. However, it seems like a range connecting both points is never actually created and read.
Are you expecting a different Automation Event? Do I need to configure my TermControlAutomationPeer any further, maybe? Let me know what's going on from your end :)
NOTE: you'll have to run WindowsTerminal.exe here, NOT OpenConsole.exe. ConHost has not been switched over to the new signaling model yet, whereas Windows Terminal has.
WT_signal_output.zip

Thanks @carlos-zamora This looks sufficient for implementing terminal support in NVDA. Things are mostly working (see nvaccess/nvda#10784), except we're getting doubled characters when typing (but I think I know where the issue is, and it's on our side). Are there plans to have this merged soon?

If I can get it working for Narrator too, that'd be ideal. But, I guess there's no harm in merging what I have then iterating on master. I'll submit a PR today then. Thanks!

codeofdusk · 2020-03-06T17:42:48Z

@codeofdusk @michaelDCurran @LeonarddeR
CC @DHowett-MSFT
I've attached a release x64 build of dev/cazamor/acc/sig/output. I'll be working off of that branch to get automation events working for output. The changes are pretty naive but simple: whenever we need to render new content to the screen, fire a TextChanged automation event.
For something as simple as echo hello, I can see that some degenerate ranges are created at the end of the written command, then at the end of the prompt after the command has been executed. However, it seems like a range connecting both points is never actually created and read.
Are you expecting a different Automation Event? Do I need to configure my TermControlAutomationPeer any further, maybe? Let me know what's going on from your end :)
NOTE: you'll have to run WindowsTerminal.exe here, NOT OpenConsole.exe. ConHost has not been switched over to the new signaling model yet, whereas Windows Terminal has.
WT_signal_output.zip

Thanks @carlos-zamora This looks sufficient for implementing terminal support in NVDA. Things are mostly working (see nvaccess/nvda#10784), except we're getting doubled characters when typing (but I think I know where the issue is, and it's on our side). Are there plans to have this merged soon?

If I can get it working for Narrator too, that'd be ideal. But, I guess there's no harm in merging what I have then iterating on master. I'll submit a PR today then. Thanks!

I've tested with Narrator and observed, as you did, that we aren't getting new text announcements.

In NVDA, we have an NVDAObjects.behaviors.LiveText class that waits for text change events from various APIs (such as UIA), diffs the current text against the last-reported text, and speaks the changes. Does Narrator have a similar paraddigm for consoles?

carlos-zamora · 2020-03-06T17:48:52Z

@codeofdusk @michaelDCurran @LeonarddeR
CC @DHowett-MSFT
I've attached a release x64 build of dev/cazamor/acc/sig/output. I'll be working off of that branch to get automation events working for output. The changes are pretty naive but simple: whenever we need to render new content to the screen, fire a TextChanged automation event.
For something as simple as echo hello, I can see that some degenerate ranges are created at the end of the written command, then at the end of the prompt after the command has been executed. However, it seems like a range connecting both points is never actually created and read.
Are you expecting a different Automation Event? Do I need to configure my TermControlAutomationPeer any further, maybe? Let me know what's going on from your end :)
NOTE: you'll have to run WindowsTerminal.exe here, NOT OpenConsole.exe. ConHost has not been switched over to the new signaling model yet, whereas Windows Terminal has.
WT_signal_output.zip

Thanks @carlos-zamora This looks sufficient for implementing terminal support in NVDA. Things are mostly working (see nvaccess/nvda#10784), except we're getting doubled characters when typing (but I think I know where the issue is, and it's on our side). Are there plans to have this merged soon?

If I can get it working for Narrator too, that'd be ideal. But, I guess there's no harm in merging what I have then iterating on master. I'll submit a PR today then. Thanks!

I've tested with Narrator and observed, as you did, that we aren't getting new text announcements.

In NVDA, we have an NVDAObjects.behaviors.LiveText class that waits for text change events from various APIs (such as UIA), diffs the current text against the last-reported text, and speaks the changes. Does Narrator have a similar paraddigm for consoles?

Not sure. Their code can be a bit complicated haha. I'm working with one of their devs to get a better look. I'm hoping that it can get fixed by next week because we're definitely VERY close.

LeonarddeR · 2020-03-06T17:56:10Z

If narrator doesn't use a diffing method like NVDA does, the only alternative I can think of is making the terminal a live region. Pretty sure it doesn't work like this, though, as it has several drawbacks.

LeonarddeR · 2020-03-06T17:57:34Z

Ah and in theory, UIA notification events could be used as well. If they are fired by conhost, NVDA would ignore them as NVDA only speaks notification events fire by the foreground application.

## Summary of the Pull Request `GetTextForClipboard` already exists in the TextBuffer. It makes sense to use that for UIA as well. This changes the behavior or `GetText()` such that it does not remove leading/trailing whitespace anymore. That is more of an expected behavior. ## References This also contributes to... - #4509: UIA Box Selection - #2447: UIA Signaling for Selection - #1354: UIA support for Wide Glyphs Now that the expansion occurs at before render-time, the selection anchors are an accurate representation of what is selected. We just need to move GetText to the TextBuffer. Then we can have those three issues just rely on code from the text buffer. This also means ConHost gets some of this stuff for free 😀 ## Detailed Description of the Pull Request / Additional comments - `TextBuffer::GetTextForClipboard()` --> `GetText()` - `TextBuffer::GetText()` no longer requires GetForegroundColor/GetBackgroundColor. If either of these are not defined, we return a `TextAndColor` with only the `text` field populated. - renamed a few parameters for copying text to the clipboard for clarity - Updated `UiaTextRange::GetText()` to use `TextBuffer::GetText()` ## Validation Steps Performed Manual tests for UIA using accessibility insights and Windows Terminal's copy action (w/ and w/out shift) Added tests as well.

codeofdusk · 2020-03-09T18:17:33Z

But, I guess there's no harm in merging what I have then iterating on master.

@carlos-zamora Do you still plan to do this?

carlos-zamora · 2020-03-09T19:20:34Z

But, I guess there's no harm in merging what I have then iterating on master.

@carlos-zamora Do you still plan to do this?

Yup! Here it is: #4826

## Summary of the Pull Request This notifies automation clients (i.e.: NVDA, narrator, etc...) of new output being rendered to the screen. ## References Close #2447 - Signaling for new output and cursor Close #3791 - fixed by signaling cursor changes ## Detailed Description of the Pull Request / Additional comments - Added tracing for UiaRenderer. This makes it easier to debug issues with notifying an automation client. - Fire TextChanged automation events when new content is output to the screen. ## Validation Steps Performed Verified with NVDA [1] ## Narrator Narrator works _better_, but is unable to detect new output consistently. There is no harm for narrator when this change goes in. [1] #2447 (comment)

ghost · 2020-04-22T17:17:06Z

🎉This issue was addressed in #4826, which has now been successfully released as Windows Terminal Preview v0.11.1121.0.:tada:

Handy links:

## Summary of the Pull Request This notifies automation clients (i.e.: NVDA, narrator, etc...) of new output being rendered to the screen. ## References Close #2447 - Signaling for new output and cursor Close #3791 - fixed by signaling cursor changes ## Detailed Description of the Pull Request / Additional comments - Added tracing for UiaRenderer. This makes it easier to debug issues with notifying an automation client. - Fire TextChanged automation events when new content is output to the screen. ## Validation Steps Performed Verified with NVDA [1] ## Narrator Narrator works _better_, but is unable to detect new output consistently. There is no harm for narrator when this change goes in. [1] microsoft/terminal#2447 (comment)

carlos-zamora added the Issue-Feature Complex enough to require an in depth planning process and actual budgeted, scheduled work. label Aug 15, 2019

ghost added Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting Needs-Tag-Fix Doesn't match tag requirements labels Aug 15, 2019

carlos-zamora added Area-Accessibility Issues related to accessibility Product-Terminal The new Windows Terminal. and removed Needs-Tag-Fix Doesn't match tag requirements Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting labels Aug 15, 2019

carlos-zamora self-assigned this Aug 15, 2019

carlos-zamora mentioned this issue Aug 15, 2019

Accessibility: Refactor Providers #2414

Merged

5 tasks

DHowett-MSFT added this to the Terminal v1.0 milestone Aug 16, 2019

carlos-zamora mentioned this issue Aug 29, 2019

Improve Accessibility Reliability #2609

Merged

5 tasks

This was referenced Sep 27, 2019

Introduce UiaRenderer project #2930

Merged

Attach UiaRenderer and Fire Selection Changed Events #2989

Merged

Windows Terminal: Set up UIA tree and hook it up properly #634

Closed

codeofdusk mentioned this issue Sep 30, 2019

Add support for Windows Terminal nvaccess/nvda#10305

Closed

cinnamon-msft modified the milestones: Terminal v1.0, Terminal-1911 Oct 25, 2019

cinnamon-msft modified the milestones: Terminal-1911, Terminal-1912 Nov 11, 2019

cinnamon-msft modified the milestones: Terminal-1912, Terminal-2001 Jan 6, 2020

zadjii-msft mentioned this issue Jan 23, 2020

Terminal is unsable with Screen Reader (NVDA) #4341

Closed

This was referenced Feb 3, 2020

ScreenInfoUiaProvider::GetSelection() returns an improper result #4452

Closed

Selection Expansion should not be occurring during Rendering #4465

Closed

Make ScreenInfoUiaProvider::GetSelection() Return One Selection #4466

Merged

carlos-zamora modified the milestones: Terminal v0.9, Terminal v0.10 Feb 7, 2020

carlos-zamora mentioned this issue Feb 11, 2020

Scenario: Add Support for UI Automation #4533

Closed

22 tasks

DHowett-MSFT removed this from the Terminal v0.10 milestone Feb 11, 2020

DHowett-MSFT added this to the Terminal v1.0 milestone Feb 11, 2020

carlos-zamora mentioned this issue Feb 13, 2020

Move rect expansion to textbuffer; refactor selection code #4560

Merged

codeofdusk mentioned this issue Feb 13, 2020

Add support for Windows Terminal nvaccess/nvda#10784

Merged

carlos-zamora mentioned this issue Feb 14, 2020

Abstract GetTextForClipboard() for UIA #4578

Merged

2 tasks

cinnamon-msft added Priority-1 A description (P1) v1-Scrubbed labels Feb 19, 2020

codeofdusk mentioned this issue Mar 6, 2020

Improve Word Navigation/Selection Performance #4797

Merged

carlos-zamora mentioned this issue Mar 6, 2020

Fire UIA Events for Output and Cursor Movement #4826

Merged

ghost added the In-PR This issue has a related PR label Mar 17, 2020

ghost closed this as completed in #4826 Mar 18, 2020

ghost added Resolution-Fix-Committed Fix is checked in, but it might be 3-4 weeks until a release. and removed In-PR This issue has a related PR labels Mar 18, 2020

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Accessibility: Signaling Model #2447

Accessibility: Signaling Model #2447

carlos-zamora commented Aug 15, 2019 •

edited

Loading

codeofdusk commented Sep 30, 2019

carlos-zamora commented Mar 6, 2020

codeofdusk commented Mar 6, 2020

carlos-zamora commented Mar 6, 2020

codeofdusk commented Mar 6, 2020

carlos-zamora commented Mar 6, 2020

LeonarddeR commented Mar 6, 2020

LeonarddeR commented Mar 6, 2020

codeofdusk commented Mar 9, 2020

carlos-zamora commented Mar 9, 2020

ghost commented Apr 22, 2020

Accessibility: Signaling Model #2447

Accessibility: Signaling Model #2447

Comments

carlos-zamora commented Aug 15, 2019 • edited Loading

Description of the new feature/enhancement

Proposed technical implementation details (optional)

codeofdusk commented Sep 30, 2019

carlos-zamora commented Mar 6, 2020

codeofdusk commented Mar 6, 2020

carlos-zamora commented Mar 6, 2020

codeofdusk commented Mar 6, 2020

carlos-zamora commented Mar 6, 2020

LeonarddeR commented Mar 6, 2020

LeonarddeR commented Mar 6, 2020

codeofdusk commented Mar 9, 2020

carlos-zamora commented Mar 9, 2020

ghost commented Apr 22, 2020

carlos-zamora commented Aug 15, 2019 •

edited

Loading