Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Epic] Shared UI Automation Improvements #7000

Open
DHowett opened this issue Jul 22, 2020 · 9 comments
Open

[Epic] Shared UI Automation Improvements #7000

DHowett opened this issue Jul 22, 2020 · 9 comments
Assignees
@carlos-zamora
Labels
Area-Accessibility Issues related to accessibility Issue-Scenario Priority-1 A description (P1) Product-Terminal The new Windows Terminal.
Milestone
Megathreads

Comments

@DHowett
Copy link
Member

DHowett commented Jul 22, 2020 •
edited by zadjii-msft
Loading

created and maintained by @carlos-zamora

A number of screen readers still don't use the UIA framework we've exposed in ConHost and Windows Terminal. After speaking with @codeofdusk, there's a number of suggested improvements we can do to convince screen reader manufacturers to switch over from the legacy framework.

With this changes, we can...

see also:
@ghost ghost added Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting Needs-Tag-Fix Doesn't match tag requirements labels Jul 22, 2020
@zadjii-msft zadjii-msft added ⛺ Reserved For future use and removed Needs-Triage It's a new issue that the core contributor team needs to triage at the next triage meeting labels Jul 22, 2020
@DHowett DHowett changed the title <camper> [Reserved for future use] Jul 22, 2020
@carlos-zamora carlos-zamora changed the title [Reserved for future use] [Epic] Shared UI Automation Improvements Feb 26, 2021
@carlos-zamora carlos-zamora added Area-Accessibility Issues related to accessibility Issue-Scenario Product-Terminal The new Windows Terminal. and removed Needs-Tag-Fix Doesn't match tag requirements ⛺ Reserved For future use labels Feb 26, 2021
@carlos-zamora carlos-zamora added this to the Terminal v2.0 milestone Feb 26, 2021
@carlos-zamora carlos-zamora self-assigned this Feb 26, 2021
@carlos-zamora
Copy link
Member

FYI @DHowett

@codeofdusk If there are any more asks, comment them in this thread and we'll evaluate adding them in.

@codeofdusk
Copy link
Contributor

Note: these are all conhost (not Windows Terminal) related issues. Could this be re-targeted for a Windows (such as 21H2) milestone rather than Windows Terminal please?

@codeofdusk
Copy link
Contributor

Cc @feerrenrut, @michaelDCurran.

@codeofdusk codeofdusk mentioned this issue Feb 26, 2021
Merged
@carlos-zamora
Copy link
Member

Note: these are all conhost (not Windows Terminal) related issues. Could this be re-targeted for a Windows (such as 21H2) milestone rather than Windows Terminal please?

Yes and no. UIA is a shared component between Windows Terminal and ConHost. This is just in the Terminal milestone because it's easier for us to track it.

@carlos-zamora
Copy link
Member

#5243 and #6986 are both very closely related. PR #5405 is an implementation to fix these issues, but was held back because we had to update our tests (and thus, it was a bit too risky to release just before v1.0).

For now, I'm closing #5405 but keeping the branch dev/cazamor/acc/ch/word-nav/perf alive. When we allocate time to work on this epic, we'll revive the PR and update the tests altogether.

I'm closing the PR to clean up our backlog a bit. To be clear, we still fully intend to work on this.

@carlos-zamora carlos-zamora mentioned this issue Mar 29, 2021
Closed
1 task
@codeofdusk codeofdusk mentioned this issue Apr 15, 2021
Closed
@carlos-zamora carlos-zamora mentioned this issue Jun 4, 2021
Merged
2 tasks
@carlos-zamora carlos-zamora mentioned this issue Jun 25, 2021
Closed
4 tasks
ghost pushed a commit that referenced this issue Jul 9, 2021
@carlos-zamora
Expose Text Attributes to UI Automation (#10336)
a0e5085 
## Summary of the Pull Request
This implements `GetAttributeValue` and `FindAttribute` for `UiaTextRangeBase` (the shared `ITextRangeProvider` for Conhost and Windows Terminal). This also updates `UiaTracing` to collect more useful information on these function calls. 

## References
#7000 - Epic
[Text Attribute Identifiers](https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-textattribute-ids)
[ITextRangeProvider::GetAttributeValue](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-getattributevalue)
[ITextRangeProvider::FindAttribute](https://docs.microsoft.com/en-us/windows/win32/api/uiautomationcore/nf-uiautomationcore-itextrangeprovider-findattribute)

## PR Checklist
* [X] Closes #2161 
* [X] Tests added/passed

## Detailed Description of the Pull Request / Additional comments
- `TextBuffer`:
   - Exposes a new `TextBufferCellIterator` that takes in an end position. This simplifies the logic drastically as we can now use this iterator to navigate through the text buffer. The iterator can also expose the position in the buffer.
- `UiaTextRangeBase`:
   - Shared logic & helper functions:
      - Most of the text attributes are stored as `TextAttribute`s in the text buffer. To extract them, we generate an attribute verification function via `_getAttrVerificationFn()`, then use that to verify if a given cell has the desired attribute.
      - A few attributes are special (i.e. font name, font size, and "is read only"), in that they are (1) acquired differently and (2) consistent across the entire text buffer. These are handled separate from the attribute verification function.
   - `GetAttributeValue`: Retrieve the attribute verification of the first cell in the range. Then, verify that the entire range has that attribute by iterating through the text range. If a cell does not have that attribute, return the "reserved mixed attribute value".
   - `FindAttribute`: Iterate through the text range and leverage the attribute verification function to find the first contiguous range with that attribute. Then, make the end exclusive and output a `UiaTextRangeBase`. This function must be able to perform a search backwards, so we abstract the "start" and "end" into `resultFirstAnchor` and `resultSecondAnchor`, then perform post processing to output a valid `UiaTextRangeBase`.
- `UiaTracing`:
   - `GetAttributeValue`: Log uia text range, desired attribute, resulting attribute metadata, and the type of the result.
   - `FindAttribute`: Log uia text range, desired attribute and attribute metadata, if we were searching backwards, the type of the result, and the resulting text range.
   - `AttributeType` is a nice way to understand/record if the result was either of the reserved UIA values, a normal result, or an error.
- `UiaTextRangeTests`:
   - `GetAttributeValue`:
      - verify that we know which attributes we support
      - test each of the known text attributes (expecting 100% code coverage for `_getAttrVerificationFn()`)
   - `FindAttribute`: 
      - test each of the known _special_ text attributes
      - test `IsItalic`. NOTE: I'm explicitly only testing one of the standard text attributes because the logic is largely the same between all of them and they leverage `_getAttrVerificationFn()`.

## Validation Steps Performed
- @codeofdusk has been testing this Conhost build
- Tests added for Conhost and shared implementation
- Windows Terminal changes were manually verified using accessibility insights and NVDA
@carlos-zamora
Copy link
Member

Note about NVDA:

@carlos-zamora
Copy link
Member

Since I'll be adding a lot of testing for #6986 (via #10925), I should keep an eye on #10819 (that might be fixed as a part of this).

Also, add these to the epic:

@carlos-zamora carlos-zamora mentioned this issue Sep 2, 2021
Merged
2 tasks
ghost pushed a commit that referenced this issue Sep 16, 2021
@carlos-zamora
[A11y] Treat last character as 'end of buffer' (#11122)
d08afc4 
## Summary of the Pull Request
Updates our `UiaTextRange` to no longer treat the end of the buffer as the "document end". Instead, we consider the "document end" to be the line beneath the cursor or last legible character (whichever is further down). In the event where the last legible character is on the last line of the buffer, we use the "end exclusive" position (left-most point on a line one past the end of the buffer). 

When movement of any kind occurs, we clamp each endpoint to the document end. Since the document end is an actual spot in the buffer (most of the time), this should improve stability because we shouldn't be pointing out-of-bounds anymore.

The biggest benefit is that this significantly improves the performance of word navigation because screen readers no longer have to take into account the whitespace following the end of the prompt.

Word navigation tests were added to the `TestTableWriter` (see #10886). 24 of the 85 tests were failing, however, they don't seem to interact with the document end, so I've marked them as skip and will fix them in a follow-up. This PR is large enough as-is, so I'm hoping I can take time in the follow-up to clean some things on the side (aka `preventBoundary` and `allowBottomExclusive` being used interchangeably).

## References
#7000 - Epic
Closes #6986 
Closes #10925

## Validation Steps Performed
- [X] Tests pass
- [X] @codeofdusk has been personally testing this build (and others)
@carlos-zamora carlos-zamora mentioned this issue Sep 17, 2021
Closed
@zadjii-msft zadjii-msft added the Priority-1 A description (P1) label Jan 4, 2022
@zadjii-msft zadjii-msft modified the milestones: Terminal v2.0, 22H1 Jan 4, 2022
@zadjii-msft zadjii-msft modified the milestones: 22H1, 22H2 Aug 3, 2022
@codeofdusk
Copy link
Contributor

codeofdusk commented Oct 4, 2022 •
edited
Loading

NVDA 2022.3 has been released, and now uses UIA by default in the version of ConHost included with Windows SV2+! Thanks everyone for your excellent work and collaboration!

@DHowett
Copy link
Member Author

DHowett commented Oct 4, 2022

Thanks @codeofdusk! This is great news!

@zadjii-msft zadjii-msft modified the milestones: 22H2, Megathreads Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carlos-zamora carlos-zamora

Labels
Area-Accessibility Issues related to accessibility Issue-Scenario Priority-1 A description (P1) Product-Terminal The new Windows Terminal.
Projects
None yet
Milestone
Megathreads
Development

No branches or pull requests
5 participants
@DHowett @codeofdusk @carlos-zamora @zadjii-msft @cinnamon-msft