List: Orca announces typed characters instead of element when typing to navigate

[jvesouza]

Issue Type: Feature Request

In Breadcrumbs , when I am looking for an element, the orca stops announcing the name of the element as soon as it is found, even if I keep typing its name.

For a better understanding try the following steps:

1. Launch orca.
2. Create a file and declare a function called f123.
3. Press ctrl+shift+.
4. Type quickly f123.

As soon as the letter f is typed, the orca tries to announce the name of the function found but is immediately interrupted because the number 1 was typed. It is difficult for me to know if the function was found or not.
Ideally for me, the function name would continue to be announced as we type, even if it was the same.

CC @isidorn

VS Code version: Code - Insiders 1.45.0-insider (d342048, 2020-04-23T05:54:24.582Z)
OS version: Linux x64 5.6.6-arch1-1

[jrieken]

@isidorn Is this tree-filtering specific or something I need to opt-in for breadcrumbs

[isidorn]

@jrieken oh we are editing at the same time..

[isidorn]

Ok, investigted. Breadcrumbs is using a list, and on key press list navigates to that element based on prefix matching. This works just fine for me and I think breadcrumbd are doing everything proper.

The reason why Orca decided to announce the character typed is up to Orca. I do not get this behavior on the Mac.
@joanmarie can you also take a look once you get a chance? When does Orca decide to announce characters typed and can we control that somehow from VS Code?
@jvesouza do you also get this behavior for example when you open the explorer with files aaa.txt and bbb.txt and you start typing aaa or bbb that Orca would say the characters typed and would interupt the announcment of aaa.txt?

[jvesouza]

@isidorn Orca does not stop announcing the characters as I type.
Using BreadCrumbs when I type a, orca announces aaa.txt as expected, assuming aaa.txt is a present element.
But if I type aa orca does not announce aaa.txt.

Yes, the same behaviour when I open explorer.
If I type aa or bb, the file name aaa.txt or bbb.txt is not announced.

[isidorn]

@jvesouza thanks for clarifying. Thus this is not a breadcrumbs issue but a general list issue with Orca.
So the best would be to wait for @joanmarie to give us some insights on when orca decides to stop the annoucment to announce a character.

Since by typing aa you are still on the aaa.txt element, nothing changed and orca does not annoucnce since the original annoucment was interupted by the second a.

[joanmarie]

@isidorn Every time a key is pressed Orca interrupts speech whether or not it has key echo enabled. I think that behavior is expected and not an issue/bug in Orca. Is NVDA doing something different?

I'm happy to discuss the best way to solve this.

[joanmarie]

I'll toss out the first idea to get the brainstorming started: Could VSCode detect this condition (user keeps typing and selection hasn't changed) and re-announce the selected item via ARIA live regions?

[isidorn]

@joanmarie yes NVDA behaves the same. And I think we should handle it on the VS Code side.
We coudl re-announce it, however it feels a bit hackish. Since currently in our whole tree I believe we never use aria-lie regions, only based on focus do things get read out.
So the heuristic would be: if by typing we re-navigate to the same element use alert to announce it's ariaLabel.
Assinging to May to potentially tackle then and we might get new ideas in the meantime.

fyi @joaomoreno

[isidorn]

Created this PR for discussion #97057
Currently work in progress.

[joanmarie]

@jvesouza: When this problem occurs, have you tried Orca's basic "where Am I?" command? For me it solves the problem without there needing to be any changes made in VSCode.

BTW, Orca saying things like "0 of 0" when you use "where Am I?" in some of VSCode's trees and lists is due to a Chromium issue or two which have already been fixed. But with that nit aside, "where Am I" might be the way to go. Thoughts?

[jvesouza]

@joanmarie Yes, it works and is what I normally use. The only other problem, although this is not a big problem, is that I need to use one more key.

[isidorn]

Fixed via pr #97057
@jvesouza please try it with VS Code insiders from Monday and let us know if it behaves good for you
@joanmarie thanks again for the investigation!

[isidorn]

@jvesouza did you get a chance to try this out? So we add a verified label if all is good.

[jvesouza]

I'm sorry for the delay in testing, I should have returned earlier.

It seems that nothing has changed because when I look for something in bread crumbs only the first item on the list is announced.

I have in my code two functions, open_input and open_output.
If I quickly type op nothing is announced by orca. Probably the fact that I type the letter p interrupts the orca's speech.
As this only happens if I type quickly, it is not a big problem. For me we can close this bug.
Thanks.

[isidorn]

@jvesouza thanks for trying it out. Probably things do not get read out because we use live-regions to send out an alert and orca does not see a change between alerts and probably does not read anything.

So even though we pushed a fix for this the experience is not that nice.
@joanmarie is there some way to force orca to re-read an alert?

[joanmarie]

I'll do two separate tests. Here are my findings using Orca master and the steps in the opening report. I just tried it and Orca spoke "f123" to me after I released the "3".

From my debug.out, I see:

vvvvv PROCESS ATSPI_KEY_RELEASED_EVENT: '3' (12) vvvvv
13:34:26.467028 - HOST_APP: [application | code-insiders]
13:34:26.467600 - WINDOW:   [frame | foo.py - Visual Studio Code - Insiders]
13:34:26.468154 - LOCATION: [tree item | f123]
13:34:26.468226 - CONSUME:  False (Script indication)
13:34:26.468306 - TOTAL PROCESSING TIME: 0.0018
^^^^^ PROCESS ATSPI_KEY_RELEASED_EVENT: '3' (12) ^^^^^

A bit later I see:

vvvvv PROCESS OBJECT EVENT object:text-changed:insert vvvvv
13:34:27.036398 - OBJECT EVENT: object:text-changed:insert (0, 4, f123)
                  app.name='code-insiders' name='' role='alert' 
                  states='enabled sensitive visible' 
                  relations='' 
                  interfaces='Accessible Action Collection Component Hypertext Hyperlink Text' 
                  attributes='class:monaco-alert container-busy:false container-atomic:true container-relevant:additions text container-live:assertive atomic:true relevant:additions text live:assertive xml-roles:alert tag:div display:block'
13:34:27.037340 - EVENT MANAGER: Getting script for [application | code-insiders] (check: False)
13:34:27.037526 - EVENT MANAGER: Script is <orca.scripts.toolkits.Chromium.script.Script object at 0x7f6c77d63b90>
13:34:27.037558 - EVENT MANAGER: Change active script: False (The script for this event is already active.)
13:34:27.039645 - WEB: Event to be handled as live region
13:34:27.042025 - EVENT MANAGER: locusOfFocus: [tree item | f123] activeScript: <orca.scripts.toolkits.Chromium.script.Script object at 0x7f6c77d63b90>
[...]
TOTAL PROCESSING TIME: 0.0057
^^^^^ PROCESS OBJECT EVENT object:text-changed:insert ^^^^^

And shortly after that:

vvvvv PRESENT LIVE REGION MESSAGE vvvvv
13:34:27.144070 - SPEECH OUTPUT: 'f123' voice=system{'established': True, 'family': {'name': 'zoe-embedded-high', 'gender': None, 'lang': 'en', 'dialect': '', 'variant': 'US'}}
13:34:27.146495 - SPEECH DISPATCHER: Speaking '<speak><mark name="0:4"/>f123</speak>' 
                  ORCA rate 40.0, pitch 5.0, volume 5.0, language en, punctuation: NONE 
                  SD rate -18, pitch 10, volume 100, language en
13:34:27.148408 - LIVE REGIONS: messages in queue: 0
^^^^^ PRESENT LIVE REGION MESSAGE ^^^^^

Now I heard Orca speak it so I know it wasn't interrupted. But pretending it wasn't my debug.out, I then did a search for "Interrupting presentation" to see if Orca might have immediately cut off speech after it sent it to be spoken but before it was actually heard. I don't see that until the window is deactivated.

[joanmarie]

Now I tried this case from @jvesouza:

I have in my code two functions, open_input and open_output.
If I quickly type op nothing is announced by orca. Probably the fact that I type the letter p interrupts the orca's speech.

I heard Orca say "open_output" after I typed "op" quickly. Again, looking at my debug.out:

Here's the release of "p":

vvvvv PROCESS ATSPI_KEY_RELEASED_EVENT: 'p' (33) vvvvv
13:49:05.646674 - HOST_APP: [application | code-insiders]
13:49:05.647254 - WINDOW:   [frame | foo.py - Visual Studio Code - Insiders]
13:49:05.647811 - LOCATION: [tree item | open_output]
13:49:05.647881 - CONSUME:  False (Script indication)
13:49:05.647950 - TOTAL PROCESSING TIME: 0.0018
^^^^^ PROCESS ATSPI_KEY_RELEASED_EVENT: 'p' (33) ^^^^^

Here's the text insertion from the alert:

vvvvv PROCESS OBJECT EVENT object:text-changed:insert vvvvv
13:49:06.400060 - OBJECT EVENT: object:text-changed:insert (0, 11, open_output)
                  app.name='code-insiders' name='' role='alert' 
                  states='enabled sensitive visible' 
                  relations='' 
                  interfaces='Accessible Action Collection Component Hypertext Hyperlink Text' 
                  attributes='class:monaco-alert container-busy:false container-atomic:true container-relevant:additions text container-live:assertive atomic:true relevant:additions text live:assertive xml-roles:alert tag:div display:block'
13:49:06.402261 - EVENT MANAGER: Getting script for [application | code-insiders] (check: False)
13:49:06.402404 - EVENT MANAGER: Script is <orca.scripts.toolkits.Chromium.script.Script object at 0x7fa475750ad0>
13:49:06.402436 - EVENT MANAGER: Change active script: False (The script for this event is already active.)
13:49:06.409254 - WEB: Event to be handled as live region
13:49:06.414915 - EVENT MANAGER: locusOfFocus: [tree item | open_output] activeScript: <orca.scripts.toolkits.Chromium.script.Script object at 0x7fa475750ad0>
[...]
TOTAL PROCESSING TIME: 0.0149
^^^^^ PROCESS OBJECT EVENT object:text-changed:insert ^^^^^

Here's Orca speaking the live region:

vvvvv PRESENT LIVE REGION MESSAGE vvvvv
13:49:06.516505 - SPEECH OUTPUT: 'open_output' voice=system{'established': True, 'family': {'name': 'zoe-embedded-high', 'gender': None, 'lang': 'en', 'dialect': '', 'variant': 'US'}}
13:49:06.519018 - SPEECH DISPATCHER: Speaking '<speak><mark name="0:11"/>open_output</speak>' 
                  ORCA rate 40.0, pitch 5.0, volume 5.0, language en, punctuation: NONE 
                  SD rate -18, pitch 10, volume 100, language en
13:49:06.520605 - LIVE REGIONS: messages in queue: 0
^^^^^ PRESENT LIVE REGION MESSAGE ^^^^^

And there's no "Interrupting presentation" until I clicked with the mouse.

[joanmarie]

@jvesouza Can you please capture a debug.out in which Orca fails to present the live region to you? I'm hoping based on my previous two examples, you'll be able to identify the following:

1. Did we get a text insertion for the live region after you released the final key?
2. Did Orca identify it as something to handle as a live region?
3. If so, did Orca present it?
4. If Orca presented it according to your debug.out, but you didn't hear it, what interrupted speech?

Or you could just send the file to me.

[isidorn]

@joanmarie thanks a lot for looking into this.

[jvesouza]

I found that the problem only seems to happen when the procedure is performed a second time.
I am attaching a python script that I used in the tests to reproduce the problem.

1. Launch orca
2. Open the attached file in VSCode
3. Press ctrl+home
4. Press ctrl+shift+.
5. Quickly type op

In my environment orca announces 'open input' as expected.

Now press the esc key and repeat the last two steps.
In my environment orca announces nothing after I type the letter p.

I will also send via e-mail orca debug file to @joanmarie
stat.zip

[joanmarie]

Looking at @jvesouza's debug.out, the first time we get an object:text-changed:insert event for the alert; the second time we don't; we only get an object:children-changed:add event. And Orca ignores that event because the child being added is a static-text leaf (and the text insertion event is expected).

So I guess the thing to do is come up with a stand-alone test case and file a bug against Chromium for the missing accessibility event. Either that and/or figure out how to force Chromium to emit the text insertion event.

[jvesouza]

So I guess the thing to do is come up with a stand-alone test case and file a bug against Chromium for the missing accessibility event. Either that and/or figure out how to force Chromium to emit the text insertion event.

Has this not been fixed in more current versions of chromium?
@isidorn Would it be possible to test with an insiders version that uses electron 8 or higher?
Thanks.

[isidorn]

@jvesouza here's the latest Electron 8 exploration build so if you can try it out. If not I can try it out later today

Tar.gz
https://az764295.vo.msecnd.net/exploration/63c2c8d5f8c8a1e32c4f1bd7c3b4eabbbaffbe68/code-exploration-1591599442.tar.gz

Deb
https://az764295.vo.msecnd.net/exploration/63c2c8d5f8c8a1e32c4f1bd7c3b4eabbbaffbe68/code-exploration_1.47.0-1591599061_amd64.deb

Rpm
https://az764295.vo.msecnd.net/exploration/63c2c8d5f8c8a1e32c4f1bd7c3b4eabbbaffbe68/code-exploration-1.47.0-1591599242.el7.x86_64.rpm

Snap
https://az764295.vo.msecnd.net/exploration/63c2c8d5f8c8a1e32c4f1bd7c3b4eabbbaffbe68/code-exploration-1591599995.snap

[jvesouza]

@isidorn Thank you for the link. Unfortunately the behavior appears to be exactly the same.
Just to be documented, the electron version in exploration is 8.3.1.