[Accessibility] Add accessibility signal for progress status in notebooks

[jooyoungseo]

Type: Feature Request

CC @meganrogge

I think progress status is used in many cases, but I encounter this when using Jupyter Notebook. If current cell execution takes a long time, sighted people may see some visual cues indicating its progress status. However, there is no non-visual indication for screen reader users. Please consider adding audio cues.

Also, please consider adding alert as an independent configuration for deaf-blind and braille users.

VS Code version: Code - Insiders 1.84.0-insider (7dff078, 2023-10-24T05:33:54.929Z)
OS version: Windows_NT x64 10.0.22621
Modes:

[meganrogge]

Another use case @AFre100

[AFre100]

@jooyoungseo thank you! Pretty sure this will be implemented, but I had a sidebar question: I’m always updating my audit of assistive audio in the wild—do you have any favorite progress audio cues you might name?

[meganrogge]

Would also apply to settings search progress.

We have the in progress chat response cue already, should that be reused?

[AFre100]

@meganrogge would it make sense to implement the same system as Copilot chat progress -- i.e. only triggering after x seconds, looping every y seconds, leaving/returning to window, etc?

[jooyoungseo]

If it is possible to estimate the progress ratio, I suggest an incremental tone just like Gauge Sounds. Otherwise, using audio cue the same as Copilot Chat would be fine.

[AFre100]

@jooyoungseo thanks! Love this idea. However, I'd really want to hear it in context to confirm that I'm understanding it right. I've searched w/out luck -- do you know how I might get a screen recording of it?

[AFre100]

@jooyoungseo Just wanted to check in about this -- you don't have the gauge sounds implemented, do you? If so, might you send me a screen recording of it for more context, or do you have any leads on how I might get a hold of some footage?

[AFre100]

@meganrogge @jooyoungseo Having footage of the incremental tone system, like Gauge Sounds would still be helpful if Jooyoung, you have a way to make that happen. In the meantime though, I'm curious what you all think (?):

If scenarios where progress takes greater than x seconds create a significant enough pain point among screenreader users, I do think having a system like this would be greatly beneficial (like a realtime progress bar via visual feedback).

From my understanding, there would need to be data that quantifies estimated percentage of completion, which informs the real time audio system as it increments. Flagging that we'd also need to investigate what that wait time threshold needs to be, in oder to justify the engineering needed for incrementing — e.g. maybe it's >7, 12, 18 sec?

If the issue isn't significant enough, then yes, we could use the chat progress audio loop system (or something similar).

[AFre100]

status is used in many cases, but I encounter this when using Jupyter Notebook. If current cell execution takes a long time, sighted people may see some visual cues indicating its progress status

@meganrogge Trying to pick this conversation back up, since the speech to text ticket might be relevant. I have been working on this. Can you please send me screen grab videos of both the Settings search progress and Jupyter Notebook current cell execution examples? Thanks!

[amunger]

Are you just trying to see what cell execution looks like? Here is an example.
We don't have a way to know how long the execution is going to take since it's executing arbitrary code, it could be hours.

I'm not sure about settings search - if I understand what this is referring to, I've never seen the search take longer than a few hundred milliseconds.

[AFre100]

@amunger Thank you!

[AFre100]

@amunger @meganrogge @jooyoungseo Just wanted to see if we can come to a consensus. What kind of wait times exist for these use cases? Is it usually max 2-3 seconds, does it ever last >5, 12, 18 sec?

If we aren't using an incremental tone system, do we need an additional cue to indicate completion in cases where there are multiple progress statuses?

Microsoft team, what do we think about leveraging progress ratio calculation for an incremental tone system?

[meganrogge]

I believe I have answered your questions on the other issue @AFre100.

Are there downsides to using an incremental tone system?

[AFre100]

@meganrogge heads up -- if progress loop is the consensus, we're most likely updating the existing Copilot Chat progress loop sound (which would be used here too). That's being done in order to improve upon some feedback. I expect it to be close enough to the original sound where it shouldn't cause any friction for users who depend on it.

What's the next audio file delivery deadline for sound to be:

a) implemented for any non-copilot chat use of progress status loop (notebook cells, settings search, etc)?

b) swapped into existing copilot chat usage

[meganrogge]

Sounds good @AFre100. If we want this in by April, then April 17th

[AFre100]

@meganrogge thanks! And if it needed until the following release, that would be roughly one month later, a mid-May deadline, correct?

[rperez030]

@meganrogge heads up -- if progress loop is the consensus, we're most likely updating the existing Copilot Chat progress loop sound (which would be used here too).

curious how you will determine the ETA for copilot generation so that the progress sound can reflect reality.

[meganrogge]

We cannot ATM determine the ETA, thus I don't think we should go with an incremental tone. Rather, we should play the sound every few seconds to indicate something is still happening.

[meganrogge]

Another use case is in the scm view, when changes are getting committed or pushed, there's a progress bar. We should play the cue for this to indicate the non-visual state.

Screen.Recording.2024-04-04.at.7.53.21.AM.mov

[rperez030]

Another use case is in the scm view, when changes are getting committed or pushed, there's a progress bar. We should play the cue for this to indicate the non-visual state.

Screen.Recording.2024-04-04.at.7.53.21.AM.mov

if the proper markup is used, screen readers such as NVDA that support the progress bar pattern can communicate that using their standard mechanism. I don't oppose providing a vS Code signal if we believe it to be valuable, but I would prioritize supporting established accessibility practices first whenever possible.

[meganrogge]

That's a great point @rperez030.

Does VoiceOver have a similar concept @rperez030 ?

Is it the ARIA: progressbar role ?

[meganrogge]

One concern I have with playing this audio cue in the SCM, chat response view, settings UI, etc is these could happen simultaneously, which would be very disruptive and confusing.

The alternative would be to cancel one that's playing when focus is moved from that area, but I worry that is also a bad solution.

Perhaps we can do as @rperez030 suggests for the cases that have an actual progress bar rendered (SCM, settings search UI, etc.) instead, using this sound only for the cases that do not - notebook cell progress, chat response pending, etc.

[rperez030]

That's a great point @rperez030.

Does VoiceOver have a similar concept @rperez030 ?

Is it the ARIA: progressbar role ?

Yes to all. I don't have Chrome install in this machine so cannot test chromium / voiceover support, but there is a working example to test here: #196396 (comment)

I tested with Safari and it plays the default voiceover progress sound as expected, similar to NVDA. Of course, you also have braille feedback, which we wouldn't have if we rely on audio cues along.

[meganrogge]

@amunger, unassigning myself as this issue is now strictly about notebooks adopting chatResponsePending. We should probably rename that and migrate the setting. LMK if you need help with that.

Creating a separate issue about progress status in places with the progress bar, which we will add the role for as @rperez030 has suggested.

[AFre100]

@meganrogge @amunger There's still the possibility that both notebook cell progress and chat response pending will take place simultaneously, right? If, so, what did you all have in mind — keeping only one at a time via a focus shift/context switching?

One concern I have with playing this audio cue in the SCM, chat response view, settings UI, etc is these could happen simultaneously, which would be very disruptive and confusing.

The alternative would be to cancel one that's playing when focus is moved from that area, but I worry that is also a bad solution.

Perhaps we can do as @rperez030 suggests for the cases that have an actual progress bar rendered (SCM, settings search UI, etc.) instead, using this sound only for the cases that do not - notebook cell progress, chat response pending, etc.

[meganrogge]

That's unlikely to happen. But yeah, we'll probably cancel based on focus

[AFre100]

@meganrogge @amunger Feel free to use the existing chatResponsePending progress sound if we're antsy for notebooks to have a progress sound for April. Would be nice to start with the updated sound in this new use case but it's not worth blocking users from its use. Otherwise, I expect to have the updated progress sound for May (as discussed above), which would be used wherever chatResponsePending would be used. (I'm OOF 4/17-4/24 just fyi.)

[venkateshpotluri]

i've been taking advantage of these audio cues when running long-running operations in notebooks in the latest insider build, thank you for implementing this! I wanted to make a few suggestions based on my observations to improve the experience.

• determinant Vs. indeterminant progress bars could use different audio cues. I observe that VSCode plays the chat response pending sound for any operation that takes above a certain time. It may be useful to have a different sound as suggested by @jooyoungseo that conveys the actual progress for long operations that surface this information (e.g. operations using the TQDM progress bars), with the chat response pending sound reserved for operations with indeterminant progress.
• option to set smart repeat frequency based on desired verbosity for determinant progress bars. The current experience results in an audio cue being played every few seconds irrespective of the duration of the operation. This could result in noisy output for long-running operations. For example, hearing an alert every 3 seconds for an operation that is expected to take a few hours may not be desirable. updates with lesser verbosity, combined with VSCode's ability to play these cues in the background could enable an amazing multi-tasking developer experience. TQDM progress bars provide the following information to screen readers which makes me think that smart repeat frequency may be possible: " 57%|█████▊ | 115/200 [1:55:47<1:25:38, 60.46s/it]"
• Option to disable audio cues when VSCode is out of focus. This may be less necessary if the alert frequency can be adjusted based on operation time for tasks that provide information for the runtime to be estimated as suggested in my previous point but may still be useful for long-running tasks where runtime cannot be estimated.
• potentially confusing reasons for progress. I do not have a suggestion yet on how to address this, but I notice that VSCode plays a few progress sounds when I wake my computer up even if there is no running operation. I work by connecting to a computer via remote-ssh so I am not sure if this sound is a result of my local IDE re-connecting to VSCode Server on the remote machine. Being able to identify what a progress bar corresponds to could be helpful.

[AFre100]

@venkateshpotluri thanks! Checking this out when I return tomorrow. Stay tuned