Spec should provide algorithms to expose individual devices via enumerateDevices

[youennf]

As part of media capture-output and getUserMedia in chrome picker, there is an effort to only expose a device after device has been picked by user. Algorithms should be added to allow this.

[guest271314]

Some recent notes

To resolve a series of issues the formal algorithm should be

1. enumerateDevices() => lists all devices => before current getUserMedia() algorithm
2. For backward compatibility getUserMedia() implicitly calls enumerateDevices() => lists all devices
3. User can select any and all devices for that permission session, which avoids needing to execute getUserMedia() more than once
4. All of the captured devices each have single MediaStreamTrack representation of capture, necessarily meaning there can be more than 1 microphone and 1 camera in MediaStream from getUserMedia(); it is up to user to process the tracks thereafter; there should not be an asumption that only 1 microphone and 1 camera are being captured for the duration of permissions, with speakers, cameras and other accessible media capable of being added as a device during permission being granted
5. However, to reign in confusion as to which devices are being captured the list of captured devices needs to be dynamic; and to do that with existing enumerateDevices() would mean the function must perform at least two different task; replace getUserMedia() as primary permission and device selection mechanism and thereafter list captured devices.
6. Avoid overloading algorithms designed for a different purpose
   &. Describe a getSystemDevices() function used to first get permission to list all devices, not just microphone and camera; user selects devices for that permission session, thereafter enumerateDevices() lists only those initially selected devices for that session; if plug-and-play is expected to work ondevicechange needs to work by user checking to watch for plug-and-play that the user explicitly grants permission to do so for meaning enumerateDevices() will be consistent and reliable due to being dynamic, otherwise only return selected devices
7. The algorithm needs to be basic enough to be implemented universsally with the same code, to avoid retuning back to this issue due to Media Capture and Streams not mandating a uniform UI, to that end a simple form that can be compised of plain text to either check or not check spekers, monitor of device, cameras, microphones, MIDI, et al., e.g., (Only reveal labels of devices user has given permission to #640 (comment)) to avoid workarounds and inconsistent behaviour moving forward it must be acknowledged that getUserMedia() is capable of and is being used for other or more than 1 micorphone and 1 camera, that is, an extensible API design-pattern from the beginning instead of attempting to amend the functionality of prior art
8. Thereafter enumerateDevices() must only return devices selected during the getSystemDevices() selection, with ondevicechange being the capability to discover new devices - that the user explicitly grants permission to do so; for separation of concerns

getSystemDevices() => [audio output devices (speakers), cameras, monitors of audio devices (What-U-Hear), microphones, ...]
=> User selects any and all devices they want to capture, this is global permission grant for this session, encrypt, de-fingerprint surface, et al here.
=> If getUserMedia() is called 1 microphone, 1 camera selected for that puprose by user in MediaStream
=> enumerateDevices() returns only devices selected by user, makes sure the device listed is the one being captured else loses value with regard to reliability #693

[bradisbell]

@youennf Could you please clarify what this issue is?

As part of media capture-output and getUserMedia in chrome picker

Is this related to the capture of output audio devices? Or the usage of audiooutput, like setSinkId today?

there is an effort to only expose a device after device has been picked by user

Where is it proposed that the device picking occurs? How could an app allow a user to pick a device without knowing of the device existence?

[youennf]

Fixed by #718

[bradisbell]

It seems you've removed the ability for the user to give permission to all audio devices to the application, as well as the ability for the application to access all audio devices. This is disastrous for several of my use cases.

Is my understanding correct?

[youennf]

This PR is specifically about audio output devices.
No algorithm was really defined before so we now have an algorithm and can build upon it based on use cases.
Currently, the rules are:

• expose all audio output tied to an exposed microphone
• expose all audio output the user selected through prompts.

Can you describe which of your scenarios would not be supported?

[bradisbell]

Audio output or input, I have several projects where the application needs to be in control of selecting audio input/output as several devices are used at simultaneously. Once the user has given permission, the application should be able to enumerate devices and use them.

I also have two projects where the choice of input and output devices happens remotely. Think of a situation where multiple users are collaborating on a live stream. They can mix audio/video for each other, as well as switch devices when necessary. A user performing live can't be bothered to fiddle with their controls, but a remote operator can, once that user has allowed the application permission initially.

[youennf]

For microphones, getting access to one will give you access to all microphones.

As of audio output, the model we are building is the following.

• If the user never did any set-up, web page will have to use selectAudioOutput to ask user to select which audio output to use. If user set-up requires 3 different audio outputs, the web page will call selectAudioOutput 3 times (with a prompt each time).
• If the user did the set-up previously, on page load, the web page has device IDs it wants to reuse. Assuming the user set-up is exactly the same as previously and user granted persistent access, the web page will call selectAudioOutput 3 times, no prompt will happen and all 3 devices will be usable from that point on. If device is not there, user will be prompted to select the device to use.
  How does this fit with the use-cases you have in mind?

[guest271314]

expose all audio output the user selected through prompts.

selectAudioOutput does not select any devices that capture system audio output,
selectAudioOutput selects a deviceId to for setSinkId(), correct?