[Mac] Allow audio device selection #21

hiroshihorie · 2022-03-10T18:25:44Z

Goals

Audio device list & selection at runtime.
Detect and switch device when default device changes at runtime. (Connect AirPods etc)
Feed audio sample buffers directly.

bypassAudioProcessing is temporarily removed, but I will make it available again.
-> Now available again.

davidzhao

incredible work @hiroshihorie!

For the new files such as: RTCAudioDeviceModule.mm, are those from somewhere else in the webrtc tree?

davidzhao · 2022-03-14T05:27:48Z

modules/audio_device/mac/audio_device_mac.cc

@@ -867,12 +864,9 @@ int32_t AudioDeviceMac::RecordingDeviceName(uint16_t index,
  }

  memset(name, 0, kAdmMaxDeviceNameSize);
-
-  if (guid != NULL) {


can anything else call this function and pass in NULL for RecordingDeviceName ?

RecordingDeviceName is supposed to populate name and guid of the device,
but populating guid was not implemented in their code.

So I implemented to return the AudioDeviceID which is not technically guid but
it works for identifying the device.

hiroshihorie · 2022-03-14T07:40:27Z

Yes there is a webrtc::AudioDeviceModule class.
https://chromium.googlesource.com/external/webrtc/stable/webrtc/+/15a8a6e7edb2c8df7f5b21e759ace213a4ab7292/modules/audio_device/include/audio_device.h#19

davidzhao · 2022-03-14T18:02:00Z

Yes there is a webrtc::AudioDeviceModule class. https://chromium.googlesource.com/external/webrtc/stable/webrtc/+/15a8a6e7edb2c8df7f5b21e759ace213a4ab7292/modules/audio_device/include/audio_device.h#19

But is the ObjC wrapper class from the tree? or is that a new file you authored?

hiroshihorie · 2022-03-14T18:46:05Z

Yes there's a few new files I authored.

for RTCAudioDeviceModule, In the future,
I might want to add ability to create virtual devices.
RTCAudioDeviceModule.addVirtualDevice(...)

then user can call the existing (implemented in this PR) RTCAudioDeviceModule.setInputDevice(...),
to switch to their virtual device.

This virtual device class will conform to a new RTCVirtualAudioDevice protocol (or base class) so users can feed their own CMSampleBuffers through here.

This is just a thought for design 🙂

I want to achieve unified design that will work for both iOS / Mac platforms, but the underlying audio system is pretty different for each platform. So I'm not sure if it's possible.

Another concern is mixing the audio, for example game streaming, user would like to speak while having game audio. ReplayKit delivers audio buffer separately for app and mic.
https://developer.apple.com/documentation/replaykit/rpsamplebuffertype
So it needs to get mixed somehow... still investigating ...

davidzhao · 2022-03-14T18:47:32Z

Got it, for the new files, could you add a LiveKit copyright in it? This will help differentiate which files originated from upstream vs us.

davidzhao · 2022-03-14T18:48:14Z

This virtual device class will conform to a new RTCVirtualAudioDevice protocol (or base class) so users can feed their own CMSampleBuffers through here.

This is just a thought for design 🙂

I want to achieve unified design that will work for both iOS / Mac platforms, but the underlying audio system is pretty different for each platform. So I'm not sure if it's possible.

YES! This is going to be awesome!!

Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail.

hiroshihorie · 2022-05-27T10:32:41Z

RTC's thread checks were failing so improved thread handling.
All access to ADM methods are from worker thread now.

RTC_DCHECK_RUN_ON(&main_thread_checker_); checks whether its called from the same thread that the ADM was created on (which must be created on worker thread).
https://github.com/webrtc-sdk/webrtc/blob/main/modules/audio_device/audio_device_buffer.cc#L135
https://github.com/webrtc-sdk/webrtc/blob/main/modules/audio_device/audio_device_buffer.cc#L149

hiroshihorie · 2022-05-29T09:18:53Z

Additional required logic

Active device (lost) disconnected ✅

Current: Nothing happens (audio input/output stops)
Expected: Switch to default device and continue input/output.

OS's default device changed ✅

Current: Audio input/output continues on the same device even if default device is changed.
Expected: if default device is switched by the OS, change input/output to the new default device.

hiroshihorie · 2022-07-27T11:01:26Z

hey @hiroshihorie, It seems that changing the PR from target m97 to m104 introduces additional diffs, should we still target m97 after merging we get the minimal diff for the patch?

Yup, I changed the target which wasn't a good idea..

cloudwebrtc · 2022-07-27T15:12:13Z

The code looks good.

davidzhao · 2022-07-28T16:36:01Z

@hiroshihorie shall we merge this as well as into 104?

* first attempt * remove unused dep * init playout / recording * use AudioDeviceID as guid * switch device method * equality * default device * `isDefault` property * dont format default device name * type param * bypass * refactor * fix * append Audio to thread labels * ref * lk headers * low level apis * fix thread checks Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail. * switch to default device when removed * close mixerManager if didn't switch to default device * default audio device switched * expose devices update handler * fix ios compile * fix bug: don't always recreate RTCAudioDeviceModule

* [Mac] Allow audio device selection (#21) * first attempt * remove unused dep * init playout / recording * use AudioDeviceID as guid * switch device method * equality * default device * `isDefault` property * dont format default device name * type param * bypass * refactor * fix * append Audio to thread labels * ref * lk headers * low level apis * fix thread checks Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail. * switch to default device when removed * close mixerManager if didn't switch to default device * default audio device switched * expose devices update handler * fix ios compile * fix bug: don't always recreate RTCAudioDeviceModule * handle guid. Co-authored-by: Hiroshi Horie <[email protected]>

allow listen-only mode in AudioUnit, adjust when category changes release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) * use `AVAudioSession` defaults * remove isRecordingEnabled feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) * feat: add audio device changes detect for windows. * Update audio_device_core_win.cc fix iOS/macOS/Android compile. fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * progress * tweak * clean * simplify audio unit restart call to SetupAudioBuffersForActiveAudioSession() might not be needed since sample rate won't change during restart. This might help reduce the unwanted noise when restarting audio unit. * clean Stop recording on mute (turn off mic indicator) (#55) * initial impl * more comments * more comment * adjust indent * comments Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * first attempt * remove unused dep * init playout / recording * use AudioDeviceID as guid * switch device method * equality * default device * `isDefault` property * dont format default device name * type param * bypass * refactor * fix * append Audio to thread labels * ref * lk headers * low level apis * fix thread checks Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail. * switch to default device when removed * close mixerManager if didn't switch to default device * default audio device switched * expose devices update handler * fix ios compile * fix bug: don't always recreate RTCAudioDeviceModule * handle guid. Co-authored-by: Hiroshi Horie <[email protected]>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]> Co-authored-by: davidliu <[email protected]>

allow listen-only mode in AudioUnit, adjust when category changes (#2) release mic when category changes (#5) Change defaults to iOS defaults (#7) Sync audio session config (#8) feat: support bypass voice processing for iOS. (#15) Remove MacBookPro audio pan right code (#22) fix: Fix can't open mic alone when built-in AEC is enabled. (#29) feat: add audio device changes detect for windows. (#41) fix Linux compile (#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) Stop recording on mute (turn off mic indicator) (#55) Cherry pick audio selection from m97 release (#35) [Mac] Allow audio device selection (#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) Allow custom audio processing by exposing AudioProcessingModule (#85) Expose audio sample buffers for Android (#89) feat: add external audio processor for android. (#103) android: make audio output attributes modifiable (#118) Fix external audio processor sample rate calculation (#108) Expose remote audio sample buffers on RTCAudioTrack (#84) Fix memory leak when creating audio CMSampleBuffer #86 Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]> Co-authored-by: davidliu <[email protected]>

Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization 7454824 * allow listen-only mode in AudioUnit, adjust when category changes (#2) * release mic when category changes (#5) * Change defaults to iOS defaults (#7) * Sync audio session config (#8) * feat: support bypass voice processing for iOS. (#15) * Remove MacBookPro audio pan right code (#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (#29) * feat: add audio device changes detect for windows. (#41) * fix Linux compile (#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * Stop recording on mute (turn off mic indicator) (#55) * Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) * Allow custom audio processing by exposing AudioProcessingModule (#85) * Expose audio sample buffers for Android (#89) * feat: add external audio processor for android. (#103) * android: make audio output attributes modifiable (#118) * Fix external audio processor sample rate calculation (#108) * Expose remote audio sample buffers on RTCAudioTrack (#84) * Fix memory leak when creating audio CMSampleBuffer #86 ## 3. Simulcast/SVC support for iOS/Android. b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. 9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. 841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]> Co-authored-by: davidliu <[email protected]> Co-authored-by: Angelika Serwa <[email protected]> Co-authored-by: Théo Monnom <[email protected]>

allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk#2) release mic when category changes (webrtc-sdk#5) Change defaults to iOS defaults (webrtc-sdk#7) Sync audio session config (webrtc-sdk#8) feat: support bypass voice processing for iOS. (webrtc-sdk#15) Remove MacBookPro audio pan right code (webrtc-sdk#22) fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk#29) feat: add audio device changes detect for windows. (webrtc-sdk#41) fix Linux compile (webrtc-sdk#47) AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk#52) Stop recording on mute (turn off mic indicator) (webrtc-sdk#55) Cherry pick audio selection from m97 release (webrtc-sdk#35) [Mac] Allow audio device selection (webrtc-sdk#21) RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk#80) Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]>

Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]> Co-authored-by: davidliu <[email protected]> Co-authored-by: Angelika Serwa <[email protected]> Co-authored-by: Théo Monnom <[email protected]> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm

* Update to m125. (#119) Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md webrtc-sdk/webrtc@b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization webrtc-sdk/webrtc@7454824 * allow listen-only mode in AudioUnit, adjust when category changes (webrtc-sdk/webrtc#2) * release mic when category changes (webrtc-sdk/webrtc#5) * Change defaults to iOS defaults (webrtc-sdk/webrtc#7) * Sync audio session config (webrtc-sdk/webrtc#8) * feat: support bypass voice processing for iOS. (webrtc-sdk/webrtc#15) * Remove MacBookPro audio pan right code (webrtc-sdk/webrtc#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (webrtc-sdk/webrtc#29) * feat: add audio device changes detect for windows. (webrtc-sdk/webrtc#41) * fix Linux compile (webrtc-sdk/webrtc#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (webrtc-sdk/webrtc#52) * Stop recording on mute (turn off mic indicator) (webrtc-sdk/webrtc#55) * Cherry pick audio selection from m97 release (webrtc-sdk/webrtc#35) * [Mac] Allow audio device selection (webrtc-sdk/webrtc#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (webrtc-sdk/webrtc#80) * Allow custom audio processing by exposing AudioProcessingModule (webrtc-sdk/webrtc#85) * Expose audio sample buffers for Android (webrtc-sdk/webrtc#89) * feat: add external audio processor for android. (webrtc-sdk/webrtc#103) * android: make audio output attributes modifiable (webrtc-sdk/webrtc#118) * Fix external audio processor sample rate calculation (webrtc-sdk/webrtc#108) * Expose remote audio sample buffers on RTCAudioTrack (webrtc-sdk/webrtc#84) * Fix memory leak when creating audio CMSampleBuffer webrtc-sdk/webrtc#86 ## 3. Simulcast/SVC support for iOS/Android. webrtc-sdk/webrtc@b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. webrtc-sdk/webrtc@9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements webrtc-sdk/webrtc@a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. webrtc-sdk/webrtc@841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. webrtc-sdk/webrtc@fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. webrtc-sdk/webrtc@eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: David Zhao <[email protected]> Co-authored-by: davidliu <[email protected]> Co-authored-by: Angelika Serwa <[email protected]> Co-authored-by: Théo Monnom <[email protected]> # Conflicts: # README.md # media/engine/webrtc_video_engine.cc # media/engine/webrtc_video_engine.h # modules/audio_device/audio_device_impl.cc # sdk/BUILD.gn # sdk/android/BUILD.gn # sdk/android/api/org/webrtc/RtpParameters.java # sdk/android/api/org/webrtc/SimulcastVideoEncoder.java # sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java # sdk/android/api/org/webrtc/VideoCodecInfo.java # sdk/android/src/jni/pc/rtp_parameters.cc # sdk/android/src/jni/simulcast_video_encoder.cc # sdk/android/src/jni/simulcast_video_encoder.h # sdk/android/src/jni/video_codec_info.cc # sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.h # sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm # sdk/objc/api/peerconnection/RTCAudioTrack.mm # sdk/objc/api/peerconnection/RTCIODevice+Private.h # sdk/objc/api/peerconnection/RTCIODevice.mm # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h # sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h # sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm # sdk/objc/base/RTCAudioRenderer.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h # sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm * fix: duplicate simulcast entries * remove duplicate declaration * remove duplicate audioDeviceModule * fix: removed livekit's external audio processor * fix: add back simulcast factories * Fix missing RTC_OBJC_TYPE macros * Fix missing headers and Metal linking # Conflicts: # sdk/BUILD.gn * Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126) * Fix set frame transformer (#125) * Fix webrtc_voice_engine not notifying mute change (#128) Looks like this line was missed during the m125 update. webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289 Adding it back in so that mic is properly released when muted. # Conflicts: # media/engine/webrtc_voice_engine.cc * android: Allow for skipping checking the audio playstate if needed (#129) Pausing/stopping the audio track can lead to a race condition against the AudioTrackThread due to this assert. Normally this is fine since directly pausing/stopping isn't possible, but user is using reflection to workaround another audio issue (muted participants still have a sending audio stream which keeps the audio alive, affecting global sound if in the background). Not a full fix, as would like to manually control the audio track directly (needs a bigger fix to handle proper synchronization before allowing public access), but this will work through reflection (user takes responsibility for usage). * Allow to pass in capture session to RTCCameraVideoCapturer (#132) Expose initializers to pass in capture session to RTCCameraVideoCapturer so we can use AVCaptureMultiCamSession etc to capture front and back simultaneously for iOS. * Fix NetworkMonitor race condition when dispatching native observers (#135) There is a race condition in NetworkMonitor where native observers may be removed concurrently with a notification being dispatched, leading to a dangling pointer dereference (trying to dispatch an observer that was already removed and destroyed), and from there a crash with access violation. By ensuring dispatching to native observers is done within the synchronization lock that guards additions/removals of native observers protects against this race condition. Since native observers callbacks are posted to the networking thread in the C++ side anyway, there should be no risk of deadlock/starvation due to long-running observers. Bug: webrtc:15837 Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643 Reviewed-by: Taylor Brandstetter <[email protected]> Commit-Queue: Harald Alvestrand <[email protected]> Reviewed-by: Harald Alvestrand <[email protected]> Cr-Commit-Position: refs/heads/main@{#42256} Co-authored-by: Guy Hershenbaum <[email protected]> * Support for Vision Pro (#131) TODO: - [x] fix compile for RTCCameraVideoCapturer - [ ] fix RTCMTLRenderer ? --------- Co-authored-by: Hiroshi Horie <[email protected]> * Multicam support (#137) TODO: - [x] Return `.systemPreferredCamera` for devices (visionOS only). - [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is true. - [x] Silence statusBarOrientation warning. --------- Co-authored-by: [email protected] <[email protected]> * tvOS support (#139) 17.0+ only atm --------- Co-authored-by: cloudwebrtc <[email protected]> * Add isDisposed to MediaStreamTrack (#140) * chore: handle invalid cipher from key size. (#142) * Allow software AEC for Simulator (#143) ~Allow to use "googEchoCancellation" constraint for software AEC. For devices "googEchoCancellation" should be false to use VoiceProcessingIO.~ * Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144) * fix: Fix bug for bypass voice processing. (#147) * chore: remove aes cbc for framecryptor. (#145) * Change audio renderer output format (#149) Instead of converting to Float, output original Int data without conversion. Output the raw format and convert when required. * Fixed issue with missing network interfaces on iOS (#151) Related issue: webrtc-sdk/webrtc#148 Cherry-pick : https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f Fixed issue with network interfaces due to a missing return value in the "nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18, RTCNetworkMonitor::initWithObserver will only enumerate the first interface, instead of all device interfaces Bug: webrtc:359245764 Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541 Auto-Submit: Corby <[email protected]> Commit-Queue: Jonas Oreland <[email protected]> Reviewed-by: Kári Helgason <[email protected]> Reviewed-by: Jonas Oreland <[email protected]> Cr-Commit-Position: refs/heads/main@{#42818} Co-authored-by: Corby Hoback <[email protected]> * Custom audio input for Android (#154) # Conflicts: # sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java # sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java --------- Co-authored-by: CloudWebRTC <[email protected]> Co-authored-by: Hiroshi Horie <[email protected]> Co-authored-by: davidliu <[email protected]> Co-authored-by: Guy Hershenbaum <[email protected]> Co-authored-by: Corby Hoback <[email protected]>

hiroshihorie added 8 commits March 11, 2022 03:24

first attempt

3c17eff

remove unused dep

42029a7

init playout / recording

4c929eb

use AudioDeviceID as guid

efc3c32

switch device method

a5042dc

equality

dfd4ace

default device

eda19e1

isDefault property

507913c

hiroshihorie changed the title ~~WIP: Expose audio device module~~ [Mac] Allow device selection Mar 12, 2022

hiroshihorie added 3 commits March 13, 2022 03:02

dont format default device name

c08e4cf

type param

e063d93

bypass

aba2227

hiroshihorie changed the title ~~[Mac] Allow device selection~~ [Mac] Allow audio device selection Mar 12, 2022

hiroshihorie requested a review from davidzhao March 13, 2022 09:18

refactor

149c99f

davidzhao approved these changes Mar 14, 2022

View reviewed changes

hiroshihorie added 3 commits March 14, 2022 17:39

fix

a00a74c

append Audio to thread labels

e5410c8

ref

c4f1305

hiroshihorie added 4 commits March 15, 2022 04:17

lk headers

a35ce56

low level apis

bdf5e7e

Merge branch 'm97_release' into expose-audio-device-module

9de0e4d

fix thread checks

bda204c

Some methods of ADM needs to be run on worker thread, otherwise RTC's thread check will fail.

hiroshihorie changed the base branch from m104_release to m97_release July 27, 2022 13:14

cloudwebrtc mentioned this pull request Jul 29, 2022

[WIP] Add selectAudioOutput method. flutter-webrtc/flutter-webrtc#1030

Merged

13 tasks

hiroshihorie merged commit f48bd24 into m97_release Jul 31, 2022

hiroshihorie deleted the expose-audio-device-module branch July 31, 2022 21:39

cloudwebrtc mentioned this pull request Jun 12, 2023

update to m114 #76

Merged

cloudwebrtc mentioned this pull request Jul 7, 2023

Migrate to m114 livekit/client-sdk-swift#225

Merged

cloudwebrtc mentioned this pull request May 21, 2024

Update to m125. #119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mac] Allow audio device selection #21

[Mac] Allow audio device selection #21

hiroshihorie commented Mar 10, 2022 •

edited

Loading

davidzhao left a comment

davidzhao Mar 14, 2022

hiroshihorie Mar 14, 2022 •

edited

Loading

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented May 27, 2022

hiroshihorie commented May 29, 2022 •

edited

Loading

hiroshihorie commented Jul 27, 2022

cloudwebrtc commented Jul 27, 2022

davidzhao commented Jul 28, 2022

[Mac] Allow audio device selection #21

[Mac] Allow audio device selection #21

Conversation

hiroshihorie commented Mar 10, 2022 • edited Loading

Goals

davidzhao left a comment

Choose a reason for hiding this comment

davidzhao Mar 14, 2022

Choose a reason for hiding this comment

hiroshihorie Mar 14, 2022 • edited Loading

Choose a reason for hiding this comment

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented Mar 14, 2022

davidzhao commented Mar 14, 2022

davidzhao commented Mar 14, 2022

hiroshihorie commented May 27, 2022

hiroshihorie commented May 29, 2022 • edited Loading

Additional required logic

Active device (lost) disconnected ✅

OS's default device changed ✅

hiroshihorie commented Jul 27, 2022

cloudwebrtc commented Jul 27, 2022

davidzhao commented Jul 28, 2022

hiroshihorie commented Mar 10, 2022 •

edited

Loading

hiroshihorie Mar 14, 2022 •

edited

Loading

hiroshihorie commented May 29, 2022 •

edited

Loading