Streams and Effects

• Introduction
• New Streams
• Effects
• Virtual Background
  • Camera stream and virtual background effect
    • Helper Methods
• Noise Reduction
  • Microphone stream and noise reduction effect

NOTE: This feature will be currently available only in the stream-classes (later on will be merged to next) branch of webex-js-sdk.

Introduction

Previously, we were using streams generated by the getUserMedia() and getDisplayMedia() methods to manage microphone, camera, and screen sharing functionalities.

In alignment with the Web JS SDK v3 initiative, we are introducing new APIs designed to facilitate the management of local media streams, including those for the microphone, camera, and screen sharing. This document explain how these new streams function with two newly added effects: VirtualBackgroundEffect and NoiseReductionEffect. These effects and corresponding methods are available in the plugin-meetings package. Developers can explore and play around with them in the samples app provided by the Web JS SDK here.

New Streams

We have the following new local stream classes and methods:

1. LocalCameraStream
2. LocalMicrophoneStream
3. LocalDisplayStream
4. LocalSystemAudioStream

We have the following methods to create the above local streams:

1. createCameraStream
2. createMicrophoneStream
3. createDisplayStream
4. createDisplayStreamWithAudio

These all can be imported via plugin-meetings package, for e.g.,

import { 
  LocalCameraStream,
  LocalMicrophoneStream,
  LocalDisplayStream,
  LocalSystemAudioStream,
  createCameraStream,
  createMicrophoneStream,
  createDisplayStream,
  createDisplayStreamWithAudio
 } from '@webex/plugin-meetings';

To create any of the mentioned local streams (camera, microphone, or display), simply invoke the respective method while optionally passing the required parameters, such as:

const cameraStream = await createCameraStream(cameraConstraints);

const microphoneStream = await createMicrophoneStream(audioConstraints);

const [localShareVideoStream, localShareAudioStream] = await createDisplayStreamWithAudio();

The following are optional camera and audio constraints that can be provided to generate the mentioned local streams:

const cameraConstraints = {
    deviceId?: ConstrainDOMString;
    width?: ConstrainULong;
    height?: ConstrainULong;
    aspectRatio?: ConstrainDouble;
    frameRate?: ConstrainDouble;
    facingMode?: ConstrainDOMString;
};

const audioConstraints = {
    deviceId?: string;
    autoGainControl?: boolean;
    echoCancellation?: boolean;
    noiseSuppression?: boolean;
};

Now, Let's dive into the discussion of the effects.

Effects

We have 2 new media effects created and ready for use, namely:

1. VirtualBackgroundEffect - for blur, image replacement, video replacement
2. NoiseReductionEffect - for background noise removal

Virtual Background

The virtual background can take the form of an image, an MP4 video, or the user's background with applied blur. The blur option offers flexibility with adjustable strength and quality levels, with higher settings demanding greater computational resources.

We have following API to create the effect. Access this API via the meetings object as mentioned below:

await webex.meetings.createVirtualBackgroundEffect(options);

Asynchronous	Yes
Parameters	options Name Description Values Required generator Determines where the model runs (on main thread or background thread) string local worker Optional, Defaults to worker frameRate Determines how many frames are sent to the model per second number 0-60 Optional, Defaults to 30 quality Determines the accuracy of the model (higher accuracy requires additional CPU resources) string LOW MEDIUM HIGH ULTRA Optional, Defaults to LOW mirror Whether the output image should be flipped horizontally boolean Optional, Defaults to false mode Determines the kind of background to render behind the user string BLUR IMAGE VIDEO Optional, Defaults to BLUR blurStrength How strongly the background should be blurred string WEAK MODERATE STRONG STRONGER STRONGEST Required in BLUR mode bgImageUrl Path to the background image to replace the original background string Fully qualified URL Required in IMAGE mode bgVideoUrl Path to the background video to replace the original background string Fully qualified URL (mp4 only) Required in VIDEO mode
Returns	A promise that returns virtual background effect

Camera stream and virtual background effect

Now, let's explore how to utilize the new local camera stream API and the virtual background effect to create and apply a basic blurred background to a video stream. Follow these steps:

1. To start, let's create a local camera stream, which we'll later apply the blur background effect to. Simply use the createCameraStream() API, as discussed earlier, like this:

   const cameraConstraints = { width: 640, height: 480 };
   
   const cameraStream = await createCameraStream(cameraConstraints);

2. Add the camera stream on your video srcObject in your code like,

   meetingStreamsLocalVideo.srcObject = cameraStream.outputStream

3. Then, create the virtual background effect using createVirtualBackgroundEffect() API like,

   const effect = await webex.meetings.createVirtualBackgroundEffect();   //we're not passing any optional parameters as default mode is BLUR

4. After creating the blur background effect, we will utilize the addEffect() method of the local camera stream to apply the newly created effect to the camera stream. This can be done as follows:

   const effect = await cameraStream.addEffect("blur-background", effect);   //first parameter can be any string name you want to give to your effect

5. Now that we've added the effect to the camera stream, we need to enable the effect to observe it in action on the video. This can be achieved by calling the enable method on the effect, like this:

   await effect.enable();

6. After calling enable, you will see the virtual background effect applied to your camera stream.

Helper Methods

There are some more methods available on effect and local stream which can be conveniently used by developers. Let's discuss them below,

• effect.disable() - disables the effect
• effect.dispose() - tears down the effect
• effect.setEnabled(enable) - single method to enable or disable the effect    // pass true to enable the effect and false for disable
• stream.getEffect('effectName') - get the effect added on the stream
• stream.getAllEffects(****) - get all the effects added on the stream
• stream.disposeEffects() - tears down all the effects from the stream

Noise Reduction

The noise reduction effect is designed to eliminate background noise from an audio stream, ensuring clear audio during calls. To create this effect, you can access the following API through the meetings object, as detailed below:

await webex.meetings.createNoiseReductionEffect(options);

Asynchronous	Yes
Parameters	options Name Description Values Required audioContext An optional AudioContext for custom behaviour, for e.g., const audioContext = new AudioContext({sampleRate: 48000}); object No mode Determines whether to run in WORKLET mode or LEGACY mode for older browsers string WORKLET LEGACY No, Defaults to WORKLET
Returns	A promise that returns noise reduction effect

Microphone stream and noise reduction effect

Now, let's explore how to use the new local microphone stream API along with the noise reduction effect to eliminate background noise from an audio stream. Follow this consolidated code snippet:

const microphoneStream = await createMicrophoneStream();

const effect = await webex.meetings.createNoiseReductionEffect();
               
meetingStreamsLocalAudio.srcObject = microphoneStream.outputStream;

const effect = await microphoneStream.addEffect("noise-reduction", effect);

await effect.enable();

Helper methods are same as discussed in virtual background example.

If you have any questions or need further clarification, please don't hesitate to reach out to us

Thanks for reading it out!!