diff --git a/docs/web-sdk/introduction.md b/docs/web-sdk/introduction.md index d910174f..84dab0df 100644 --- a/docs/web-sdk/introduction.md +++ b/docs/web-sdk/introduction.md @@ -11,10 +11,10 @@ import TabItem from '@theme/TabItem'; --- :::note In Beta Phase -This feature is in the Beta phase. If you have any questions, ideas or suggestions please reach out to us at devrelations@symbl.ai. +This feature is in the Beta phase. If you have any questions, ideas or suggestions please reach out to us at devrelations@symbl.ai. ::: -The Web SDK allows you to add Symbl’s Conversation Intelligence into your JavaScript application directly into the browser. It provides a pre-defined set of classes for easy utilization of our Streaming and Subscribe APIs. +The Web SDK is a Typescript application that allows you to add Symbl’s Conversation Intelligence into your JavaScript application directly into the browser. It provides a pre-defined set of classes for easy utilization of our Streaming and Subscribe APIs. :::info The Web SDK is currently available with Symbl’s Streaming and Subscribe APIs. @@ -126,6 +126,7 @@ const symbl = Symbl({ ## Getting Started + In order to get started with the Symbl Web SDK we will start with a basic Hello World implementation ### Create a Hello World Application @@ -185,7 +186,7 @@ try { ### Subscribing to an existing connection -You can use the `subscribeToConnection` method to subscribe to an existing connection using the Subscribe API. +You can use the [`subscribeToConnection`](#subscribetoconnectionsessionid-string) method to subscribe to an existing connection using the Subscribe API. ```js @@ -225,7 +226,7 @@ try { ### Using an external AudioStream -You can use an external [AudioContext](https://developer.mozilla.org/en-US/docs/Web/API/AudioContext) using the Web SDK [AudioStream](#audio-stream) interface +You can use an external [AudioContext](https://developer.mozilla.org/en-US/docs/Web/API/AudioContext) using the Web SDK [AudioStream](#audiostream-class) interface ```js @@ -283,11 +284,78 @@ try { ``` +### Processing data from an audio file + +You can process audio from an audio file using the [`attachAudioSourceElement`](#attachaudiosourceelementaudiosourcedomelement) method on the [AudioStream](#audiostream-class) + +:::caution note +Currently, only LINEAR16 encoding is supported for audio elements. Opus support will be addressed in a later update. +::: + + +```js + +import { Symbl, LINEAR16AudioStream } from "@symblai/symbl-web-sdk"; + +try { + + // We recommend to remove appId and appSecret authentication for production applications. + // See authentication section for more details + const symbl = new Symbl({ + appId: '', + appSecret: '', + // accessToken: '' + }); + + // Create your audio element + const myAudioElement = new Audio(); + myAudioElement.type = "audio/mp3"; + myAudioElement.src = "link-to-file.mp3"; + + // Attach audio element to AudioStream + const audioStream = new LINEAR16AudioStream(); + audioStream.attachAudioSourceElement(myAudioElement); + + // Create connection and start processing audio + const connection = symbl.createConnection("abc123", audioStream); + connection.startProcessing({ + config: { + encoding: "LINEAR16" + } + }); + + // Play the element once audio is ready to be processed. + connection.on("processing_started", () => { + myAudioElement.play(); + }); + + // Retrieve real-time transcription from the conversation + connection.on("speech_recognition", (speechData) => { + const { punctuated } = speechData; + const name = speechData.user ? speechData.user.name : "User"; + console.log(`${name}: `, punctuated.transcript); + }); + + // This is just a helper method meant for testing purposes. + // Waits 60 seconds before continuing to the next API call. + await symbl.wait(60000); + + // Stops processing audio, but keeps the WebSocket connection open. + await connection.stopProcessing(); + + // Closes the WebSocket connection. + connection.disconect(); +} catch(e) { + // Handle errors here. +} + +``` + ## Configurations Reference ### Connection Configuration -The following code shows an example connection configuration for Web SDK. The connection configuration is passed into the Streaming API connection during the `startProcessing` method. +The following code shows an example connection configuration for Web SDK. The connection configuration is passed into the Streaming API connection during the [`startProcessing`](#startprocessingoptions-streamingapiconnectionconfig) method. ```js const connectionConfig = { @@ -463,7 +531,7 @@ Field | Required | Supported Value ## Events / Callbacks -Both the conection and audio stream objects have an `on` method which can be used to subscribe to events and perform callbacks. +Both the conection and audio stream objects have an [`on`](#startprocessingoptions-streamingapiconnectionconfig) method which can be used to subscribe to events and perform callbacks. ### Connection Events @@ -528,7 +596,7 @@ connection.audioStream.on("audio_source_disconnected", () => { ### Global Events -Listeners can subscribe to the following global Symbl Events using the `window` object. +Listeners can subscribe to the following global Symbl Events using the [`window`](https://developer.mozilla.org/en-US/docs/Web/API/Window) object. #### Example @@ -976,11 +1044,11 @@ symbl.init({ #### `createConnection(sessionId?: string, audioStream?: AudioStream)` -Accepts an optional sessionId and an optional instance of [AudioStream](#audio-stream). +Accepts an optional sessionId and an optional instance of [AudioStream](#audiostream-class). Validates that SessionID is unique and then opens a Symbl Streaming API WebSocket connection. -Returns an instance of [StreamingAPIConnection](#streaming-api-connection). +Returns an instance of [StreamingAPIConnection](#streamingapiconnection-class). #### Example @@ -993,11 +1061,11 @@ const connection = symbl.createConnection("abc123"); #### `createAndStartNewConnection(options: StreamingAPIConnectionConfig, audioStream?: AudioStream)` -Accepts a required [Connection Config](#connection-config) object and an optional instance of [AudioStream](#audio-stream). +Accepts a required [Connection Config](#connection-configuration) object and an optional instance of [AudioStream](#audiostream-class). Opens a new connection and starts processing audio. -Returns an instance of [StreamingAPIConnection](#streaming-api-connection). +Returns an instance of [StreamingAPIConnection](#streamingapiconnection-class). #### Example @@ -1018,7 +1086,7 @@ Accepts a required Session ID to subscribe to. Establishes a Subscribe API connection with session ID. -Returns an instance of [SubscribeAPIConnection](#subscribe-api-connection) +Returns an instance of [SubscribeAPIConnection](#subscribeapiconnection-class) #### Example @@ -1040,6 +1108,304 @@ const symbl = new Symbl(); const connection = symbl.wait(5000); ``` +### StreamingAPIConnection Class + +The `StreamingAPIConnection` class represents a Streaming API WebSocket connection. In most instances you would be interfacing with this class from the return variable on [`createConnection`](#createconnectionsessionid-string-audiostream-audiostream) or [`createAndStartNewConnection`](#createandstartnewconnectionoptions-streamingapiconnectionconfig-audiostream-audiostream). You can also import it from the Web SDK and use it separately: `import { StreamingAPIConnection } from '@symblai/symbl-web-sdk';`. `StreamingAPIConnection` inherits from the `BaseConnection` base class. + +#### `connect()` + +Will open a Symbl Streaming API WebSocket connection. If already connected will log a warning. Once successfully connected, will send out the `connected` event. + +##### Example + +```js +const connection = await symbl.createConnection(); +await connection.connect(); +``` + +--- + +#### `disconnect()` + +Disconnects from Symbl Streaming API WebSocket + +##### Example + +```js +connection.disconnect(); +``` + +--- + +#### `startProcessing(options: StreamingAPIConnectionConfig)` + +Accepts a required [Connection Config](#connection-configuration) + +Triggers the streaming connection to begin processing audio through Symbl websocket + +##### Example + +```js +connection.startProcessing({ + config: { + encoding: "OPUS" + } +}); +``` + +--- + +#### `stopProcessing()` + +Triggers the streaming connection to stop processing audio through Symbl websocket. If `disconnectOnStopRequest` is set to `false` then the WebSocket will be put into a non-processing state which can be resumed later by calling `startProcessing` again. If `disconnectOnStopRequest` is not set or set to `true` the WebSocket connection will need to be re-opened to start processing audio again. + +##### Example + +```js +connection.startProcessing({ + config: { + encoding: "OPUS" + } +}); +``` + +--- + +#### `on(eventName: EventTypes, callback: Function)` + +Subscribe to an event and perform a callback when it's fired. + +##### Example + +```js +connection.on('connected', () => { + console.log('I am connected!'); +}) +``` + +--- + +#### `getSessionId()` + +Getter for the `sessionId`. + +##### Example + +```js +const sessionId = connection.getSessionId(); +``` + +--- + +#### `isProcessing()` + +Returns true if the connection is processing audio. + +##### Example + +```js +connection.isProcessing(); +``` + +--- + +#### `isConnected()` + +Returns true if connected to the WebSocket. + +##### Example + +```js +connection.isConnected(); +``` + + +#### `updateAudioStream(audioStream: AudioStream)` + +Accepts an [AudioStream](#audiostream-class) instance. + +Replaces the existin audio stream with the one provided. Will stop processing if currently processing audio. + +##### Example + +```js +const audioStream = new OpusAudioStream(); +connection.updateAudioStream(audioStream); +``` + +### SubscribeAPIConnection Class + +The `SubscribeAPIConnection` class represents a Subscribe API WebSocket connection. In most instances you would be interfacing with this class from the return variable on [`subscribeToConnection`](#subscribetoconnectionsessionid-string). You can also import it from the Web SDK and use it separately: `import { SubscribeAPIConnection } from '@symblai/symbl-web-sdk';`. `SubscribeAPIConnection` inherits from the `BaseConnection` base class. + +#### `connect()` + +Will open a Symbl Subscribe API WebSocket connection. If already connected will log a warning. Once successfully connected, will send out the `connected` event. + +##### Example + +```js +const connection = await symbl.createConnection(); +await connection.connect(); +``` + +--- + +#### `disconnect()` + +Disconnects from Symbl Subscribe API WebSocket + +##### Example + +```js +connection.disconnect(); +``` + +--- + +#### `on(eventName: EventTypes, callback: Function)` + +Subscribe to an event and perform a callback when it's fired. + +##### Example + +```js +connection.on('connected', () => { + console.log('I am connected!'); +}) +``` + +--- + +#### `isConnected()` + +Returns true if connected to the WebSocket. + +##### Example + +```js +connection.isConnected(); +``` + +### AudioStream Class + +`LINEAR16AudioStream` and `OpusAudioStream` both inherit from the `AudioStream` base class, so their usage is identical. The main difference is `LINEAR16AudioStream` is meant to be paired with the `LINEAR16` encoding type, and the `OpusAudioStream` is meant to be paired with the `OPUS` encoding type. + +#### `attachAudioDevice(deviceId: string, mediaStream: MediaStream)` + +Accepts an optional [deviceId](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo/deviceId) and an optional [MediaStream](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream). If no `deviceId` is passed, the default device is used. + +Attaches audio device either through default browser method creating a MediaStream or via a passed in MediaStream + +##### Example + +```js +audioStream.attachAudioDevice("my-device-id"); +``` + +--- + +#### `detachAudioDevice()` + +Disconnects processor to cleanly detach the audio input device. + +##### Example + +```js +audioStream.detachAudioDevice(); +``` + +--- + +#### `updateAudioDevice(deviceId: string, mediaStream: MediaStream)` + +Accepts an optional [deviceId](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo/deviceId) and an optional [MediaStream](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream). If no `deviceId` is passed, the default device is used. + +Updates the audio device in use by the audio stream. + +##### Example + +```js +audioStream.updateAudioDevice("my-device-id"); +``` + +--- + +#### `attachAudioSourceElement(audioSourceDomElement)` + +Accepts a required [HTMLAudioElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement) or [HTMLSourceElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLSourceElement). A `type` with the Content-Type is required and the `src` attribute is also required. The `src` attribute can be a URL or a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) + +Attaches an audio element to the processor and starts processing audio data from the audio file. In order to start processing you need to call `.play()` on the audio element. We recommend doing this after the `processing_started` Event has been fired. + +:::caution note +Currently, only LINEAR16 encoding is supported for audio elements. Opus support will be addressed in a later update. +::: + + +##### Example + +```js +// Authenticate with Symbl +const symbl = new Symbl({ + accessToken: "< MY ACCESS TOKEN >" +}); + +// Create your audio element +const myAudioElement = new Audio(); +myAudioElement.type = "audio/mp3"; +myAudioElement.src = "link-to-file.mp3"; + +// Attach audio element to AudioStream +const audioStream = new LINEAR16AudioStream(); +audioStream.attachAudioSourceElement(myAudioElement); + +// Create connection and start processing audio +const connection = symbl.createConnection("abc123", audioStream); +connection.startProcessing({ + config: { + encoding: "LINEAR16" + } +}); + +// Play the element once audio is ready to be processed. +connection.on("processing_started", () => { + myAudioElement.play(); +}); +``` + +--- + +#### `detachAudioSourceElement()` + +Disconnects processor to cleanly detach the audio source element. + +##### Example + +```js +audioStream.detachSourceElement(); +``` + +--- + +#### `updateAudioSourceElement(audioSourceDomElement)` + +Accepts a required [HTMLAudioElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLAudioElement) or [HTMLSourceElement](https://developer.mozilla.org/en-US/docs/Web/API/HTMLSourceElement). A `type` with the Content-Type is required and the `src` attribute is also required. The `src` attribute can be a URL or a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) + +Updates the audio element attached to the audio stream. + +:::caution note +Currently, only LINEAR16 encoding is supported for audio elements. Opus support will be addressed in a later update. +::: + +##### Example + +```js +audioStream.updateAudioSourceElement(myAudioElement); +``` + +--- + + + ## Known Issues