asr-python-advanced-wordboosting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2bf1191b",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/notebooks/dlsw-notebooks/riva_asr_asr-python-advanced-wordboosting/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# How do I boost specific words at runtime with word boosting?\n",
    "\n",
    "This tutorial walks you through some of the advanced features for customization of Riva Speech Skills ASR Services at runtime with word boosting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcfba84d",
   "metadata": {},
   "source": [
    "## NVIDIA Riva Overview\n",
    "\n",
    "NVIDIA Riva is a GPU-accelerated SDK for building Speech AI applications that are customized for your use case and deliver real-time performance. <br/>\n",
    "Riva offers a rich set of speech and natural language understanding services such as:\n",
    "\n",
    "- Automated speech recognition (ASR)\n",
    "- Text-to-Speech synthesis (TTS)\n",
    "- A collection of natural language processing (NLP) services, such as named entity recognition (NER), punctuation, intent classification.\n",
    "\n",
    "In this tutorial, we will customize Riva ASR to boost specific words at runtime with word boosting. <br> \n",
    "To understand the basics of Riva ASR APIs, refer to [Getting started with Riva ASR in Python](https://github.com/nvidia-riva/tutorials/tree/stable/asr-python-basics.ipynb). <br>\n",
    "\n",
    "For more information about Riva, refer to the [Riva developer documentation](https://developer.nvidia.com/riva)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27f37136",
   "metadata": {},
   "source": [
    "## Word boosting with Riva ASR APIs\n",
    "\n",
    "Word boosting is one of the customizations Riva offers. It allows you to bias the ASR engine to recognize particular words of interest at request time by giving them a higher score when decoding the output of the acoustic model.  \n",
    "\n",
    "Now, let's use word boosting with Riva APIs for some sample audio clips with an OOTB (out-of-the-box) English pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d75d4887",
   "metadata": {},
   "source": [
    "#### Requirements and setup\n",
    "\n",
    "1. Start the Riva Speech Skills server.  \n",
    "Follow the instructions in the [Riva Skills Quick Start Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html#) to deploy OOTB ASR models on the Riva Speech Skills server before running this tutorial. By default, only the English models are deployed.\n",
    "\n",
    "\n",
    "2. Install the Riva Client library.  \n",
    "Follow the steps in the [Requirements and setup for the Riva Client](https://github.com/nvidia-riva/tutorials#running-the-riva-client) to install the Riva Client library."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41f7589e",
   "metadata": {},
   "source": [
    "#### Import the Riva client libraries\n",
    "\n",
    "Let's import some of the required libraries, including the Riva Client libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73626be6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import IPython.display as ipd\n",
    "import grpc\n",
    "\n",
    "import riva.client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0ab5f9f",
   "metadata": {},
   "source": [
    "#### Create a Riva client and connect to the Riva Speech API server\n",
    "\n",
    "The following URI assumes a local deployment of the Riva Speech API server is on the default port. In case the server deployment is on a different host or via a Helm chart on Kubernetes, use an appropriate URI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "244caad0",
   "metadata": {},
   "outputs": [],
   "source": [
    "auth = riva.client.Auth(uri='localhost:50051')\n",
    "\n",
    "riva_asr = riva.client.ASRService(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24cd7f2b",
   "metadata": {},
   "source": [
    "#### ASR inference without word boosting\n",
    "First, let's run ASR on our sample audio clip without word boosting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02154940",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Load a sample audio file from local disk\n",
    "# This example uses a .wav file with LINEAR_PCM encoding.\n",
    "path = \"audio_samples/en-US_wordboosting_sample1.wav\"\n",
    "with io.open(path, 'rb') as fh:\n",
    "    content = fh.read()\n",
    "ipd.Audio(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22312b2b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Creating RecognitionConfig\n",
    "config = riva.client.RecognitionConfig(\n",
    "  language_code=\"en-US\",\n",
    "  max_alternatives=1,\n",
    "  enable_automatic_punctuation=True,\n",
    "  audio_channel_count = 1\n",
    ")\n",
    "\n",
    "# ASR Inference call with Recognize \n",
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript without Word Boosting:\", asr_best_transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df24b66d",
   "metadata": {},
   "source": [
    "As you can see, ASR is having a hard time recognizing domain specific terms like `AntiBERTa` and `ABlooper`. <br>\n",
    "\n",
    "Now, let's use word boosting to try to improve ASR for these domain specific terms."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e9ebb1",
   "metadata": {},
   "source": [
    "#### ASR inference with word boosting\n",
    "\n",
    "Let's look at how to add the boosted words to `RecognitionConfig` with `SpeechContext`. (For more information about `SpeechContext`, refer to the docs [here](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/reference/protos/protos.html#_CPPv413SpeechContext)). The simplest way to add word boosting is to use function\n",
    "[riva.client.add_word_boosting_to_config()](https://github.com/nvidia-riva/python-clients/blob/928c63273176a939500e01ce176c463f1606a1ff/riva_api/asr.py#L78)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70607575",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating SpeechContext for word boosting\n",
    "boosted_lm_words = [\"AntiBERTa\", \"ABlooper\"]\n",
    "boosted_lm_score = 20.0\n",
    "riva.client.add_word_boosting_to_config(config, boosted_lm_words, boosted_lm_score)\n",
    "\n",
    "# ASR Inference call with Recognize \n",
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript with Word Boosting:\", asr_best_transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2164633c",
   "metadata": {},
   "source": [
    "As you can see, with word boosting, ASR is able to correctly transcribe the domain specific terms `AntiBERTa` and `ABlooper`.\n",
    "\n",
    "_Boost Score_: The recommended range for the boost score is 20 to 100. The higher the boost score, the more biased the ASR engine is towards this word.  \n",
    "_OOV Word Boosting_: OOV words can also be word boosted; in the exact same way as in-vocabulary words, as described above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3693db3c",
   "metadata": {},
   "source": [
    "#### Boosting different words at different levels\n",
    "With Riva ASR, we can also have different boost values for different words. For example, here _AntiBERTa_ is boosted by 10 and _ABlooper_ is boosted by 20:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16a5ee2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a sample audio file from local disk\n",
    "# This example uses a .wav file with LINEAR_PCM encoding.\n",
    "path = \"audio_samples/en-US_wordboosting_sample1.wav\"\n",
    "with io.open(path, 'rb') as fh:\n",
    "    content = fh.read()\n",
    "ipd.Audio(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0a67320",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating RecognitionConfig\n",
    "config = riva.client.RecognitionConfig(\n",
    "  language_code=\"en-US\",\n",
    "  max_alternatives=1,\n",
    "  enable_automatic_punctuation=True,\n",
    "  audio_channel_count = 1\n",
    ")\n",
    "riva.client.add_word_boosting_to_config(config, [\"AntiBERTa\"], 10.)\n",
    "riva.client.add_word_boosting_to_config(config, [\"ABlooper\"], 20.)\n",
    "\n",
    "# ASR Inference call with Recognize \n",
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript with Word Boosting:\", asr_best_transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "047a29cd",
   "metadata": {},
   "source": [
    "#### Negative word boosting for undesired words\n",
    "We can even use word boosting to discourage prediction of some words, by using negative boost scores.  \n",
    "\n",
    "Let's load a sample audio file and get the transcription results from it without any word boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faf2c6af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load a sample audio file from local disk\n",
    "# This example uses a .wav file with LINEAR_PCM encoding.\n",
    "path = \"audio_samples/en-US_wordboosting_sample3.wav\"\n",
    "with io.open(path, 'rb') as fh:\n",
    "    content = fh.read()\n",
    "ipd.Audio(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f60707f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating RecognitionConfig\n",
    "config = riva.client.RecognitionConfig(\n",
    "  language_code=\"en-US\",\n",
    "  max_alternatives=1,\n",
    "  enable_automatic_punctuation=True,\n",
    "  audio_channel_count = 1\n",
    ")\n",
    "\n",
    "# ASR Inference call with Recognize \n",
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript without Word Boosting:\", asr_best_transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "008f908b-2996-465e-b0cf-9cfaae74a1ff",
   "metadata": {},
   "source": [
    "Now, let us get the transcription results with negative word boosting for the word `been`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7894ae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating RecognitionConfig\n",
    "config = riva.client.RecognitionConfig(\n",
    "  language_code=\"en-US\",\n",
    "  max_alternatives=1,\n",
    "  enable_automatic_punctuation=True,\n",
    "  audio_channel_count = 1\n",
    ")\n",
    "riva.client.add_word_boosting_to_config(config, [\"been\"], -100.)\n",
    "\n",
    "# ASR Inference call with Recognize \n",
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript with Negative Word Boosting:\", asr_best_transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25cf06ce-af72-4a33-bd01-c378994d4121",
   "metadata": {},
   "source": [
    "As you can see, the word `been` was not predicted this time. Also note that in this case, `be` was predicted in its place. In some cases, no word might be predicted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a94fbe96",
   "metadata": {},
   "source": [
    "Now let us see how we can combine negative and positive word boosting:  \n",
    "Let us combine the negative word boosting from above example, with positive word boosting to predict the word `middle` instead of `little`. \n",
    "We perform positive word boosting, as we did in earlier examples, for word `middle`. Note that we are using the same `config` instance, that we created in the above example for negative word boosting for `little`. So it already has the `SpeechContext` instance for `little`. So, now, we just need to add a `SpeechContext` instance for positive word boosting for `middle`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55605d1c-3a4e-4c05-9ed8-f285dd964ba7",
   "metadata": {},
   "source": [
    "Now, let us see how we can combine negative and positive word boosting:  \n",
    "Let us combine the negative word boosting from the above example with positive word boosting to predict the word `bin` instead of `been`. \n",
    "We perform positive word boosting, as we did in earlier examples, by adding a new `SpeechContext` instance for `bin` to the `RecognitionConfig` instance `config`. Note that we are using the same `config` instance that we created in the above example for negative word boosting for `been`. So, it already has the `SpeechContext` instance for `been`. Now, we just need to add a `SpeechContext` instance for positive word boosting for `bin`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc74e7b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating SpeechContext for Word Boosting\n",
    "positive_boosted_lm_word = \"bin\"\n",
    "positive_boosted_lm_score = 200.0\n",
    "riva.client.add_word_boosting_to_config(config, [positive_boosted_lm_word], positive_boosted_lm_score)\n",
    "\n",
    "# ASR Inference call with Recognize \n",
    "response = riva_asr.offline_recognize(content, config)\n",
    "asr_best_transcript = response.results[0].alternatives[0].transcript\n",
    "print(\"ASR Transcript with Negative and Positive Word Boosting:\", asr_best_transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01931dfe-1fc5-4cc7-bf26-05c8b1c497ac",
   "metadata": {},
   "source": [
    "As you can see, in this example, we were able to generate the transcript with `bin` instead of `been`, by combining negative and positive word boosting."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3995d8e1",
   "metadata": {},
   "source": [
    "Note:\n",
    "\n",
    "- There is no limit to the number of words that can be boosted. You should see no impact on latency for all requests, even for ~100 boosted words, except for the first request, which is expected.\n",
    "- Boosting phrases or a combination of words is not yet fully supported (but do work). We will revisit finalizing this support in an upcoming release.\n",
    "- By default, no words are boosted on the server side. Only words passed by the client are boosted.\n",
    "\n",
    "Information about word boosting can also be found in the documentation [here](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-customizing.html#word-boosting). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64c36fe7",
   "metadata": {},
   "source": [
    "## Go deeper into Riva capabilities\n",
    "\n",
    "Now that you have a basic introduction to the Riva ASR APIs, you can try:\n",
    "\n",
    "### Additional Riva tutorials\n",
    "\n",
    "Checkout more Riva ASR (and TTS) tutorials [here](https://github.com/nvidia-riva/tutorials) to understand how to use some of the advanced features of Riva ASR, including customizing ASR for your specific needs.\n",
    "\n",
    "\n",
    "### Sample applications\n",
    "\n",
    "Riva comes with various sample applications. They demonstrate how to use the APIs to build applications such as a [chatbot](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/weather.html), a domain specific speech recognition, [keyword (entity) recognition system](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/callcenter.html), or simply how Riva allows scaling out for handling massive amounts of requests at the same time. Refer to ([SpeechSquad)](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/samples/speechsquad.html) for more information.  \n",
    "Refer to the *Sample Application* section in the [Riva developer documentation](https://developer.nvidia.com/) for more information.\n",
    "\n",
    "\n",
    "###  Riva Text-To-Speech (TTS)\n",
    "\n",
    "Riva's TTS offering comes with two OOTB voices that can be used in streaming or batch inference modes. They can be easily deployed using the [Riva Quick Start scripts](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html). Follow [this link](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-overview.html) to understand Riva's TTS capabilities. Explore how to use Riva TTS APIs with the OOTB voices with [this Riva TTS tutorial](https://github.com/nvidia-riva/tutorials/blob/dev/22.04/tts-python-basics.ipynb).\n",
    "\n",
    "\n",
    "### Additional resources\n",
    "\n",
    "For more information about each of the APIs and their functionalities, refer to the [documentation](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/protobuf-api/protobuf-api-root.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "987a1914",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}