asr-python-advanced-finetune-am-conformer-ctc-tao-finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/notebooks/dlsw-notebooks/riva_asr_asr-python-advanced-finetune-am-conformer-ctc-tao-finetuning/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# How to Fine-Tune a Riva ASR Acoustic Model (Conformer-CTC) with TAO Toolkit\n",
    "This tutorial walks you through how to fine-tune an NVIDIA Riva ASR acoustic model (Conformer-CTC) with NVIDIA TAO Toolkit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NVIDIA Riva Overview\n",
    "\n",
    "NVIDIA Riva is a GPU-accelerated SDK for building speech AI applications that are customized for your use case and deliver real-time performance. <br/>\n",
    "Riva offers a rich set of speech and natural language understanding services such as:\n",
    "\n",
    "- Automated speech recognition (ASR). \n",
    "- Text-to-Speech synthesis (TTS). \n",
    "- A collection of natural language processing (NLP) services, such as named entity recognition (NER), punctuation, and intent classification.\n",
    "\n",
    "In this tutorial, we will fine-tune a Riva ASR acoustic model (Conformer-CTC) with TAO Toolkit. <br> \n",
    "To understand the basics of Riva ASR APIs, refer to [Getting started with Riva ASR in Python](https://github.com/nvidia-riva/tutorials/blob/stable/asr-python-basics.ipynb). <br>\n",
    "\n",
    "For more information about Riva, refer to the [Riva developer documentation](https://developer.nvidia.com/riva)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Adapt Optimize (TAO) Toolkit\n",
    "Train Adapt Optimize (TAO) Toolkit is a Python-based AI toolkit for taking purpose-built pre-trained AI models and customizing them with your own data. Developers, researchers, and software partners building intelligent vision AI applications and services, can bring their own data to fine-tune pre-trained models instead of going through the hassle of training the models from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Train Adapt Optimize (TAO) Toolkit](https://developer.nvidia.com/sites/default/files/akamai/embedded-transfer-learning-toolkit-software-stack-1200x670px.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Transfer learning extracts learned features from an existing neural network into a new one. Transfer learning is often used when creating a large training dataset is not feasible. The goal of this toolkit is to reduce that 80 hour workload to an 8 hour workload, which can enable data scientists to have considerably more train-test iterations in the same time frame.\n",
    "\n",
    "Let's see this in action with a use case for the ASR acoustic model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic Speech Recognition (ASR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automatic Speech Recognition (ASR) is often the first step in building a conversational AI model. An ASR model converts audible speech into text. The main metric for these models is to reduce Word Error Rate (WER) while transcribing the text. Simply put, the goal is to take an audio file and transcribe it.\n",
    "\n",
    "In this tutorial, we are going to discuss the Conformer-CTC model, which is an end-to-end ASR model that takes in audio and produces text.\n",
    "\n",
    "Conformer-CTC supports both character-level and sub-word-level encodings. It employs a combination of self-attention and convolution modules to achieve the best of the two approaches. The self-attention layers support both absolute and relative positional encodings and can learn the global interaction, while the convolutions efficiently capture the local correlations.\n",
    "\n",
    "![Conformer-CTC](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/_images/conformer_ctc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## ASR using TAO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing and Setting up TAO\n",
    "\n",
    "Install TAO inside a Python virtual environment. We recommend performing this step first and then launching the tutorial from the virtual environment.\n",
    "\n",
    "In addition to installing the TAO Python package, ensure you meet the following software requirements:\n",
    "\n",
    "1. `python` 3.6.9\n",
    "2. `docker-ce` > 19.03.5\n",
    "3. `docker-API` 1.40\n",
    "4. `nvidia-container-toolkit` > 1.3.0\n",
    "5. `nvidia-container-runtime` > 3.4.0\n",
    "6. `nvidia-docker2` > 2.5.0\n",
    "7. `nvidia-driver` >= 455.23"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installing TAO is a simple `pip` install."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install nvidia-pyindex\n",
    "! pip install nvidia-tao"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After installing TAO, the next step is to setup the mounts for TAO. The TAO launcher uses Docker containers under the hood, and **for our data and results directory to be visible to Docker, they need to be mapped**. The launcher can be configured using the config file `~/.tao_mounts.json`. Apart from the mounts, you can also configure additional options like the environment variables and the amount of shared memory available to the TAO launcher. <br>\n",
    "\n",
    "`Important:` The following code creates a sample `~/.tao_mounts.json`  file. Here, we can map directories in which we save the data, specs, results, and cache. You should configure it for your specific use case so these directories are correctly visible to the Docker container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# please define these paths on your local host machine\n",
    "%env HOST_DATA_DIR=/path/to/your/host/data\n",
    "%env HOST_SPECS_DIR=/path/to/your/host/specs\n",
    "%env HOST_RESULTS_DIR=/path/to/your/host/results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p $HOST_DATA_DIR\n",
    "! mkdir -p $HOST_SPECS_DIR\n",
    "! mkdir -p $HOST_RESULTS_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mapping the Local Directories to the TAO Docker.\n",
    "import json\n",
    "import os\n",
    "mounts_file = os.path.expanduser(\"~/.tao_mounts.json\")\n",
    "tlt_configs = {\n",
    "   \"Mounts\":[\n",
    "       {\n",
    "           \"source\": os.environ[\"HOST_DATA_DIR\"],\n",
    "           \"destination\": \"/data\"\n",
    "       },\n",
    "       {\n",
    "           \"source\": os.environ[\"HOST_SPECS_DIR\"],\n",
    "           \"destination\": \"/specs\"\n",
    "       },\n",
    "       {\n",
    "           \"source\": os.environ[\"HOST_RESULTS_DIR\"],\n",
    "           \"destination\": \"/results\"\n",
    "       },\n",
    "       {\n",
    "           \"source\": os.path.expanduser(\"~/.cache\"),\n",
    "           \"destination\": \"/root/.cache\"\n",
    "       }\n",
    "   ],\n",
    "   \"DockerOptions\": {\n",
    "        \"shm_size\": \"16G\", \n",
    "        \"ulimits\": {\n",
    "            \"memlock\": -1,\n",
    "            \"stack\": 67108864\n",
    "         }\n",
    "   }\n",
    "}\n",
    "# Writing the mounts file.\n",
    "with open(mounts_file, \"w\") as mfile:\n",
    "    json.dump(tlt_configs, mfile, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat ~/.tao_mounts.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the Docker image versions and the tasks that it performs. You can also check by issuing `tao --help` or:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tao info --verbose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Relevant Paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: The following paths are set from the perspective of the TAO Docker.\n",
    "\n",
    "# The data is saved here:\n",
    "DATA_DIR = \"/data\"\n",
    "SPECS_DIR = \"/specs\"\n",
    "RESULTS_DIR = \"/results\"\n",
    "\n",
    "# Set your encryption key and use the same key for all commands.\n",
    "KEY = 'tlt_encode'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command structure for the TAO interface can be broken down as follows: `tao <task name> <subcommand>` <br> \n",
    "\n",
    "Let's see this in further detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Downloading Specs\n",
    "TAO's conversational AI toolkit works off of spec files which make it easy to edit hyperparameters on the fly. We can proceed to downloading the spec files. You may choose to modify/rewrite these specs or even individually override them through the launcher. You can download the default spec files by using the `download_specs` command.<br>\n",
    "\n",
    "The `-o` argument indicates the folder where the default specification files will be downloaded. The `-r` argument instructs the script on where to save the logs. **Ensure the `-o` points to an empty folder.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete the conformer specs directory if it already exists to avoid errors\n",
    "! tao speech_to_text_conformer download_specs \\\n",
    "    -r $RESULTS_DIR/conformer \\\n",
    "    -o $SPECS_DIR/conformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will use the popular AN4 dataset. Let's download it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget https://dldata-public.s3.us-east-2.amazonaws.com/an4_sphere.tar.gz  # for the original source, please visit http://www.speech.cs.cmu.edu/databases/an4/an4_sphere.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After downloading, untar the dataset and move it to the correct directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tar -xvf an4_sphere.tar.gz \n",
    "! mv an4 $HOST_DATA_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step converts the `.mp3` files into `.wav` files and splits the data into training and testing sets. It also generates a \"meta-data\" file to be consumed by the data-loader for training and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tao speech_to_text_conformer dataset_convert \\\n",
    "    -e $SPECS_DIR/conformer/dataset_convert_an4.yaml \\\n",
    "    -r $RESULTS_DIR/conformer/dataset_convert \\\n",
    "    source_data_dir=$DATA_DIR/an4 \\\n",
    "    target_data_dir=$DATA_DIR/an4_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's listen to a sample audio file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change path of the file here\n",
    "import os\n",
    "import IPython.display as ipd\n",
    "path = os.environ[\"HOST_DATA_DIR\"] + '/an4_converted/wavs/an268-mbmg-b.wav'\n",
    "ipd.Audio(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training commands for Conformer-CTC is similar to those of QuartzNet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can do the actual training, we need to pre-process the text. This step is called subword tokenization that creates a subword vocabulary for the text. This is different from Jasper/QuartzNet because only single characters are regarded as elements in the vocabulary in their cases, while in Conformer-CTC, the subword can be one or multiple characters. We can use the `create_tokenizer` command to create the tokenizer that generates the subword vocabulary for us for use in training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer create_tokenizer \\\n",
    "-e $SPECS_DIR/conformer/create_tokenizer.yaml \\\n",
    "-r $RESULTS_DIR/conformer/create_tokenizer \\\n",
    "manifests=$DATA_DIR/an4_converted/train_manifest.json \\\n",
    "output_root=$DATA_DIR/an4 \\\n",
    "vocab_size=32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TAO interface enables you to configure the training parameters from the command-line interface. <br>\n",
    "\n",
    "The process of opening the training script, finding the parameters of interest (which might be spread across multiple files), and making the changes needed, is being replaced by a simple command-line interface.\n",
    "\n",
    "For example, if the number of epochs are needed to be modified along with a change in the learning rate, you can add `trainer.max_epochs=10` and `optim.lr=0.02` and train the model. Sample commands are given below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>A list of some of the customizable parameters along with their default values is as follows:</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`trainer:`\n",
    "  \n",
    "* `gpus: 1` \n",
    "* `num_nodes: 1` \n",
    "* `max_epochs: 5` \n",
    "* `max_steps: null` \n",
    "* `checkpoint_callback: false` \n",
    "\n",
    "\n",
    "`training_ds:`\n",
    "  \n",
    "* `sample_rate: 16000` \n",
    "* `batch_size: 32` \n",
    "* `trim_silence: true` \n",
    "* `max_duration: 16.7` \n",
    "* `shuffle: true` \n",
    "* `is_tarred: false` \n",
    "* `tarred_audio_filepaths: null` \n",
    "  \n",
    "\n",
    "`validation_ds:`\n",
    "  \n",
    "* `sample_rate: 16000` \n",
    "* `batch_size: 32` \n",
    "* `shuffle: false` \n",
    "  \n",
    "`optim:`\n",
    "\n",
    "* `name: adam` \n",
    "* `lr: 0.1` \n",
    "* `betas: [0.9, 0.999]` \n",
    "* `weight_decay: 0.0001` \n",
    "\n",
    "The following steps may take a considerable amount of time depending on the GPU being used. For the best experience, we recommend using an A100 GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training a Conformer-CTC ASR model in TAO, we use the `tao speech_to_text_conformer train` command with the following arguments:\n",
    "* `-e`: Path to the spec file \n",
    "* `-g`: Number of GPUs to use \n",
    "* `-r`: Path to the results folder \n",
    "* `-m`: Path to the model \n",
    "* `-k`: User specified encryption key to use while saving/loading the model \n",
    "* Any overrides to the spec file. For example, `trainer.max_epochs`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Conformer-CTC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To fully train Conformer-CTC from scratch, you'll need to increase trainer.max_epochs from 1\n",
    "# Empirical evidence suggests that 100-200 epochs should suffice\n",
    "!tao speech_to_text_conformer train \\\n",
    "     -e $SPECS_DIR/conformer/train_conformer_bpe_large.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -r $RESULTS_DIR/conformer/train \\\n",
    "     training_ds.manifest_filepath=$DATA_DIR/an4_converted/train_manifest.json \\\n",
    "     validation_ds.manifest_filepath=$DATA_DIR/an4_converted/test_manifest.json \\\n",
    "     trainer.max_epochs=1 \\\n",
    "     training_ds.num_workers=4 \\\n",
    "     validation_ds.num_workers=4 \\\n",
    "     model.tokenizer.dir=$DATA_DIR/an4/tokenizer_spe_unigram_v32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ASR Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a model trained, we need to check how well it performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer evaluate \\\n",
    "     -e $SPECS_DIR/conformer/evaluate.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -m $RESULTS_DIR/conformer/train/checkpoints/trained-model.tlt \\\n",
    "     -r $RESULTS_DIR/conformer/evaluate \\\n",
    "     test_ds.manifest_filepath=$DATA_DIR/an4_converted/test_manifest.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ASR Fine-Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the model is trained, evaluated, and there is a need for fine-tuning, the following command can be used to fine-tune the ASR model. This step can also be used for transfer learning by making changes in the `train.json` and `dev.json` files to add new data.\n",
    "\n",
    "The list for customizations is the same as the training parameters with the exception for parameters which affect the model architecture. Also, instead of `training_ds` we have `finetuning_ds`.\n",
    "\n",
    "Note: If you want to proceed with a trained dataset for better inference results, you can find a `.nemo` model [here](\n",
    "https://ngc.nvidia.com/catalog/collections/nvidia:nemotrainingframework).\n",
    "\n",
    "Simply rename the `.nemo` file to `.tlt` and pass it through the fine-tune pipeline.\n",
    "\n",
    "Note: The fine-tune spec files contain specifics to fine-tune the English model we just trained to Russian. If you want to proceed with English, ensure the changes are in the spec file `finetune.yaml` which you can find in the `SPEC_DIR` folder you mapped. Ensure to delete older fine-tuning checkpoints if you choose to change the language after fine-tuning it as-is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer finetune \\\n",
    "     -e $SPECS_DIR/conformer/finetune.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -m $RESULTS_DIR/conformer/train/checkpoints/trained-model.tlt \\\n",
    "     -r $RESULTS_DIR/conformer/finetune \\\n",
    "     finetuning_ds.manifest_filepath=$DATA_DIR/an4_converted/train_manifest.json \\\n",
    "     validation_ds.manifest_filepath=$DATA_DIR/an4_converted/test_manifest.json \\\n",
    "     trainer.max_epochs=1 \\\n",
    "     finetuning_ds.num_workers=20 \\\n",
    "     validation_ds.num_workers=20 \\\n",
    "     trainer.gpus=1 \\\n",
    "     tokenizer.dir=$DATA_DIR/an4/tokenizer_spe_unigram_v32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ASR Model Export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With TAO, you can also export your model in a format that can be deployed using NVIDIA Riva; a highly performant application framework for multi-modal conversational AI services using GPUs. The same command for exporting to ONNX can be used here. The only small variation is the configuration for `export_format` in the spec file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Export to Riva"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer export \\\n",
    "     -e $SPECS_DIR/conformer/export.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -m $RESULTS_DIR/conformer/train/checkpoints/trained-model.tlt \\\n",
    "     -r $RESULTS_DIR/conformer/riva \\\n",
    "     export_format=RIVA \\\n",
    "     export_to=asr-model.riva"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Export to ONNX\n",
    "Note: Export to ONNX is not needed for Riva."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer export \\\n",
    "     -e $SPECS_DIR/conformer/export.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -m $RESULTS_DIR/conformer/train/checkpoints/trained-model.tlt \\\n",
    "     -r $RESULTS_DIR/conformer/export \\\n",
    "     export_format=ONNX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ASR Inference using TLT Checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ASR Inference with TAO Toolkit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we are going to run inference on the TLT checkpoint with TAO Toolkit. \n",
    " For real-time inference and best latency, we need to deploy this model on Riva. Refer to the [How to Deploy a Custom Acoustic Model (Conformer-CTC) Trained with TAO Toolkit on Riva](https://github.com/nvidia-riva/tutorials/blob/stable/asr-python-advanced-finetune-am-conformer-ctc-tao-deployment.ipynb) tutorial. \n",
    " You might have to work with the `infer.yaml` file to select the files you want for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer infer \\\n",
    "     -e $SPECS_DIR/conformer/infer.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -m $RESULTS_DIR/conformer/train/checkpoints/trained-model.tlt \\\n",
    "     -r $RESULTS_DIR/conformer/infer \\\n",
    "     file_paths=[$DATA_DIR/an4_converted/wavs/an268-mbmg-b.wav]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ASR Inference using ONNX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TAO provides the capability to use the exported `.eonnx` model for inference. The command `tao speech_to_text infer_onnx` is very similar to the inference command for `.tlt` models. Again, the inputs in the spec file used is just for demo purposes, you may choose to try out your custom input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tao speech_to_text_conformer infer_onnx \\\n",
    "     -e $SPECS_DIR/conformer/infer_onnx_conformer.yaml \\\n",
    "     -g 1 \\\n",
    "     -k $KEY \\\n",
    "     -m $RESULTS_DIR/conformer/export/exported-model.eonnx \\\n",
    "     -r $RESULTS_DIR/conformer/infer_onnx \\\n",
    "     file_paths=[$DATA_DIR/an4_converted/wavs/an268-mbmg-b.wav]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What's Next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use TAO to build custom models for your own applications, or you could [deploy the custom model to NVIDIA Riva](https://github.com/nvidia-riva/tutorials/blob/stable/asr-python-advanced-finetune-am-conformer-ctc-tao-deployment.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv-riva-tutorials",
   "language": "python",
   "name": "venv-riva-tutorials"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}