tts-python-evaluation-QA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fe74c6db",
   "metadata": {},
   "source": [
    "# TTS Evaluation QA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a833017",
   "metadata": {},
   "source": [
    "In this tutorial, we will use  Automatic speech recognition(ASR) to generate transcripts from TTS synthesized data and compare the generated transcripts against the groundtruth using character error rate (CER) and word error rate (WER).\n",
    "\n",
    "These metrics are useful to find any inconsistencies between audio, transcript pair by comparing ASR generated transcripts with the ground truth transcripts.\n",
    "\n",
    "The tutorial will include:\n",
    " - Downloading 5 minutes of hifiTTS audio transcript pairs.\n",
    " - Generating transcripts for the audios using a pretrained NeMo ASR model.\n",
    " - Calculating character error rate and word error rate between ground truth transcripts, and ASR generated transcripts."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769a7496",
   "metadata": {},
   "source": [
    "### Download data\n",
    "For our tutorial, we will use a small part of the Hi-Fi Multi-Speaker English TTS (Hi-Fi TTS) dataset. You can read more about dataset [here](https://arxiv.org/abs/2104.01497). We will use speaker 6097 as the target speaker, and only a 5-minute subset of audio will be used for this evaluation example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e37ec58a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://nemo-public.s3.us-east-2.amazonaws.com/6097_5_mins.tar.gz  # Contains 10MB of data\n",
    "!tar -xzf 6097_5_mins.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da0f5b95",
   "metadata": {},
   "outputs": [],
   "source": [
    "manifest_file = \"6097_5_mins/manifest.json\"\n",
    "asr_pred = \"6097_5_mins/asr_pred.json\"\n",
    "\n",
    "\n",
    "## Fix audiopaths in manifest.json\n",
    "!sed -i 's,audio/,6097_5_mins/audio/,g' {manifest_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cff3073",
   "metadata": {},
   "source": [
    "Looking at `manifest.json`, we see a standard NeMo json that contains the filepath, text, and duration. Please make sure that  `manifest.json` contains the relative path.\n",
    "\n",
    "The manifest file should look this:\n",
    "\n",
    "    {\"audio_filepath\": \"6097_5_mins/audio/presentpictureofnsw_02_mann_0532.wav\", \"text\": \"not to stop more than ten minutes by the way\", \"duration\": 2.6, \"text_no_preprocessing\": \"not to stop more than ten minutes by the way,\", \"text_normalized\": \"not to stop more than ten minutes by the way,\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f5b4681",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Print the first line of manifest file.\n",
    "!head -n 1 {manifest_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f33cbb",
   "metadata": {},
   "source": [
    "### Synthesize text from asr."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9097cab0",
   "metadata": {},
   "source": [
    "We will need `nemo toolkit` and `transcribe_speech.py` to generate transcripts for our audio samples.\n",
    "\n",
    "\n",
    "Lets install the `nemo toolkit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f5002b0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## Clone the latest NeMo.\n",
    "!pip install nemo_toolkit['all']\n",
    "!pip install --upgrade protobuf==3.20.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71bb2723",
   "metadata": {},
   "source": [
    "Now download `transcribe_speech.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "609abb95",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/NVIDIA/NeMo/stable/examples/asr/transcribe_speech.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b5dca8",
   "metadata": {},
   "source": [
    "Transcribe audio samples using nemo and transcribe_speech.py. This will be later used to calcualte character error rate and word error rate.\n",
    "\n",
    "The model used is an English pretrained [conformer CTC ASR model](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/models.html#conformer-ctc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cab9eca4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Generate transcriptions\n",
    "!python transcribe_speech.py \\\n",
    "    pretrained_name=stt_en_conformer_ctc_large \\\n",
    "    dataset_manifest={manifest_file} \\\n",
    "    output_filename={asr_pred} \\\n",
    "    batch_size=32 ++compute_langs=False cuda=0 amp=True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d35cd57",
   "metadata": {},
   "source": [
    "Lets take a look at the asr_pred file and make sure we have a `text` field and an `pred_text` field. The asr_pred file should look like this:\n",
    "\n",
    "\n",
    "    {\"audio_filepath\": \"6097_5_mins/audio/presentpictureofnsw_02_mann_0532.wav\", \"text\": \"not to stop more than ten minutes by the way\", \"duration\": 2.6, \"text_no_preprocessing\": \"not to stop more than ten minutes by the way,\", \"text_normalized\": \"not to stop more than ten minutes by the way,\", \"pred_text\": \"not to stop more than ten minutes by the way\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ad668f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -2 {asr_pred}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f069fd",
   "metadata": {},
   "source": [
    "## Calculate character error rate (CER).\n",
    "\n",
    "Edit distance or Levenshtein distance is a metric to measure the similarity of two strings. The metric accounts for any additions, deletions or substitutions in ground truth to get the evaluation string.\n",
    "\n",
    "\n",
    "Use [Levenshtein distance](https://pypi.org/project/editdistance/) to measure `edit distance` and `character error rate` between generated transcript and ground truth transcript. \n",
    "\n",
    "`character error rate` is edit distance per word of ground truth. It can also be interpreted as normalised edit distance.\n",
    "\n",
    "\n",
    "$error\\ rate = \\frac{edit\\ distance}{no\\ of\\ words\\ in\\ ground\\ truth}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ca1016",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Install the edit distance package\n",
    "!pip install editdistance\n",
    "## Install ndjson to read the asr_pred file\n",
    "!pip install ndjson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae839737",
   "metadata": {},
   "outputs": [],
   "source": [
    "import editdistance\n",
    "import ndjson\n",
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59de72b",
   "metadata": {},
   "source": [
    "Set thresholds for edit distance and error rate. Any utterance with that exceeds these thresholds requires investigation. These values can be finetuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56a371ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "distance_threshold = 5\n",
    "cer_threshold = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcdbfcb4",
   "metadata": {},
   "source": [
    "Since ASR transcripts does not contain any punctuation, remove punctuation from original transcript before calculating edit distance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9936dfed",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Punctuation translation dictionary.\n",
    "punct_dict = str.maketrans('', '', string.punctuation)\n",
    "\n",
    "f = open(asr_pred)\n",
    "manifest = ndjson.load(f)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63a3e309",
   "metadata": {},
   "source": [
    "Calculate edit distance and print all utterances with:\n",
    " - error_rate > error_threshold\n",
    " - distance > distance_threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2988bf0",
   "metadata": {},
   "outputs": [],
   "source": [
    "for line in manifest:\n",
    "    transcript = line[\"text\"].lower().translate(punct_dict)\n",
    "    pred_text = line[\"pred_text\"]\n",
    "    try:\n",
    "        distance = editdistance.eval(transcript, pred_text)\n",
    "        cer = distance / len(transcript)\n",
    "    except Exception as e:\n",
    "        print(f\"Got error: {e} for line: {line}\")\n",
    "        distance = 0\n",
    "        cer = 0\n",
    "    if distance > distance_threshold or cer > cer_threshold:\n",
    "        print(f\"Low confidence for {line}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd6ce7c2",
   "metadata": {},
   "source": [
    "## Calculate WER(Word error rate)\n",
    "Now we have listed all the sentences with high character error rate, we will list all the sentences with high Word error rate.\n",
    "\n",
    "\n",
    "Word error rate as the name suggests measures the errors at word level instead of character level in `character error rate`. This metric accounts for number of word substitution, word insertions and word deletions from reference text.\n",
    "\n",
    "\n",
    "The formula for calculation is:\n",
    "$$\n",
    "WER=\\frac{S+I+D}{N}\n",
    "$$\n",
    "S = Number of substitutions  \n",
    "I = Number of insertions  \n",
    "D = Number of deletions  \n",
    "N = Total number of words in reference text\n",
    "\n",
    "We will use python package [jiwer](https://github.com/jitsi/jiwer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69acc907",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Install python package to calculate word error rate.\n",
    "!pip install jiwer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f94d7e9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from jiwer import wer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d002ac3",
   "metadata": {},
   "source": [
    "Set threshold for word error rate. Any utterance with WER greater than this value requires investigation. This value can be finetuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d7e6d97",
   "metadata": {},
   "outputs": [],
   "source": [
    "wer_threshold = 0.8 #Can be finetuned."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9a6d758",
   "metadata": {},
   "source": [
    "Calculate word error rate and print all the utterances with high word error rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cff6959",
   "metadata": {},
   "outputs": [],
   "source": [
    "for line in manifest:\n",
    "    transcript = line[\"text\"].lower().translate(punct_dict)\n",
    "    pred_text = line[\"pred_text\"]\n",
    "    try:\n",
    "        error_rate = wer(transcript, pred_text)\n",
    "    except Exception as e:\n",
    "        print(f\"Got error: {e} for line: {line}\")\n",
    "        error_rate = 0\n",
    "    if error_rate > wer_threshold:\n",
    "        print(f\"Low confidence for file: {line['audio_filepath']} --- Transcript: {transcript} --- Predicted text: {pred_text} --- Word error rate: {error_rate}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c55f2532",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this tutorial we have learned to calculate edit distance, character error rate and word rate. We also learned how to apply these metrics to evaluate the quality of an audio, transcript pair. \n",
    "\n",
    "These types of metrics can be useful smoke tests and selecting a candidate model. But at the end, the only way to measure the quality of TTS model is to use subjective methods for evaluating and comparing models such as MOS and CMOS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcabf9b2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py38_speech_ml0.6",
   "language": "python",
   "name": "py38_speech_ml0.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}