transcribe_audio_with_whisper.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tanyaclement/audio-class/blob/main/transcribe_audio_with_whisper.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HuR-jeyP-An8"
      },
      "source": [
        "# ✨ README\n",
        "\n",
        "This notebook is based on the companion Colab for the article \"[How to transcribe your audio to text, for free (with SRTs/VTTs!)](https://wandb.ai/wandb_fc/gentle-intros/reports/How-to-transcribe-your-audio-to-text-for-free-with-SRTs-VTTs---VmlldzozNDczNTI0)\". It was modified by Tanya Clement January 2, 2025.\n",
        "\n",
        "This Colab shows how to use OpenAI's Whisper to transcribe audio and audiovisual files, and how to save that transcription as a plain text file or as a VTT/SRT caption file.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 📎 Documentation\n",
        "\n",
        "* `input_format`: The source of the audio/video file to be transcribed\n",
        "  * `youtube`: A YouTube video\n",
        "    * The transcribed file(s) are saved to this Colab, and will be deleted when the Colab runtime is disconnected.\n",
        "  * `link`: A direct URL to an online .mp3 file.\n",
        "  * `gdrive`: A file in your Google Drive account\n",
        "    * If you select this option, you will need to allow this notebook to connect to your Google Drive account.\n",
        "    * The transcribed file(s) are saved to the same folder as the original file.\n",
        "  * `local`: A local file that you have uploaded to this Colab\n",
        "    * If you select this option, you will need to first upload the file to the Files tab (see Step 1 [here](https://wandb.ai/wandb_fc/gentle-intros/reports/How-to-transcribe-your-audio-to-text-for-free-with-SRTs-VTTs---VmlldzozMzc1MzU3)).\n",
        "    * The transcribed file(s) are saved to this Colab, and will be deleted when the Colab runtime is disconnected.\n",
        "* `file`: The URL of the YouTube video or the path of the audio file to be transcribed.\n",
        "  * Example: `file = \"https://www.youtube.com/watch?v=AUDIO\"` (transcribing a YouTube video)\n",
        "  * Example: `file = \"/content/drive/My Drive/AUDIO.mp3\"` (transcribing a Google Drive file)\n",
        "  * Example: `file = \"/content/AUDIO.mp3\"` (transcribing a local file)\n",
        "* `plain`: Whether to save the transcription as a text file or not.\n",
        "* `srt`: Whether to save the transcription as an SRT file or not.\n",
        "* `vtt`: Whether to save the transcription as a VTT file or not.\n",
        "* `tsv`: Whether to save the transcription as a TSV (tab-separated values) file or not.\n",
        "* `download`: Whether to download the transcribed file(s) or not.\n"
      ],
      "metadata": {
        "id": "EjLJ6mHptbIx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title 🌴 Change the values in this section\n",
        "\n",
        "# @markdown Select the source of the audio/video file to be transcribed\n",
        "input_format = \"link\" #@param [\"youtube\", \"gdrive\", \"local\", \"link\"]\n",
        "\n",
        "# @markdown Enter the URL of the YouTube video or the path of the audio file to be transcribed\n",
        "file = \"\" #@param {type:\"string\"}\n",
        "\n",
        "#@markdown Click here if you'd like to save the transcription as text file\n",
        "plain = False #@param {type:\"boolean\"}\n",
        "\n",
        "# @markdown Click here if you'd like to save the transcription as an SRT file\n",
        "srt = False #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown Click here if you'd like to save the transcription as a VTT file\n",
        "vtt = False #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown Click here if you'd like to save the transcription as a TSV file\n",
        "tsv = False #@param {type:\"boolean\"}\n",
        "\n",
        "#@markdown Click here if you'd like to download the transcribed file(s) locally\n",
        "download = False #@param {type:\"boolean\"}"
      ],
      "metadata": {
        "id": "fAQKStuINe3G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLoNmM0sKyIf"
      },
      "source": [
        "# 🛠 Set Up\n",
        "\n",
        "The blocks below install all of the necessary Python libraries (including Whisper), configures Whisper, and contains code for various helper functions.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 🤝 Dependencies"
      ],
      "metadata": {
        "id": "hfnRc8yPM79j"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "bF1enPzG-qKE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e96a74f8-584a-4ad8-d541-08f4071d474a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "# Dependencies\n",
        "\n",
        "!pip install -q pytube\n",
        "!pip install -q git+https://github.com/openai/whisper.git\n",
        "\n",
        "import os, re\n",
        "import torch\n",
        "from pathlib import Path\n",
        "from pytube import YouTube\n",
        "\n",
        "import whisper\n",
        "from whisper.utils import get_writer\n",
        "\n",
        "import csv\n",
        "import requests"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 👋 Whisper configuration\n",
        "\n",
        "This Colab use `medium.en`, [the medium-sized, English-only](https://github.com/openai/whisper#available-models-and-languages) Whisper model.\n"
      ],
      "metadata": {
        "id": "E4eLQzNOo5_r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use CUDA, if available\n",
        "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "\n",
        "# Load the desired model\n",
        "model = whisper.load_model(\"medium.en\").to(DEVICE)"
      ],
      "metadata": {
        "id": "YCNc3EfV4EIt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "81539d7c-b80e-4f3c-f197-5862357666ab"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/whisper/__init__.py:150: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
            "  checkpoint = torch.load(fp, map_location=device)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 💪 YouTube helper functions\n",
        "\n",
        "Code for helper functions when running Whisper on a YouTube video."
      ],
      "metadata": {
        "id": "IvN1wRXbo-7C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def to_snake_case(name):\n",
        "    return name.lower().replace(\" \", \"_\").replace(\":\", \"_\").replace(\"__\", \"_\")\n",
        "\n",
        "def download_youtube_audio(url,  file_name = None, out_dir = \".\"):\n",
        "    \"Download the audio from a YouTube video\"\n",
        "    yt = YouTube(url)\n",
        "    if file_name is None:\n",
        "        file_name = Path(out_dir, to_snake_case(yt.title)).with_suffix(\".mp4\")\n",
        "    yt = (yt.streams\n",
        "            .filter(only_audio = True, file_extension = \"mp4\")\n",
        "            .order_by(\"abr\")\n",
        "            .desc())\n",
        "    return yt.first().download(filename = file_name)"
      ],
      "metadata": {
        "id": "RLmwvJ3tM-CD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# ✍ Transcribing with Whisper\n",
        "\n",
        "Ultimately, calling Whisper is as easy as one line!\n",
        "* `result = model.transcribe(file)`\n",
        "\n",
        "The majority of this new `transcribe_file` function is actually just for exporting the results of the transcription as a text, VTT, or SRT file."
      ],
      "metadata": {
        "id": "ech5wPCwtO_P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def transcribe_file(model, file, plain, srt, vtt, tsv, download, input_format): # Add input_format as an argument\n",
        "    \"\"\"\n",
        "    Runs Whisper on an audio file\n",
        "\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    model: Whisper\n",
        "        The Whisper model instance.\n",
        "\n",
        "    file: str\n",
        "        The file path of the file to be transcribed.\n",
        "\n",
        "    plain: bool\n",
        "        Whether to save the transcription as a text file or not.\n",
        "\n",
        "    srt: bool\n",
        "        Whether to save the transcription as an SRT file or not.\n",
        "\n",
        "    vtt: bool\n",
        "        Whether to save the transcription as a VTT file or not.\n",
        "\n",
        "    tsv: bool\n",
        "        Whether to save the transcription as a TSV file or not.\n",
        "\n",
        "    download: bool\n",
        "        Whether to download the transcribed file(s) or not.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    A dictionary containing the resulting text (\"text\") and segment-level details (\"segments\"), and\n",
        "    the spoken language (\"language\"), which is detected when `decode_options[\"language\"]` is None.\n",
        "    \"\"\"\n",
        "    if input_format == \"link\":\n",
        "         # Download the audio from the link\n",
        "         audio_content = requests.get(file).content\n",
        "\n",
        "         # Save the audio to a temporary file\n",
        "         with open(\"temp_audio.mp3\", \"wb\") as temp_audio_file:\n",
        "             temp_audio_file.write(audio_content)\n",
        "\n",
        "         file_path = Path(\"temp_audio.mp3\")\n",
        "\n",
        "    else:\n",
        "\n",
        "         file_path = Path(file)\n",
        "    print(f\"Transcribing file: {file_path}\\n\")\n",
        "\n",
        "    output_directory = file_path.parent\n",
        "\n",
        "    # Run Whisper\n",
        "    result = model.transcribe(file, verbose = False, language = \"en\")\n",
        "\n",
        "    if plain:\n",
        "        txt_path = file_path.with_suffix(\".txt\")\n",
        "        print(f\"\\nCreating text file\")\n",
        "\n",
        "        with open(txt_path, \"w\", encoding=\"utf-8\") as txt:\n",
        "            txt.write(result[\"text\"])\n",
        "    if srt:\n",
        "        print(f\"\\nCreating SRT file\")\n",
        "        srt_writer = get_writer(\"srt\", output_directory)\n",
        "        srt_writer(result, str(file_path.stem))\n",
        "\n",
        "    if vtt:\n",
        "        print(f\"\\nCreating VTT file\")\n",
        "        vtt_writer = get_writer(\"vtt\", output_directory)\n",
        "        vtt_writer(result, str(file_path.stem))\n",
        "\n",
        "    if tsv:\n",
        "        print(f\"\\nCreating TSV file\")\n",
        "        tsv_path = file_path.with_suffix(\".tsv\")\n",
        "\n",
        "        with open(tsv_path, \"w\", newline=\"\", encoding=\"utf-8\") as tsvfile:\n",
        "            writer = csv.writer(tsvfile, delimiter=\"\\t\")\n",
        "            writer.writerow([\"start\", \"end\", \"text\"])  # Write header row\n",
        "\n",
        "            for segment in result[\"segments\"]:\n",
        "                # Round start and end times to the nearest whole number\n",
        "                start = round(segment[\"start\"])\n",
        "                end = round(segment[\"end\"])\n",
        "                text = segment[\"text\"]\n",
        "                writer.writerow([start, end, text])\n",
        "\n",
        "    if download:\n",
        "        from google.colab import files\n",
        "\n",
        "        colab_files = Path(\"/content\")\n",
        "        stem = file_path.stem\n",
        "\n",
        "        for colab_file in colab_files.glob(f\"{stem}*\"):\n",
        "            if colab_file.suffix in [\".txt\", \".srt\", \".vtt\", \".tsv\"]:\n",
        "                print(f\"Downloading {colab_file}\")\n",
        "                files.download(str(colab_file))\n",
        "\n",
        "    return result"
      ],
      "metadata": {
        "id": "22CwQZnOtGO1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 💬 Whisper it!\n",
        "\n",
        "This block actually calls `transcribe_file` 😉\n"
      ],
      "metadata": {
        "id": "CLC_tpz6tgq6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if input_format == \"youtube\":\n",
        "    # Download the audio stream of the YouTube video\n",
        "    print(f\"Downloading audio stream: {audio}\")\n",
        "    audio = download_youtube_audio(file)\n",
        "\n",
        "    # Run Whisper on the audio stream\n",
        "    result = transcribe_file(model, audio, plain, srt, vtt, tsv, download, input_format) # Pass input_format here\n",
        "elif input_format == \"gdrive\":\n",
        "    # Authorize a connection between Google Drive and Google Colab\n",
        "    from google.colab import drive\n",
        "    drive.mount('/content/drive')\n",
        "\n",
        "    # Run Whisper on the specified file\n",
        "    result = transcribe_file(model, file, plain, srt, vtt, tsv, download, input_format) # Pass input_format here\n",
        "elif input_format == \"local\":\n",
        "    # Run Whisper on the specified file\n",
        "    result = transcribe_file(model, file, plain, srt, vtt, tsv, download, input_format) # Pass input_format here\n",
        "elif input_format == \"link\":\n",
        "    # Run Whisper on the specified link\n",
        "    result = transcribe_file(model, file, plain, srt, vtt, tsv, download, input_format) # Pass input_format here"
      ],
      "metadata": {
        "id": "ngTGllHutSfo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "outputId": "8163ab32-cc77-4e64-8e19-ce382c23242d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Transcribing file: temp_audio.mp3\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 285722/285722 [04:42<00:00, 1009.79frames/s]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Creating TSV file\n",
            "Downloading /content/temp_audio.tsv\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_d4007f9c-f9ed-4434-bde0-c47da2467697\", \"temp_audio.tsv\", 37485)"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard",
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}