technical_notes_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "authorship_tag": "ABX9TyM5LuEfdLCCLg1V+8t+9dAp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/michaelwnau/TechnicalxNotes/blob/main/technical_notes_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Technical Notes 2 - Exploring Advanced NLP Tools: TikToken and Hugging Face's Chunk Visualizer - 20240224\n",
        "https://tiktokenizer.vercel.app/\n",
        "https://huggingface.co/spaces/m-ric/chunk_visualizer\n",
        "http://chunkviz.com\n",
        "\n",
        "https://phontron.com/class/anlp2024/\n",
        "\n",
        "RAI Toolkit https://rai.tradewindai.com/\n",
        "\n",
        "In today's post, we're diving deep into the world of Natural Language Processing (NLP) with a focus on two tools that we can use to interact with text data: TikToken and Hugging Face's Chunk Visualizer. Additionally, we'll incorporate a practical Colab/Jupyter Notebook tutorial that not only introduces these tools but also demonstrates how to use them effectively.\n",
        "\n",
        "## Introduction to TikToken and Chunk Visualizer\n",
        "Before we jump into the technicalities, let's briefly introduce our main characters:\n",
        "\n",
        "TikToken ([here](https://github.com/openai/tiktoken/)) is a cutting-edge tool designed to tokenize text in a way that's both efficient and intuitive. It offers a unique approach to breaking down text into manageable pieces, making it easier to process and analyze.\n",
        "\n",
        "Hugging Face's Chunk Visualizer ([here](https://huggingface.co/spaces/m-ric/chunk_visualizer)) provides a visual representation of how text \"chunks\" are understood and processed by machine learning models. This tool is invaluable for anyone looking to gain insights into the inner workings of NLP models, offering a clear and interactive way to visualize data.\n",
        "\n",
        "## Colab Notebook Tutorial: Getting Started with Tokenization and Visualization\n",
        "To kick things off, let's begin with a Colab Notebook snippet that sets up our environment for exploring these tools. This tutorial assumes you have a basic understanding of Python and its data science libraries.\n",
        "\n",
        "### Setting Up the Environment\n",
        "First, we need to import the necessary libraries for our exploration:\n"
      ],
      "metadata": {
        "id": "C3WJTbWzHWOK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install trufflehog"
      ],
      "metadata": {
        "id": "oXee4fjfqBvW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tiktoken"
      ],
      "metadata": {
        "id": "jS8FuhzudMIR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install huggingface_hub"
      ],
      "metadata": {
        "id": "UpSSQnmViVZn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import userdata\n",
        "userdata.get('HF_Token')"
      ],
      "metadata": {
        "id": "QyxrcRVTe4ZK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IQMmzKI8HVCO"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Now we want to import the Tokenizers\n",
        "\n",
        "import tokenizers as tk"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "These imports lay the foundation for our data manipulation and visualization tasks. Now, let's move on to the exciting part - working with TikToken and Hugging Face's Chunk Visualizer.\n",
        "\n",
        "### Utilizing TikToken for Efficient Text Tokenization\n",
        "TikToken simplifies the process of tokenizing text, which is the first step in text analysis and NLP. Here's how you can start using it in your projects:"
      ],
      "metadata": {
        "id": "Lzj0xr9kbi9P"
      }
    }
  ]
}