notebooks/gpt/gpt.ipynb

{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  },
  "colab": {
   "name": "gpt.ipynb",
   "provenance": []
  },
  "accelerator": "GPU"
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "cWqvLoZ5KHfW"
   },
   "source": [
    "# The GPT Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Run on Colab:\n",
    "\n",
    "Launch this link to run this on colab [here](https://colab.research.google.com/github/will-thompson-k/deeplearning-nlp-models/blob/master/notebooks/gpt/gpt.ipynb)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "JSPXhKfdKHfW"
   },
   "source": [
    "## Colab Setup\n",
    "\n",
    "You can skip this section if not running on Google's colab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "Ope1RifFKHfW"
   },
   "source": [
    "If running with GPUs, sanity check that the GPUs are enabled."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "A67s8lkTKHfW"
   },
   "source": [
    "!nvidia-smi"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "pnEYkjAXKHfW",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "5c392d2c-e661-4620-e954-2212103e8fdc"
   },
   "source": [
    "import torch\n",
    "torch.cuda.is_available()"
   ],
   "execution_count": 1,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "metadata": {
      "tags": []
     },
     "execution_count": 1
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "vH8GgVrAKHfW"
   },
   "source": [
    "Should be True. If not, debug (Note: version of pytorch I used is not capatible with CUDA drivers on colab. Follow these instructions here explicitly)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "SdLxeccRKHfW",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "d592e075-2408-4785-9ccc-49e264b2e3ed"
   },
   "source": [
    "!pwd"
   ],
   "execution_count": 2,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "/content\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "wn4MxAW3KHfW"
   },
   "source": [
    "This should be \"/content\" on Colab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "qKH_ORuxKHfX"
   },
   "source": [
    "First, if running from colab, you must install the package. (You may skip if you installed already)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "nlbkIL-uKHfX"
   },
   "source": [
    "!git clone https://github.com/will-thompson-k/deeplearning-nlp-models.git\n",
    "%cd deeplearning-nlp-models"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "AWhgwdqEKHfX"
   },
   "source": [
    "!pip install datasets"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "LFk888iGKHfX"
   },
   "source": [
    "!python setup.py install"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "adqvmbeTKHfX"
   },
   "source": [
    "## Imports\n",
    "\n",
    "Here are the packages we need to import."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "nNZ3Z82aKHfX"
   },
   "source": [
    "from nlpmodels.models import gpt\n",
    "from nlpmodels.utils import train,utils,gpt_sampler\n",
    "from nlpmodels.utils.elt import gpt_dataset\n",
    "from argparse import Namespace\n",
    "utils.set_seed_everywhere()"
   ],
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "WCJ2j5gjKHfX"
   },
   "source": [
    "## Language Model: WikiText2\n",
    "\n",
    "We will try to train our GPT model to learn how to predict the next word in WikiText2 data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "LDIaCTagKHfX"
   },
   "source": [
    "### Hyper-parameters\n",
    "\n",
    "These are the data processing and model training hyper-parameters for this run. Note that we are running a smaller model\n",
    "than cited in the paper for fewer iterations. This is meant merely to demonstrate it works."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%% Parameters\n"
    },
    "id": "JaQqbExKKHfX"
   },
   "source": [
    "args = Namespace(\n",
    "        # Model hyper-parameters\n",
    "        num_layers_per_stack=4,  # original value = 12\n",
    "        dim_model=256, #original value = 768\n",
    "        dim_ffn=1024, # original value = 3072\n",
    "        num_heads=4, # original value = 12\n",
    "        block_size=64, # original value = 512, context window\n",
    "        dropout=0.1,\n",
    "        # Training hyper-parameters\n",
    "        num_epochs=3, #obviously super short\n",
    "        learning_rate=0.0,\n",
    "        batch_size=32, #original value = 64\n",
    "    )"
   ],
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "collapsed": true,
    "id": "pC43FCeVKHfX"
   },
   "source": [
    "train_loader, vocab = gpt_dataset.GPTDataset.get_training_dataloader(args)\n",
    "model = gpt.GPT(vocab_size = len(vocab),\n",
    "            num_layers_per_stack= args.num_layers_per_stack,\n",
    "            dim_model = args.dim_model,\n",
    "            dim_ffn = args.dim_ffn,\n",
    "            num_heads = args.num_heads,\n",
    "            block_size = args.block_size,\n",
    "            dropout = args.dropout)\n",
    "trainer = train.GPTTrainer(args,vocab.mask_index,model,train_loader,vocab)"
   ],
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "N2z3sqRJKHfY"
   },
   "source": [
    "# Self-supervised training\n",
    "\n",
    "\n",
    "This is the self-supervised pre-training. After this model is trained,\n",
    "we can run then continue it onto another problem (can freeze layers to only continue training the top layers)."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "ESk2wyBUKHfY",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "ee6fc7d9-ccd6-4a01-a3b0-daf0aa058833"
   },
   "source": [
    "trainer.run()"
   ],
   "execution_count": 9,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "[Epoch 0]: 100%|██████████| 50864/50864 [1:35:04<00:00,  8.92it/s, loss=4.4]\n",
      "[Epoch 1]: 100%|██████████| 50864/50864 [1:35:28<00:00,  8.88it/s, loss=4.32]\n",
      "[Epoch 2]: 100%|██████████| 50864/50864 [1:35:19<00:00,  8.89it/s, loss=3.58]"
     ],
     "name": "stderr"
    },
    {
     "output_type": "stream",
     "text": [
      "Finished Training...\n"
     ],
     "name": "stdout"
    },
    {
     "output_type": "stream",
     "text": [
      "\n"
     ],
     "name": "stderr"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "WLwTk8fSKHfY"
   },
   "source": [
    "Note that this model was run for a **very** short period of time. The goal is just to show how this works - you can\n",
    "play with the hyper-parameters as you see fit.\n",
    "We only ran for a few epochs, on a much smaller model,\n",
    "with a smaller dataset than was suggested in the paper.\n",
    "\n",
    "Let's see if the output the model completes makes any sense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "KBvFfYmuKHfY"
   },
   "source": [
    "# GPT Completes A Sequence\n",
    "\n",
    "In the spirit of Kaparthy's minGPT::play_char notebook, we can use a sampler to see how the model\n",
    "continues a sequence."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "id": "iu0yyBuwTV2q"
   },
   "source": [
    "from typing import Tuple \n",
    "from nlpmodels.utils.elt import gpt_batch\n",
    "\n",
    "def reformat_data(data: Tuple) -> gpt_batch.GPTBatch:\n",
    "  \"\"\"\n",
    "  Args:\n",
    "      data (Tuple): The tuples of LongTensors to be converted into a Batch object.\n",
    "  Returns:\n",
    "      GPTBatch object containing data for a given batch.\n",
    "  \"\"\"\n",
    "  # (batch,seq) shape tensors\n",
    "  source_integers, target_integers = data\n",
    "\n",
    "  device = 'cpu'\n",
    "  if torch.cuda.is_available():\n",
    "      device = torch.cuda.current_device()\n",
    "  # place data on the correct device\n",
    "  source_integers = source_integers.to(device) if source_integers is not None else source_integers\n",
    "  target_integers = target_integers.to(device) if target_integers is not None else target_integers\n",
    "\n",
    "  # return a batch object with src,src_mask,tgt,tgt_mask tensors\n",
    "  batch_data = gpt_batch.GPTBatch(source_integers,\n",
    "                                  target_integers,\n",
    "                                  0)\n",
    "\n",
    "  return batch_data"
   ],
   "execution_count": 10,
   "outputs": []
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "0Gbx7PbeKHfY"
   },
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "\n",
    "prompt = \"ernest hemingway first novel , the sun also rises , \" \\\n",
    "         \"treats of certain of those younger americas concerning whom gertrude stein has remarked :\" \\\n",
    "         \" you are all a lost generation . this is the novel for which a keen appetite was stimulated by\" \\\n",
    "         \" mr . hemingway 's exciting volume of short stories. \" \\\n",
    "         \" the clear objectivity and the sustained intensity of the stories , \" \\\n",
    "         \"and their concentration upon action in the present moment, seemed to point to a failure to project \" \\\n",
    "         \"a novel in terms of the same method, yet a resort to any other method would have let down the \" \\\n",
    "         \"reader's expectations. it is a relief to find that the sun also rises maintains the same heightened , \" \\\n",
    "         \"intimate tangibility as the shorter narratives and does it in the same kind of weighted, quickening prose. \"\n",
    "prompt_tensor = torch.LongTensor([[vocab.lookup_token(s) for s in prompt.split(\" \")]])\n",
    "prompt_tensor_batch = reformat_data((prompt_tensor,None))\n",
    "steps = 64\n",
    "y_hat_indices = gpt_sampler.sampler(model=model, data=prompt_tensor_batch,\n",
    "                                          steps=steps,block_size=64,do_sample=True).src\n",
    "y_hat_tokens = ' '.join([vocab.lookup_index(int(idx)) for idx in y_hat_indices[0]])"
   ],
   "execution_count": 11,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "rzendnrIKHfY"
   },
   "source": [
    "Here is the prompt it was provided, a review of Ernest Hemingway's novel, The Sun Also Rises:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "9UysiahgKHfY",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "b030c4b2-6d5f-46df-c459-c53cc1f23eff"
   },
   "source": [
    "for idx in range(0,len(prompt.split(\" \")),8):\n",
    "    print(\" \".join(prompt.split(\" \")[idx:idx+8]))"
   ],
   "execution_count": 12,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "ernest hemingway first novel , the sun also\n",
      "rises , treats of certain of those younger\n",
      "americas concerning whom gertrude stein has remarked :\n",
      "you are all a lost generation . this\n",
      "is the novel for which a keen appetite\n",
      "was stimulated by mr . hemingway 's exciting\n",
      "volume of short stories.  the clear objectivity\n",
      "and the sustained intensity of the stories ,\n",
      "and their concentration upon action in the present\n",
      "moment, seemed to point to a failure to\n",
      "project a novel in terms of the same\n",
      "method, yet a resort to any other method\n",
      "would have let down the reader's expectations. it\n",
      "is a relief to find that the sun\n",
      "also rises maintains the same heightened , intimate\n",
      "tangibility as the shorter narratives and does it\n",
      "in the same kind of weighted, quickening prose.\n",
      "\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "-z8__hoUKHfY"
   },
   "source": [
    "Working off that sequence, here is how the model completed the next 64 words in this review:"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "id": "8FVIBgfRKHfY",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "bec63a59-7814-4189-f8da-e8d8becf5c5e"
   },
   "source": [
    "for idx in range(0,len(y_hat_tokens.split(\" \")),8):\n",
    "    print(\" \".join(y_hat_tokens.split(\" \")[idx:idx+8]))"
   ],
   "execution_count": 13,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "<UNK> is about one tenth boston globe shall\n",
      "need to the coming from each of the\n",
      "mouth he places claims in the charter in\n",
      "the church of protestant church in england encouraged\n",
      "conceded twenty albeit with winning the planned seven\n",
      "matches gunnery practice of sponsorship the town to\n",
      "reach the goal is marked with four points\n",
      "for the fourth test following the loss of\n",
      "the\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    },
    "id": "_uXI1BvcKHfY"
   },
   "source": [
    "Well, as expected... this doesn't make any sense really. Pockets of words make sense, but overall it does not.\n",
    "\n",
    "A couple of considerations for further work: (1) training for longer and (2) larger models/ different hyperparameters. \n",
    "There is a third option, (3),  which is to attempt a character-level language model, as well.\n"
   ]
  }
 ]
}