(Uncased_Squad_V1_1)_케라스로_Q&A_구현하기.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    },
    "colab": {
      "name": "(Uncased_Squad V1.1) 케라스로_Q&A_구현하기.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kimwoonggon/publicservant_AI/blob/master/(Uncased_Squad_V1_1)_%EC%BC%80%EB%9D%BC%EC%8A%A4%EB%A1%9C_Q%26A_%EA%B5%AC%ED%98%84%ED%95%98%EA%B8%B0.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fn7pKyD8dBEq",
        "colab_type": "text"
      },
      "source": [
        "<h1><strong>케라스로 Q&A 구현하기 With SQUAD</strong></h1>\n",
        "<h1><strong><u>Youtube 공무원AI</u></strong></h1>\n",
        "\n",
        "![대체 텍스트](https://i.imgur.com/XSuDmcC.png)\n",
        "\n",
        "이번 튜토리얼에서는 케라스와 BERT를 활용하여 <strong>SQUAD(Standford Question and Answering Dataset)</strong>을 실습해보고자 합니다.\n",
        "\n",
        "SQUAD는 문장과 질문(Question)을 입력하면 그에 해당하는 답(ANSWER)를 알려주는 문제입니다.  \n",
        "즉 AI가 영어 독해 문제를 풀어주는 것입니다.   \n",
        "Tensorflow나 PyTorch로 SQUAD를 구현하는 코드들은 인터넷에 많지만 초보자 입장에서는 코드를 봐도 구현하기가 상당히 어렵습니다. \n",
        "막상 코드를 돌려봐도 어떤 원리로 돌아가는지 알기 어렵습니다.  \n",
        " \n",
        "그래서 KERAS를 활용하여 쉽게 SQUAD를 구현해보고자 합니다.  \n",
        "본 튜토리얼은 1)SQUAD 이해 2) BERT INPUT 만들기 3) SQUAD 구현 4) SQUAD 예측 총 4단계로 구성되어 있습니다.  \n",
        "각 단계마다 이해하기 쉬운 설명을 곁들이도록 하겠습니다. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LeBONl3Rt6J",
        "colab_type": "text"
      },
      "source": [
        "<h2><strong> 1. SQUAD 개념 이해</strong></h2>  \n",
        "SQUAD 문제는 아래와 같습니다.\n",
        "질문을 하면 AI가 Context를 읽고 정답을 출력값으로 알려주게 됩니다.  \n",
        "처음에 나온 정답은 AI가 예측한 정답입니다. 그리고 원래 정답은 Von Miller인데 왜 von miller ##r로 예측이 되었냐면, 실제로 예측한 것은 bert에 들어간 토큰이 예측 된 것이기 때문입니다.  \n",
        "  \n",
        "\n",
        "![Imgur](https://i.imgur.com/sjf7vyj.png)  \n",
        "\n",
        "사실 SQUAD는 ANSWER를 다 예측하는 것이 아니라, ANSWER 중에서도 시작단어와 끝 단어만을 예측합니다. 시작과 끝을 알면 자연스럽게 가운데 위치한 글자들도 예측이 되는 것이겠지요. 그리고 SQUAD 문제를 풀기 위해서 BERT 알고리즘을 사용합니다.  \n",
        "  \n",
        "\n",
        "위 그림에서 SQUAD는 ANSWER를 다 예측하는 것이 아니라, ANSWER 중에서도 시작단어와 끝 단어만을 예측합니다. 시작과 끝을 알면 자연스럽게 가운데 위치한 글자들도 예측이 되는 것이겠지요. 그리고 SQUAD 문제를 풀기 위해서 BERT 알고리즘을 사용합니다.  \n",
        "  \n",
        "  ![imgur](https://i.imgur.com/xhbWa13.png)  \n",
        "  위 그림에서 버트 모형에 들어가는 인풋은 Question(질문)과 Paragraph(지문)입니다.  \n",
        "  QUESTION과 PARAGRAPH가 버트의 인풋으로 들어가면 아웃풋 값으로 처음 단어와 큰 단어를 알려주게 됩니다.  \n",
        "\n",
        "  Train Data의 생김새입니다. 총 87599개의 문장이 있습니다. Question과 Context를 인풋으로 받아서, 아웃풋으로 Text(정답)을 맞추게 됩니다.\n",
        "  ![Imgur](https://i.imgur.com/rzTXJ1W.png)\n",
        "\n",
        "  Test Data의 생김새입니다. 총 10,570개의 문장으로 이루어져 있으며, question과 context를 인풋으로 넣으면, 파인튜닝된 버트 모형이 text를 아웃풋 값으로 출력하게 됩니다.\n",
        "  ![Imgur](https://i.imgur.com/s9zqAvv.png)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rtIDGhzFBrdL",
        "colab_type": "text"
      },
      "source": [
        "<h2><strong> 2. SQUAD 실행 준비</strong></h2>  \n",
        "구글 Bert Base 모델을 다운받기로 하겠습니다. 본 분석에서 쓰이는 모델은 Bert-Base-Uncased입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sx2147Ndq3lS",
        "colab_type": "code",
        "outputId": "1580871f-6eb1-4dd4-d626-9bba0f6e4ea3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 625
        }
      },
      "source": [
        "import os\n",
        "import zipfile\n",
        "import shutil\n",
        "\n",
        "# 버트 모델 다운로드\n",
        "!wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\n",
        "\n",
        "# colab에 bert 폴더 생성\n",
        "if \"bert\" not in os.listdir():\n",
        "  os.makedirs(\"bert\")\n",
        "else:\n",
        "  pass\n",
        "\n",
        "# 다운받은 버트 모형을 bert 폴더에 압축 풀기\n",
        "bert_zip = zipfile.ZipFile('uncased_L-12_H-768_A-12.zip')\n",
        "bert_zip.extractall('bert')\n",
        " \n",
        "bert_zip.close()\n",
        "\n",
        "# 폴더를 통째로 복사하는 함수 생성\n",
        "def copytree(src, dst, symlinks=False, ignore=None):\n",
        "    for item in os.listdir(src):\n",
        "        s = os.path.join(src, item)\n",
        "        d = os.path.join(dst, item)\n",
        "        if os.path.isdir(s):\n",
        "            shutil.copytree(s, d, symlinks, ignore)\n",
        "        else:\n",
        "            shutil.copy2(s, d)\n",
        "\n",
        "# 원래 압축이 풀린 폴더는 \"bert/uncased_L-12_H-768_A-12\" 이지만\n",
        " \n",
        "# 분석에 편리하게 쓰기 위해 bert 폴더로 버트 모델을 이동시키자\n",
        "copytree(\"bert/uncased_L-12_H-768_A-12\", \"bert\")\n",
        "\n",
        "\n",
        "# 이번 실습에서 사용할 데이터 다운로드(SQuAD v1.1) Train and dev set\n",
        "!wget https://raw.githubusercontent.com/nate-parrott/squad/master/data/train-v1.1.json\n",
        "!wget https://raw.githubusercontent.com/nate-parrott/squad/master/data/dev-v1.1.json"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-02-29 08:02:32--  https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 108.177.15.128, 2a00:1450:400c:c07::80\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|108.177.15.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 407727028 (389M) [application/zip]\n",
            "Saving to: ‘uncased_L-12_H-768_A-12.zip.1’\n",
            "\n",
            "uncased_L-12_H-768_ 100%[===================>] 388.84M  78.2MB/s    in 5.3s    \n",
            "\n",
            "2020-02-29 08:02:38 (73.0 MB/s) - ‘uncased_L-12_H-768_A-12.zip.1’ saved [407727028/407727028]\n",
            "\n",
            "--2020-02-29 08:02:45--  https://raw.githubusercontent.com/nate-parrott/squad/master/data/train-v1.1.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 30288272 (29M) [text/plain]\n",
            "Saving to: ‘train-v1.1.json.1’\n",
            "\n",
            "train-v1.1.json.1   100%[===================>]  28.88M   191MB/s    in 0.2s    \n",
            "\n",
            "2020-02-29 08:02:47 (191 MB/s) - ‘train-v1.1.json.1’ saved [30288272/30288272]\n",
            "\n",
            "--2020-02-29 08:02:49--  https://raw.githubusercontent.com/nate-parrott/squad/master/data/dev-v1.1.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 4854279 (4.6M) [text/plain]\n",
            "Saving to: ‘dev-v1.1.json.1’\n",
            "\n",
            "dev-v1.1.json.1     100%[===================>]   4.63M  --.-KB/s    in 0.1s    \n",
            "\n",
            "2020-02-29 08:02:50 (46.4 MB/s) - ‘dev-v1.1.json.1’ saved [4854279/4854279]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Tkah2Z4n_z6",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "##**구글 드라이브와 Colab을 연동합니다**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hPs7cH6On_Qa",
        "colab_type": "code",
        "outputId": "cc50ff02-1fb3-4d3c-f7cb-a669ba926af7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "import os\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive/')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/gdrive/; to attempt to forcibly remount, call drive.mount(\"/content/gdrive/\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HQ6glcvV1h73",
        "colab_type": "text"
      },
      "source": [
        "<h2><strong>본격적으로 케라스와 버트를 활용하여 SQUAD 예측 모델을 만들어 보겠습니다.</strong></h2>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVGCHkKN0TZ3",
        "colab_type": "text"
      },
      "source": [
        "텐서플로우, 판다스, 넘파이, 케라스 등 필요한 모듈들을 임포트합니다"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ffa---Ov8WfF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "import tensorflow as tf\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import re\n",
        "import pickle\n",
        "\n",
        "import keras as keras\n",
        "from keras.models import load_model\n",
        "from keras import backend as K\n",
        "from keras import Input, Model\n",
        "from keras import optimizers\n",
        "\n",
        "from keras import backend as K\n",
        "from keras.layers import Layer\n",
        "\n",
        "import codecs\n",
        "from tqdm import tqdm\n",
        "import shutil\n",
        "import json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T07lROYqCqOH",
        "colab_type": "text"
      },
      "source": [
        "각종 로그들이 뜨는 것을 방지"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bniU2r9PouUO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import warnings\n",
        "import tensorflow as tf\n",
        "warnings.filterwarnings(action='ignore')\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
        "tf.logging.set_verbosity(tf.logging.ERROR)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3bWhmOP0mOv",
        "colab_type": "text"
      },
      "source": [
        "케라스에서 Bert 활용을 쉽게 만들어주는 모듈 keras-bert를 설치합니다<br>그리고 Adam optimizer의 수정판인 keras-radam 모듈을 임포트합니다.  \n",
        "keras-radam은 모델을 훈련시킬 때, learning_rate을 점진적으로 목표값까지 올려주는 방법으로 훈련 과정을 안정화시키는 optimizer입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "09QnmSZXGbbD",
        "colab_type": "code",
        "outputId": "2eeebdd3-3338-43bb-d0cc-3367e9b5b826",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 482
        }
      },
      "source": [
        "!pip install keras-bert\n",
        "!pip install keras-radam"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: keras-bert in /usr/local/lib/python3.6/dist-packages (0.81.0)\n",
            "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from keras-bert) (2.2.5)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-bert) (1.17.5)\n",
            "Requirement already satisfied: keras-transformer>=0.30.0 in /usr/local/lib/python3.6/dist-packages (from keras-bert) (0.32.0)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.0.8)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.1.0)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (2.8.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (3.13)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.12.0)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-bert) (1.4.1)\n",
            "Requirement already satisfied: keras-pos-embd>=0.10.0 in /usr/local/lib/python3.6/dist-packages (from keras-transformer>=0.30.0->keras-bert) (0.11.0)\n",
            "Requirement already satisfied: keras-position-wise-feed-forward>=0.5.0 in /usr/local/lib/python3.6/dist-packages (from keras-transformer>=0.30.0->keras-bert) (0.6.0)\n",
            "Requirement already satisfied: keras-embed-sim>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from keras-transformer>=0.30.0->keras-bert) (0.7.0)\n",
            "Requirement already satisfied: keras-layer-normalization>=0.12.0 in /usr/local/lib/python3.6/dist-packages (from keras-transformer>=0.30.0->keras-bert) (0.14.0)\n",
            "Requirement already satisfied: keras-multi-head>=0.22.0 in /usr/local/lib/python3.6/dist-packages (from keras-transformer>=0.30.0->keras-bert) (0.22.0)\n",
            "Requirement already satisfied: keras-self-attention==0.41.0 in /usr/local/lib/python3.6/dist-packages (from keras-multi-head>=0.22.0->keras-transformer>=0.30.0->keras-bert) (0.41.0)\n",
            "Requirement already satisfied: keras-radam in /usr/local/lib/python3.6/dist-packages (0.15.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from keras-radam) (1.17.5)\n",
            "Requirement already satisfied: Keras in /usr/local/lib/python3.6/dist-packages (from keras-radam) (2.2.5)\n",
            "Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.0.8)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.1.0)\n",
            "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.4.1)\n",
            "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (2.8.0)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (3.13)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from Keras->keras-radam) (1.12.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cyIoeOBH04lW",
        "colab_type": "text"
      },
      "source": [
        "keras-bert 라이브러리에서 버트 모형 활용에 필요한 모듈들을 임포트합니다"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FZv-5ALnGdwI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras_bert import load_trained_model_from_checkpoint, load_vocabulary\n",
        "from keras_bert import Tokenizer\n",
        "from keras_bert import AdamWarmup, calc_train_steps\n",
        "\n",
        "from keras_radam import RAdam"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kVGK2j5tKJgS",
        "colab_type": "text"
      },
      "source": [
        "SQUAD JSON파일을 PANDAS DATAFRAME으로 만들어주는 함수를 정의합니다.  \n",
        "출처 : https://www.kaggle.com/sanjay11100/squad-stanford-q-a-json-to-pandas-dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgWmFtiaIZQK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def squad_json_to_dataframe_train(input_file_path, record_path = ['data','paragraphs','qas','answers'],\n",
        "                           verbose = 1):\n",
        "    \"\"\"\n",
        "    input_file_path: path to the squad json file.\n",
        "    record_path: path to deepest level in json file default value is\n",
        "    ['data','paragraphs','qas','answers']\n",
        "    verbose: 0 to suppress it default is 1\n",
        "    \"\"\"\n",
        "    if verbose:\n",
        "        print(\"Reading the json file\")    \n",
        "    file = json.loads(open(input_file_path).read())\n",
        "    if verbose:\n",
        "        print(\"processing...\")\n",
        "    # parsing different level's in the json file\n",
        "    js = pd.io.json.json_normalize(file , record_path )\n",
        "    m = pd.io.json.json_normalize(file, record_path[:-1] )\n",
        "    r = pd.io.json.json_normalize(file,record_path[:-2])\n",
        "    \n",
        "    #combining it into single dataframe\n",
        "    idx = np.repeat(r['context'].values, r.qas.str.len())\n",
        "    ndx  = np.repeat(m['id'].values,m['answers'].str.len())\n",
        "    m['context'] = idx\n",
        "    js['q_idx'] = ndx\n",
        "    main = pd.concat([ m[['id','question','context']].set_index('id'),js.set_index('q_idx')],1,sort=False).reset_index()\n",
        "    main['c_id'] = main['context'].factorize()[0]\n",
        "    if verbose:\n",
        "        print(\"shape of the dataframe is {}\".format(main.shape))\n",
        "        print(\"Done\")\n",
        "    return main"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eoez6gHVGeP8",
        "colab_type": "text"
      },
      "source": [
        "SQUAD 데이터를 PANDAS DATAFRAME 형식으로 로드합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9j2vCEzCMW4K",
        "colab_type": "code",
        "outputId": "59568ab8-7ac8-4122-d42c-891cdcf96863",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "train = squad_json_to_dataframe_train(\"train-v1.1.json\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading the json file\n",
            "processing...\n",
            "shape of the dataframe is (87599, 6)\n",
            "Done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnnEcb9mQxV1",
        "colab_type": "text"
      },
      "source": [
        "SQUAD 예측을 위한 훈련 데이터가 잘 로드되었습니다.  \n",
        "**question 칼럼이 질문, context 칼럼이 문장으로 인풋으로 들어갑니다.  \n",
        "아웃풋 값(정답)은 text 칼럼에서 시작 단어와 끝 단어 두 개 입니다. 예를 들어서, text 값이 Saint Bernadette Soubirous라면, 정답은 시작 단어인 Saint와 끝 단어인 Soubrious입니다.**  \n",
        "  \n",
        "그리고 SQUAD 문제의 특징은, 정답에 해당하는 아웃풋 값(text)이 context 안에 있다는 것입니다. 참고로 answer_start는 무시하셔도 됩니다. 왜냐하면 answer_start는 context 내에서 단어를 쪼갠 다음 쪼갠 것을 하나 하나 세어서 몇번째에 정답이 위치하는지를 알려주는 것입니다. 예를 들자면, context를 abcdefg라고 가정했을시 e가 정답(text)이라면, answer_start는 5가 됩니다. \n",
        "본 SQAUD 문제에서는 단어를 쪼갠 것을 하나 하나의 위치를 예측하는 것이 아니라, 단어의 시작 위치와 끝 위치를 예측하는 것이기 때문에 answer_start를 무시하셔도 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "waSYuY0kMfW5",
        "colab_type": "code",
        "outputId": "c210534c-1153-4767-e6e5-5b7b8bfc5fba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 904
        }
      },
      "source": [
        "train"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>index</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answer_start</th>\n",
              "      <th>text</th>\n",
              "      <th>c_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5733be284776f41900661182</td>\n",
              "      <td>To whom did the Virgin Mary allegedly appear i...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>515</td>\n",
              "      <td>Saint Bernadette Soubirous</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5733be284776f4190066117f</td>\n",
              "      <td>What is in front of the Notre Dame Main Building?</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>188</td>\n",
              "      <td>a copper statue of Christ</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5733be284776f41900661180</td>\n",
              "      <td>The Basilica of the Sacred heart at Notre Dame...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>279</td>\n",
              "      <td>the Main Building</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5733be284776f41900661181</td>\n",
              "      <td>What is the Grotto at Notre Dame?</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>381</td>\n",
              "      <td>a Marian place of prayer and reflection</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5733be284776f4190066117e</td>\n",
              "      <td>What sits on top of the Main Building at Notre...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>92</td>\n",
              "      <td>a golden statue of the Virgin Mary</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87594</th>\n",
              "      <td>5735d259012e2f140011a09d</td>\n",
              "      <td>In what US state did Kathmandu first establish...</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>229</td>\n",
              "      <td>Oregon</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87595</th>\n",
              "      <td>5735d259012e2f140011a09e</td>\n",
              "      <td>What was Yangon previously known as?</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>414</td>\n",
              "      <td>Rangoon</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87596</th>\n",
              "      <td>5735d259012e2f140011a09f</td>\n",
              "      <td>With what Belorussian city does Kathmandu have...</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>476</td>\n",
              "      <td>Minsk</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87597</th>\n",
              "      <td>5735d259012e2f140011a0a0</td>\n",
              "      <td>In what year did Kathmandu create its initial ...</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>199</td>\n",
              "      <td>1975</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>87598</th>\n",
              "      <td>5735d259012e2f140011a0a1</td>\n",
              "      <td>What is KMC an initialism of?</td>\n",
              "      <td>Kathmandu Metropolitan City (KMC), in order to...</td>\n",
              "      <td>0</td>\n",
              "      <td>Kathmandu Metropolitan City</td>\n",
              "      <td>18890</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>87599 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                          index  ...   c_id\n",
              "0      5733be284776f41900661182  ...      0\n",
              "1      5733be284776f4190066117f  ...      0\n",
              "2      5733be284776f41900661180  ...      0\n",
              "3      5733be284776f41900661181  ...      0\n",
              "4      5733be284776f4190066117e  ...      0\n",
              "...                         ...  ...    ...\n",
              "87594  5735d259012e2f140011a09d  ...  18890\n",
              "87595  5735d259012e2f140011a09e  ...  18890\n",
              "87596  5735d259012e2f140011a09f  ...  18890\n",
              "87597  5735d259012e2f140011a0a0  ...  18890\n",
              "87598  5735d259012e2f140011a0a1  ...  18890\n",
              "\n",
              "[87599 rows x 6 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKDO3exVMU1J",
        "colab_type": "text"
      },
      "source": [
        "문장의 길이를 정의하는 칼럼을 만듭니다.  \n",
        "그리고 질문의 길이가 10 미만인 row는 분석에 용이하지 않은 질문들이므로  삭제하도록 합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3hNLPFxCzGYv",
        "colab_type": "code",
        "outputId": "95da4db3-19fa-45b5-d139-8063057a4895",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 946
        }
      },
      "source": [
        "# context의 길이를 알려주는 칼럼 생성\n",
        "train['context_len'] = train['context'].str.len()\n",
        "# 질문의 길이가 10 미만인 데이터 확인\n",
        "# 질문의 길이가 10 미만이면 이상한 데이터일 가능성 높음\n",
        "train.loc[train['question'].str.len() <= 10].head(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>index</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answer_start</th>\n",
              "      <th>text</th>\n",
              "      <th>c_id</th>\n",
              "      <th>context_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>16818</th>\n",
              "      <td>56e6eb396fe0821900b8ec24</td>\n",
              "      <td>k</td>\n",
              "      <td>Nanjing ( listen; Chinese: 南京, \"Southern Capit...</td>\n",
              "      <td>232</td>\n",
              "      <td>ks</td>\n",
              "      <td>3731</td>\n",
              "      <td>1621</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16819</th>\n",
              "      <td>56e6eb396fe0821900b8ec25</td>\n",
              "      <td>j</td>\n",
              "      <td>Nanjing ( listen; Chinese: 南京, \"Southern Capit...</td>\n",
              "      <td>18</td>\n",
              "      <td>Ch</td>\n",
              "      <td>3731</td>\n",
              "      <td>1621</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16820</th>\n",
              "      <td>56e6eb396fe0821900b8ec26</td>\n",
              "      <td>n</td>\n",
              "      <td>Nanjing ( listen; Chinese: 南京, \"Southern Capit...</td>\n",
              "      <td>2</td>\n",
              "      <td>n</td>\n",
              "      <td>3731</td>\n",
              "      <td>1621</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16821</th>\n",
              "      <td>56e6eb396fe0821900b8ec27</td>\n",
              "      <td>b</td>\n",
              "      <td>Nanjing ( listen; Chinese: 南京, \"Southern Capit...</td>\n",
              "      <td>144</td>\n",
              "      <td>b</td>\n",
              "      <td>3731</td>\n",
              "      <td>1621</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16822</th>\n",
              "      <td>56e6eb396fe0821900b8ec28</td>\n",
              "      <td>v</td>\n",
              "      <td>Nanjing ( listen; Chinese: 南京, \"Southern Capit...</td>\n",
              "      <td>108</td>\n",
              "      <td>v</td>\n",
              "      <td>3731</td>\n",
              "      <td>1621</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38422</th>\n",
              "      <td>57262473271a42140099d4e9</td>\n",
              "      <td>dd</td>\n",
              "      <td>The Hellenistic period covers the period of an...</td>\n",
              "      <td>1102</td>\n",
              "      <td>yptian Se</td>\n",
              "      <td>8322</td>\n",
              "      <td>1194</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38423</th>\n",
              "      <td>57262473271a42140099d4ea</td>\n",
              "      <td>dd</td>\n",
              "      <td>The Hellenistic period covers the period of an...</td>\n",
              "      <td>1185</td>\n",
              "      <td>Buddh</td>\n",
              "      <td>8322</td>\n",
              "      <td>1194</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38424</th>\n",
              "      <td>57262473271a42140099d4eb</td>\n",
              "      <td>dd</td>\n",
              "      <td>The Hellenistic period covers the period of an...</td>\n",
              "      <td>915</td>\n",
              "      <td>m and E</td>\n",
              "      <td>8322</td>\n",
              "      <td>1194</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38425</th>\n",
              "      <td>57262473271a42140099d4ec</td>\n",
              "      <td>dd</td>\n",
              "      <td>The Hellenistic period covers the period of an...</td>\n",
              "      <td>1185</td>\n",
              "      <td>Buddhism</td>\n",
              "      <td>8322</td>\n",
              "      <td>1194</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38426</th>\n",
              "      <td>57262473271a42140099d4ed</td>\n",
              "      <td>d</td>\n",
              "      <td>The Hellenistic period covers the period of an...</td>\n",
              "      <td>1090</td>\n",
              "      <td>the Gre</td>\n",
              "      <td>8322</td>\n",
              "      <td>1194</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                          index question  ...  c_id  context_len\n",
              "16818  56e6eb396fe0821900b8ec24        k  ...  3731         1621\n",
              "16819  56e6eb396fe0821900b8ec25        j  ...  3731         1621\n",
              "16820  56e6eb396fe0821900b8ec26        n  ...  3731         1621\n",
              "16821  56e6eb396fe0821900b8ec27        b  ...  3731         1621\n",
              "16822  56e6eb396fe0821900b8ec28        v  ...  3731         1621\n",
              "38422  57262473271a42140099d4e9       dd  ...  8322         1194\n",
              "38423  57262473271a42140099d4ea       dd  ...  8322         1194\n",
              "38424  57262473271a42140099d4eb       dd  ...  8322         1194\n",
              "38425  57262473271a42140099d4ec       dd  ...  8322         1194\n",
              "38426  57262473271a42140099d4ed        d  ...  8322         1194\n",
              "\n",
              "[10 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T3iapMr7c6M3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 질문의 길이가 10 미만인 데이터 삭제\n",
        "train = train.loc[train['question'].str.len() >= 10].reset_index(drop=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2-NebUQmG1w-",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "- bert 훈련을 위한 사전 설정을 합니다. SEQ_LEN은 문장의 최대 길이입니다. SEQ_LEN 보다 문장의 길이가 작다면 남은 부분은 0이 채워지고, 만약에 SEQ_LEN보다 문장 길이가 길다면 SEQ_LEN을 초과하는 부분이 잘리게 됩니다.  \n",
        "본 문제에서는 메모리 문제 등으로 384로 정했습니다.\n",
        "- BATCH_SIZE는 메모리 초과 같은 문제를 방지하기 위해 작은 수인 10으로 정했습니다. 그리고 총 훈련 에포크 수는 2로 정했습니다. 학습율(LR;Learning rate)은 3e-5로 작게 정했습니다.\n",
        "- pretrained_path는 bert 사전학습 모형이 있는 폴더를 의미합니다.\n",
        "- 그리고 우리가 분석할 문장이 들어있는 칼럼의 제목인 document와 긍정인지 부정인지 알려주는 칼럼을 label로 정해줍니다\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tLYWgZJAtsku",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "SEQ_LEN = 384 \n",
        "#버트 인풋으로 들어갈 문장의 길이, 즉 Question + Context의 길이를 뜻함\n",
        "#이번 문제에서는 384로 한정\n",
        "\n",
        "BATCH_SIZE = 12\n",
        "#한번에 훈련할 배치 크기를 12로 하여, 12개씩 묶음으로 훈련\n",
        "EPOCHS=2 \n",
        "#에폭 수는 2바퀴로 하여, 총 데이터를 2번 훈련시킴\n",
        "LR=1.5e-5 \n",
        "#학습율(Learning Rate)을 1.5e-5로 작게 설정, 파인튜닝이기 때문에 작게 설정해도 잘 돌아감\n",
        "\n",
        "\n",
        "pretrained_path =\"bert\"\n",
        "# 구글에서 다운받은 Bert 모형이 들어가 있는 폴더\n",
        "config_path = os.path.join(pretrained_path, 'bert_config.json')\n",
        "# 버트 레이어에 관한 설정bert_config.json\n",
        "\n",
        "# {\n",
        "#  \"attention_probs_dropout_prob\": 0.1, \n",
        "#  \"directionality\": \"bidi\", \n",
        "#  \"hidden_act\": \"gelu\", \n",
        "#  \"hidden_dropout_prob\": 0.1, \n",
        "#  \"hidden_size\": 768, \n",
        "#  \"initializer_range\": 0.02, \n",
        "#  \"intermediate_size\": 3072, \n",
        "#  \"max_position_embeddings\": 512, \n",
        "#  \"num_attention_heads\": 12, \n",
        "#  \"num_hidden_layers\": 12, \n",
        "#  \"pooler_fc_size\": 768, \n",
        "#  \"pooler_num_attention_heads\": 12, \n",
        "#  \"pooler_num_fc_layers\": 3, \n",
        "#  \"pooler_size_per_head\": 128, \n",
        "#  \"pooler_type\": \"first_token_transform\", \n",
        "#  \"type_vocab_size\": 2, \n",
        "#  \"vocab_size\": 119547\n",
        "# }\n",
        "\n",
        "\n",
        "checkpoint_path = os.path.join(pretrained_path, 'bert_model.ckpt')\n",
        "# 구글에서 다운받은 버트 모델이 있는 폴더\n",
        "vocab_path = os.path.join(pretrained_path, 'vocab.txt')\n",
        "# 버트에 쓸 어휘 뭉치들을 저장하고 있는 파일\n",
        "\n",
        "DATA_COLUMN = \"context\"\n",
        "# context를 포함하고 있는 열의 이름\n",
        "QUESTION_COLUMN = \"question\"\n",
        "# question을 포함하고 있는 열의 이름\n",
        "TEXT = \"text\"\n",
        "# text(정답)을 포함하고 있는 열의 이름"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IaU-13a9H4YJ",
        "colab_type": "text"
      },
      "source": [
        "vocab.txt에 있는 단어에 인덱스를 추가해주는 token_dict라는 딕셔너리를 생성합니다.  \n",
        "우리가 분석할 문장이 토큰화가 되고, 그 다음에는 인덱스(숫자)로 변경되어서 버트 신경망에 인풋으로 들어게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-ph2YQXg2iz5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 단어마다 인덱스를 매겨주는 딕셔너리 생성\n",
        "token_dict = {}\n",
        "with codecs.open(vocab_path, 'r', 'utf8') as reader:\n",
        "    for line in reader:\n",
        "        token = line.strip()\n",
        "        token_dict[token] = len(token_dict)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YPlFRTeIIiJi",
        "colab_type": "code",
        "outputId": "64a1f9da-f9ca-493f-a781-6c24bb69172f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# 단어에 대응하는 숫자가 넘버링 된 것을 확인 가능\n",
        "token_dict"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'[PAD]': 0,\n",
              " '[unused0]': 1,\n",
              " '[unused1]': 2,\n",
              " '[unused2]': 3,\n",
              " '[unused3]': 4,\n",
              " '[unused4]': 5,\n",
              " '[unused5]': 6,\n",
              " '[unused6]': 7,\n",
              " '[unused7]': 8,\n",
              " '[unused8]': 9,\n",
              " '[unused9]': 10,\n",
              " '[unused10]': 11,\n",
              " '[unused11]': 12,\n",
              " '[unused12]': 13,\n",
              " '[unused13]': 14,\n",
              " '[unused14]': 15,\n",
              " '[unused15]': 16,\n",
              " '[unused16]': 17,\n",
              " '[unused17]': 18,\n",
              " '[unused18]': 19,\n",
              " '[unused19]': 20,\n",
              " '[unused20]': 21,\n",
              " '[unused21]': 22,\n",
              " '[unused22]': 23,\n",
              " '[unused23]': 24,\n",
              " '[unused24]': 25,\n",
              " '[unused25]': 26,\n",
              " '[unused26]': 27,\n",
              " '[unused27]': 28,\n",
              " '[unused28]': 29,\n",
              " '[unused29]': 30,\n",
              " '[unused30]': 31,\n",
              " '[unused31]': 32,\n",
              " '[unused32]': 33,\n",
              " '[unused33]': 34,\n",
              " '[unused34]': 35,\n",
              " '[unused35]': 36,\n",
              " '[unused36]': 37,\n",
              " '[unused37]': 38,\n",
              " '[unused38]': 39,\n",
              " '[unused39]': 40,\n",
              " '[unused40]': 41,\n",
              " '[unused41]': 42,\n",
              " '[unused42]': 43,\n",
              " '[unused43]': 44,\n",
              " '[unused44]': 45,\n",
              " '[unused45]': 46,\n",
              " '[unused46]': 47,\n",
              " '[unused47]': 48,\n",
              " '[unused48]': 49,\n",
              " '[unused49]': 50,\n",
              " '[unused50]': 51,\n",
              " '[unused51]': 52,\n",
              " '[unused52]': 53,\n",
              " '[unused53]': 54,\n",
              " '[unused54]': 55,\n",
              " '[unused55]': 56,\n",
              " '[unused56]': 57,\n",
              " '[unused57]': 58,\n",
              " '[unused58]': 59,\n",
              " '[unused59]': 60,\n",
              " '[unused60]': 61,\n",
              " '[unused61]': 62,\n",
              " '[unused62]': 63,\n",
              " '[unused63]': 64,\n",
              " '[unused64]': 65,\n",
              " '[unused65]': 66,\n",
              " '[unused66]': 67,\n",
              " '[unused67]': 68,\n",
              " '[unused68]': 69,\n",
              " '[unused69]': 70,\n",
              " '[unused70]': 71,\n",
              " '[unused71]': 72,\n",
              " '[unused72]': 73,\n",
              " '[unused73]': 74,\n",
              " '[unused74]': 75,\n",
              " '[unused75]': 76,\n",
              " '[unused76]': 77,\n",
              " '[unused77]': 78,\n",
              " '[unused78]': 79,\n",
              " '[unused79]': 80,\n",
              " '[unused80]': 81,\n",
              " '[unused81]': 82,\n",
              " '[unused82]': 83,\n",
              " '[unused83]': 84,\n",
              " '[unused84]': 85,\n",
              " '[unused85]': 86,\n",
              " '[unused86]': 87,\n",
              " '[unused87]': 88,\n",
              " '[unused88]': 89,\n",
              " '[unused89]': 90,\n",
              " '[unused90]': 91,\n",
              " '[unused91]': 92,\n",
              " '[unused92]': 93,\n",
              " '[unused93]': 94,\n",
              " '[unused94]': 95,\n",
              " '[unused95]': 96,\n",
              " '[unused96]': 97,\n",
              " '[unused97]': 98,\n",
              " '[unused98]': 99,\n",
              " '[UNK]': 100,\n",
              " '[CLS]': 101,\n",
              " '[SEP]': 102,\n",
              " '[MASK]': 103,\n",
              " '[unused99]': 104,\n",
              " '[unused100]': 105,\n",
              " '[unused101]': 106,\n",
              " '[unused102]': 107,\n",
              " '[unused103]': 108,\n",
              " '[unused104]': 109,\n",
              " '[unused105]': 110,\n",
              " '[unused106]': 111,\n",
              " '[unused107]': 112,\n",
              " '[unused108]': 113,\n",
              " '[unused109]': 114,\n",
              " '[unused110]': 115,\n",
              " '[unused111]': 116,\n",
              " '[unused112]': 117,\n",
              " '[unused113]': 118,\n",
              " '[unused114]': 119,\n",
              " '[unused115]': 120,\n",
              " '[unused116]': 121,\n",
              " '[unused117]': 122,\n",
              " '[unused118]': 123,\n",
              " '[unused119]': 124,\n",
              " '[unused120]': 125,\n",
              " '[unused121]': 126,\n",
              " '[unused122]': 127,\n",
              " '[unused123]': 128,\n",
              " '[unused124]': 129,\n",
              " '[unused125]': 130,\n",
              " '[unused126]': 131,\n",
              " '[unused127]': 132,\n",
              " '[unused128]': 133,\n",
              " '[unused129]': 134,\n",
              " '[unused130]': 135,\n",
              " '[unused131]': 136,\n",
              " '[unused132]': 137,\n",
              " '[unused133]': 138,\n",
              " '[unused134]': 139,\n",
              " '[unused135]': 140,\n",
              " '[unused136]': 141,\n",
              " '[unused137]': 142,\n",
              " '[unused138]': 143,\n",
              " '[unused139]': 144,\n",
              " '[unused140]': 145,\n",
              " '[unused141]': 146,\n",
              " '[unused142]': 147,\n",
              " '[unused143]': 148,\n",
              " '[unused144]': 149,\n",
              " '[unused145]': 150,\n",
              " '[unused146]': 151,\n",
              " '[unused147]': 152,\n",
              " '[unused148]': 153,\n",
              " '[unused149]': 154,\n",
              " '[unused150]': 155,\n",
              " '[unused151]': 156,\n",
              " '[unused152]': 157,\n",
              " '[unused153]': 158,\n",
              " '[unused154]': 159,\n",
              " '[unused155]': 160,\n",
              " '[unused156]': 161,\n",
              " '[unused157]': 162,\n",
              " '[unused158]': 163,\n",
              " '[unused159]': 164,\n",
              " '[unused160]': 165,\n",
              " '[unused161]': 166,\n",
              " '[unused162]': 167,\n",
              " '[unused163]': 168,\n",
              " '[unused164]': 169,\n",
              " '[unused165]': 170,\n",
              " '[unused166]': 171,\n",
              " '[unused167]': 172,\n",
              " '[unused168]': 173,\n",
              " '[unused169]': 174,\n",
              " '[unused170]': 175,\n",
              " '[unused171]': 176,\n",
              " '[unused172]': 177,\n",
              " '[unused173]': 178,\n",
              " '[unused174]': 179,\n",
              " '[unused175]': 180,\n",
              " '[unused176]': 181,\n",
              " '[unused177]': 182,\n",
              " '[unused178]': 183,\n",
              " '[unused179]': 184,\n",
              " '[unused180]': 185,\n",
              " '[unused181]': 186,\n",
              " '[unused182]': 187,\n",
              " '[unused183]': 188,\n",
              " '[unused184]': 189,\n",
              " '[unused185]': 190,\n",
              " '[unused186]': 191,\n",
              " '[unused187]': 192,\n",
              " '[unused188]': 193,\n",
              " '[unused189]': 194,\n",
              " '[unused190]': 195,\n",
              " '[unused191]': 196,\n",
              " '[unused192]': 197,\n",
              " '[unused193]': 198,\n",
              " '[unused194]': 199,\n",
              " '[unused195]': 200,\n",
              " '[unused196]': 201,\n",
              " '[unused197]': 202,\n",
              " '[unused198]': 203,\n",
              " '[unused199]': 204,\n",
              " '[unused200]': 205,\n",
              " '[unused201]': 206,\n",
              " '[unused202]': 207,\n",
              " '[unused203]': 208,\n",
              " '[unused204]': 209,\n",
              " '[unused205]': 210,\n",
              " '[unused206]': 211,\n",
              " '[unused207]': 212,\n",
              " '[unused208]': 213,\n",
              " '[unused209]': 214,\n",
              " '[unused210]': 215,\n",
              " '[unused211]': 216,\n",
              " '[unused212]': 217,\n",
              " '[unused213]': 218,\n",
              " '[unused214]': 219,\n",
              " '[unused215]': 220,\n",
              " '[unused216]': 221,\n",
              " '[unused217]': 222,\n",
              " '[unused218]': 223,\n",
              " '[unused219]': 224,\n",
              " '[unused220]': 225,\n",
              " '[unused221]': 226,\n",
              " '[unused222]': 227,\n",
              " '[unused223]': 228,\n",
              " '[unused224]': 229,\n",
              " '[unused225]': 230,\n",
              " '[unused226]': 231,\n",
              " '[unused227]': 232,\n",
              " '[unused228]': 233,\n",
              " '[unused229]': 234,\n",
              " '[unused230]': 235,\n",
              " '[unused231]': 236,\n",
              " '[unused232]': 237,\n",
              " '[unused233]': 238,\n",
              " '[unused234]': 239,\n",
              " '[unused235]': 240,\n",
              " '[unused236]': 241,\n",
              " '[unused237]': 242,\n",
              " '[unused238]': 243,\n",
              " '[unused239]': 244,\n",
              " '[unused240]': 245,\n",
              " '[unused241]': 246,\n",
              " '[unused242]': 247,\n",
              " '[unused243]': 248,\n",
              " '[unused244]': 249,\n",
              " '[unused245]': 250,\n",
              " '[unused246]': 251,\n",
              " '[unused247]': 252,\n",
              " '[unused248]': 253,\n",
              " '[unused249]': 254,\n",
              " '[unused250]': 255,\n",
              " '[unused251]': 256,\n",
              " '[unused252]': 257,\n",
              " '[unused253]': 258,\n",
              " '[unused254]': 259,\n",
              " '[unused255]': 260,\n",
              " '[unused256]': 261,\n",
              " '[unused257]': 262,\n",
              " '[unused258]': 263,\n",
              " '[unused259]': 264,\n",
              " '[unused260]': 265,\n",
              " '[unused261]': 266,\n",
              " '[unused262]': 267,\n",
              " '[unused263]': 268,\n",
              " '[unused264]': 269,\n",
              " '[unused265]': 270,\n",
              " '[unused266]': 271,\n",
              " '[unused267]': 272,\n",
              " '[unused268]': 273,\n",
              " '[unused269]': 274,\n",
              " '[unused270]': 275,\n",
              " '[unused271]': 276,\n",
              " '[unused272]': 277,\n",
              " '[unused273]': 278,\n",
              " '[unused274]': 279,\n",
              " '[unused275]': 280,\n",
              " '[unused276]': 281,\n",
              " '[unused277]': 282,\n",
              " '[unused278]': 283,\n",
              " '[unused279]': 284,\n",
              " '[unused280]': 285,\n",
              " '[unused281]': 286,\n",
              " '[unused282]': 287,\n",
              " '[unused283]': 288,\n",
              " '[unused284]': 289,\n",
              " '[unused285]': 290,\n",
              " '[unused286]': 291,\n",
              " '[unused287]': 292,\n",
              " '[unused288]': 293,\n",
              " '[unused289]': 294,\n",
              " '[unused290]': 295,\n",
              " '[unused291]': 296,\n",
              " '[unused292]': 297,\n",
              " '[unused293]': 298,\n",
              " '[unused294]': 299,\n",
              " '[unused295]': 300,\n",
              " '[unused296]': 301,\n",
              " '[unused297]': 302,\n",
              " '[unused298]': 303,\n",
              " '[unused299]': 304,\n",
              " '[unused300]': 305,\n",
              " '[unused301]': 306,\n",
              " '[unused302]': 307,\n",
              " '[unused303]': 308,\n",
              " '[unused304]': 309,\n",
              " '[unused305]': 310,\n",
              " '[unused306]': 311,\n",
              " '[unused307]': 312,\n",
              " '[unused308]': 313,\n",
              " '[unused309]': 314,\n",
              " '[unused310]': 315,\n",
              " '[unused311]': 316,\n",
              " '[unused312]': 317,\n",
              " '[unused313]': 318,\n",
              " '[unused314]': 319,\n",
              " '[unused315]': 320,\n",
              " '[unused316]': 321,\n",
              " '[unused317]': 322,\n",
              " '[unused318]': 323,\n",
              " '[unused319]': 324,\n",
              " '[unused320]': 325,\n",
              " '[unused321]': 326,\n",
              " '[unused322]': 327,\n",
              " '[unused323]': 328,\n",
              " '[unused324]': 329,\n",
              " '[unused325]': 330,\n",
              " '[unused326]': 331,\n",
              " '[unused327]': 332,\n",
              " '[unused328]': 333,\n",
              " '[unused329]': 334,\n",
              " '[unused330]': 335,\n",
              " '[unused331]': 336,\n",
              " '[unused332]': 337,\n",
              " '[unused333]': 338,\n",
              " '[unused334]': 339,\n",
              " '[unused335]': 340,\n",
              " '[unused336]': 341,\n",
              " '[unused337]': 342,\n",
              " '[unused338]': 343,\n",
              " '[unused339]': 344,\n",
              " '[unused340]': 345,\n",
              " '[unused341]': 346,\n",
              " '[unused342]': 347,\n",
              " '[unused343]': 348,\n",
              " '[unused344]': 349,\n",
              " '[unused345]': 350,\n",
              " '[unused346]': 351,\n",
              " '[unused347]': 352,\n",
              " '[unused348]': 353,\n",
              " '[unused349]': 354,\n",
              " '[unused350]': 355,\n",
              " '[unused351]': 356,\n",
              " '[unused352]': 357,\n",
              " '[unused353]': 358,\n",
              " '[unused354]': 359,\n",
              " '[unused355]': 360,\n",
              " '[unused356]': 361,\n",
              " '[unused357]': 362,\n",
              " '[unused358]': 363,\n",
              " '[unused359]': 364,\n",
              " '[unused360]': 365,\n",
              " '[unused361]': 366,\n",
              " '[unused362]': 367,\n",
              " '[unused363]': 368,\n",
              " '[unused364]': 369,\n",
              " '[unused365]': 370,\n",
              " '[unused366]': 371,\n",
              " '[unused367]': 372,\n",
              " '[unused368]': 373,\n",
              " '[unused369]': 374,\n",
              " '[unused370]': 375,\n",
              " '[unused371]': 376,\n",
              " '[unused372]': 377,\n",
              " '[unused373]': 378,\n",
              " '[unused374]': 379,\n",
              " '[unused375]': 380,\n",
              " '[unused376]': 381,\n",
              " '[unused377]': 382,\n",
              " '[unused378]': 383,\n",
              " '[unused379]': 384,\n",
              " '[unused380]': 385,\n",
              " '[unused381]': 386,\n",
              " '[unused382]': 387,\n",
              " '[unused383]': 388,\n",
              " '[unused384]': 389,\n",
              " '[unused385]': 390,\n",
              " '[unused386]': 391,\n",
              " '[unused387]': 392,\n",
              " '[unused388]': 393,\n",
              " '[unused389]': 394,\n",
              " '[unused390]': 395,\n",
              " '[unused391]': 396,\n",
              " '[unused392]': 397,\n",
              " '[unused393]': 398,\n",
              " '[unused394]': 399,\n",
              " '[unused395]': 400,\n",
              " '[unused396]': 401,\n",
              " '[unused397]': 402,\n",
              " '[unused398]': 403,\n",
              " '[unused399]': 404,\n",
              " '[unused400]': 405,\n",
              " '[unused401]': 406,\n",
              " '[unused402]': 407,\n",
              " '[unused403]': 408,\n",
              " '[unused404]': 409,\n",
              " '[unused405]': 410,\n",
              " '[unused406]': 411,\n",
              " '[unused407]': 412,\n",
              " '[unused408]': 413,\n",
              " '[unused409]': 414,\n",
              " '[unused410]': 415,\n",
              " '[unused411]': 416,\n",
              " '[unused412]': 417,\n",
              " '[unused413]': 418,\n",
              " '[unused414]': 419,\n",
              " '[unused415]': 420,\n",
              " '[unused416]': 421,\n",
              " '[unused417]': 422,\n",
              " '[unused418]': 423,\n",
              " '[unused419]': 424,\n",
              " '[unused420]': 425,\n",
              " '[unused421]': 426,\n",
              " '[unused422]': 427,\n",
              " '[unused423]': 428,\n",
              " '[unused424]': 429,\n",
              " '[unused425]': 430,\n",
              " '[unused426]': 431,\n",
              " '[unused427]': 432,\n",
              " '[unused428]': 433,\n",
              " '[unused429]': 434,\n",
              " '[unused430]': 435,\n",
              " '[unused431]': 436,\n",
              " '[unused432]': 437,\n",
              " '[unused433]': 438,\n",
              " '[unused434]': 439,\n",
              " '[unused435]': 440,\n",
              " '[unused436]': 441,\n",
              " '[unused437]': 442,\n",
              " '[unused438]': 443,\n",
              " '[unused439]': 444,\n",
              " '[unused440]': 445,\n",
              " '[unused441]': 446,\n",
              " '[unused442]': 447,\n",
              " '[unused443]': 448,\n",
              " '[unused444]': 449,\n",
              " '[unused445]': 450,\n",
              " '[unused446]': 451,\n",
              " '[unused447]': 452,\n",
              " '[unused448]': 453,\n",
              " '[unused449]': 454,\n",
              " '[unused450]': 455,\n",
              " '[unused451]': 456,\n",
              " '[unused452]': 457,\n",
              " '[unused453]': 458,\n",
              " '[unused454]': 459,\n",
              " '[unused455]': 460,\n",
              " '[unused456]': 461,\n",
              " '[unused457]': 462,\n",
              " '[unused458]': 463,\n",
              " '[unused459]': 464,\n",
              " '[unused460]': 465,\n",
              " '[unused461]': 466,\n",
              " '[unused462]': 467,\n",
              " '[unused463]': 468,\n",
              " '[unused464]': 469,\n",
              " '[unused465]': 470,\n",
              " '[unused466]': 471,\n",
              " '[unused467]': 472,\n",
              " '[unused468]': 473,\n",
              " '[unused469]': 474,\n",
              " '[unused470]': 475,\n",
              " '[unused471]': 476,\n",
              " '[unused472]': 477,\n",
              " '[unused473]': 478,\n",
              " '[unused474]': 479,\n",
              " '[unused475]': 480,\n",
              " '[unused476]': 481,\n",
              " '[unused477]': 482,\n",
              " '[unused478]': 483,\n",
              " '[unused479]': 484,\n",
              " '[unused480]': 485,\n",
              " '[unused481]': 486,\n",
              " '[unused482]': 487,\n",
              " '[unused483]': 488,\n",
              " '[unused484]': 489,\n",
              " '[unused485]': 490,\n",
              " '[unused486]': 491,\n",
              " '[unused487]': 492,\n",
              " '[unused488]': 493,\n",
              " '[unused489]': 494,\n",
              " '[unused490]': 495,\n",
              " '[unused491]': 496,\n",
              " '[unused492]': 497,\n",
              " '[unused493]': 498,\n",
              " '[unused494]': 499,\n",
              " '[unused495]': 500,\n",
              " '[unused496]': 501,\n",
              " '[unused497]': 502,\n",
              " '[unused498]': 503,\n",
              " '[unused499]': 504,\n",
              " '[unused500]': 505,\n",
              " '[unused501]': 506,\n",
              " '[unused502]': 507,\n",
              " '[unused503]': 508,\n",
              " '[unused504]': 509,\n",
              " '[unused505]': 510,\n",
              " '[unused506]': 511,\n",
              " '[unused507]': 512,\n",
              " '[unused508]': 513,\n",
              " '[unused509]': 514,\n",
              " '[unused510]': 515,\n",
              " '[unused511]': 516,\n",
              " '[unused512]': 517,\n",
              " '[unused513]': 518,\n",
              " '[unused514]': 519,\n",
              " '[unused515]': 520,\n",
              " '[unused516]': 521,\n",
              " '[unused517]': 522,\n",
              " '[unused518]': 523,\n",
              " '[unused519]': 524,\n",
              " '[unused520]': 525,\n",
              " '[unused521]': 526,\n",
              " '[unused522]': 527,\n",
              " '[unused523]': 528,\n",
              " '[unused524]': 529,\n",
              " '[unused525]': 530,\n",
              " '[unused526]': 531,\n",
              " '[unused527]': 532,\n",
              " '[unused528]': 533,\n",
              " '[unused529]': 534,\n",
              " '[unused530]': 535,\n",
              " '[unused531]': 536,\n",
              " '[unused532]': 537,\n",
              " '[unused533]': 538,\n",
              " '[unused534]': 539,\n",
              " '[unused535]': 540,\n",
              " '[unused536]': 541,\n",
              " '[unused537]': 542,\n",
              " '[unused538]': 543,\n",
              " '[unused539]': 544,\n",
              " '[unused540]': 545,\n",
              " '[unused541]': 546,\n",
              " '[unused542]': 547,\n",
              " '[unused543]': 548,\n",
              " '[unused544]': 549,\n",
              " '[unused545]': 550,\n",
              " '[unused546]': 551,\n",
              " '[unused547]': 552,\n",
              " '[unused548]': 553,\n",
              " '[unused549]': 554,\n",
              " '[unused550]': 555,\n",
              " '[unused551]': 556,\n",
              " '[unused552]': 557,\n",
              " '[unused553]': 558,\n",
              " '[unused554]': 559,\n",
              " '[unused555]': 560,\n",
              " '[unused556]': 561,\n",
              " '[unused557]': 562,\n",
              " '[unused558]': 563,\n",
              " '[unused559]': 564,\n",
              " '[unused560]': 565,\n",
              " '[unused561]': 566,\n",
              " '[unused562]': 567,\n",
              " '[unused563]': 568,\n",
              " '[unused564]': 569,\n",
              " '[unused565]': 570,\n",
              " '[unused566]': 571,\n",
              " '[unused567]': 572,\n",
              " '[unused568]': 573,\n",
              " '[unused569]': 574,\n",
              " '[unused570]': 575,\n",
              " '[unused571]': 576,\n",
              " '[unused572]': 577,\n",
              " '[unused573]': 578,\n",
              " '[unused574]': 579,\n",
              " '[unused575]': 580,\n",
              " '[unused576]': 581,\n",
              " '[unused577]': 582,\n",
              " '[unused578]': 583,\n",
              " '[unused579]': 584,\n",
              " '[unused580]': 585,\n",
              " '[unused581]': 586,\n",
              " '[unused582]': 587,\n",
              " '[unused583]': 588,\n",
              " '[unused584]': 589,\n",
              " '[unused585]': 590,\n",
              " '[unused586]': 591,\n",
              " '[unused587]': 592,\n",
              " '[unused588]': 593,\n",
              " '[unused589]': 594,\n",
              " '[unused590]': 595,\n",
              " '[unused591]': 596,\n",
              " '[unused592]': 597,\n",
              " '[unused593]': 598,\n",
              " '[unused594]': 599,\n",
              " '[unused595]': 600,\n",
              " '[unused596]': 601,\n",
              " '[unused597]': 602,\n",
              " '[unused598]': 603,\n",
              " '[unused599]': 604,\n",
              " '[unused600]': 605,\n",
              " '[unused601]': 606,\n",
              " '[unused602]': 607,\n",
              " '[unused603]': 608,\n",
              " '[unused604]': 609,\n",
              " '[unused605]': 610,\n",
              " '[unused606]': 611,\n",
              " '[unused607]': 612,\n",
              " '[unused608]': 613,\n",
              " '[unused609]': 614,\n",
              " '[unused610]': 615,\n",
              " '[unused611]': 616,\n",
              " '[unused612]': 617,\n",
              " '[unused613]': 618,\n",
              " '[unused614]': 619,\n",
              " '[unused615]': 620,\n",
              " '[unused616]': 621,\n",
              " '[unused617]': 622,\n",
              " '[unused618]': 623,\n",
              " '[unused619]': 624,\n",
              " '[unused620]': 625,\n",
              " '[unused621]': 626,\n",
              " '[unused622]': 627,\n",
              " '[unused623]': 628,\n",
              " '[unused624]': 629,\n",
              " '[unused625]': 630,\n",
              " '[unused626]': 631,\n",
              " '[unused627]': 632,\n",
              " '[unused628]': 633,\n",
              " '[unused629]': 634,\n",
              " '[unused630]': 635,\n",
              " '[unused631]': 636,\n",
              " '[unused632]': 637,\n",
              " '[unused633]': 638,\n",
              " '[unused634]': 639,\n",
              " '[unused635]': 640,\n",
              " '[unused636]': 641,\n",
              " '[unused637]': 642,\n",
              " '[unused638]': 643,\n",
              " '[unused639]': 644,\n",
              " '[unused640]': 645,\n",
              " '[unused641]': 646,\n",
              " '[unused642]': 647,\n",
              " '[unused643]': 648,\n",
              " '[unused644]': 649,\n",
              " '[unused645]': 650,\n",
              " '[unused646]': 651,\n",
              " '[unused647]': 652,\n",
              " '[unused648]': 653,\n",
              " '[unused649]': 654,\n",
              " '[unused650]': 655,\n",
              " '[unused651]': 656,\n",
              " '[unused652]': 657,\n",
              " '[unused653]': 658,\n",
              " '[unused654]': 659,\n",
              " '[unused655]': 660,\n",
              " '[unused656]': 661,\n",
              " '[unused657]': 662,\n",
              " '[unused658]': 663,\n",
              " '[unused659]': 664,\n",
              " '[unused660]': 665,\n",
              " '[unused661]': 666,\n",
              " '[unused662]': 667,\n",
              " '[unused663]': 668,\n",
              " '[unused664]': 669,\n",
              " '[unused665]': 670,\n",
              " '[unused666]': 671,\n",
              " '[unused667]': 672,\n",
              " '[unused668]': 673,\n",
              " '[unused669]': 674,\n",
              " '[unused670]': 675,\n",
              " '[unused671]': 676,\n",
              " '[unused672]': 677,\n",
              " '[unused673]': 678,\n",
              " '[unused674]': 679,\n",
              " '[unused675]': 680,\n",
              " '[unused676]': 681,\n",
              " '[unused677]': 682,\n",
              " '[unused678]': 683,\n",
              " '[unused679]': 684,\n",
              " '[unused680]': 685,\n",
              " '[unused681]': 686,\n",
              " '[unused682]': 687,\n",
              " '[unused683]': 688,\n",
              " '[unused684]': 689,\n",
              " '[unused685]': 690,\n",
              " '[unused686]': 691,\n",
              " '[unused687]': 692,\n",
              " '[unused688]': 693,\n",
              " '[unused689]': 694,\n",
              " '[unused690]': 695,\n",
              " '[unused691]': 696,\n",
              " '[unused692]': 697,\n",
              " '[unused693]': 698,\n",
              " '[unused694]': 699,\n",
              " '[unused695]': 700,\n",
              " '[unused696]': 701,\n",
              " '[unused697]': 702,\n",
              " '[unused698]': 703,\n",
              " '[unused699]': 704,\n",
              " '[unused700]': 705,\n",
              " '[unused701]': 706,\n",
              " '[unused702]': 707,\n",
              " '[unused703]': 708,\n",
              " '[unused704]': 709,\n",
              " '[unused705]': 710,\n",
              " '[unused706]': 711,\n",
              " '[unused707]': 712,\n",
              " '[unused708]': 713,\n",
              " '[unused709]': 714,\n",
              " '[unused710]': 715,\n",
              " '[unused711]': 716,\n",
              " '[unused712]': 717,\n",
              " '[unused713]': 718,\n",
              " '[unused714]': 719,\n",
              " '[unused715]': 720,\n",
              " '[unused716]': 721,\n",
              " '[unused717]': 722,\n",
              " '[unused718]': 723,\n",
              " '[unused719]': 724,\n",
              " '[unused720]': 725,\n",
              " '[unused721]': 726,\n",
              " '[unused722]': 727,\n",
              " '[unused723]': 728,\n",
              " '[unused724]': 729,\n",
              " '[unused725]': 730,\n",
              " '[unused726]': 731,\n",
              " '[unused727]': 732,\n",
              " '[unused728]': 733,\n",
              " '[unused729]': 734,\n",
              " '[unused730]': 735,\n",
              " '[unused731]': 736,\n",
              " '[unused732]': 737,\n",
              " '[unused733]': 738,\n",
              " '[unused734]': 739,\n",
              " '[unused735]': 740,\n",
              " '[unused736]': 741,\n",
              " '[unused737]': 742,\n",
              " '[unused738]': 743,\n",
              " '[unused739]': 744,\n",
              " '[unused740]': 745,\n",
              " '[unused741]': 746,\n",
              " '[unused742]': 747,\n",
              " '[unused743]': 748,\n",
              " '[unused744]': 749,\n",
              " '[unused745]': 750,\n",
              " '[unused746]': 751,\n",
              " '[unused747]': 752,\n",
              " '[unused748]': 753,\n",
              " '[unused749]': 754,\n",
              " '[unused750]': 755,\n",
              " '[unused751]': 756,\n",
              " '[unused752]': 757,\n",
              " '[unused753]': 758,\n",
              " '[unused754]': 759,\n",
              " '[unused755]': 760,\n",
              " '[unused756]': 761,\n",
              " '[unused757]': 762,\n",
              " '[unused758]': 763,\n",
              " '[unused759]': 764,\n",
              " '[unused760]': 765,\n",
              " '[unused761]': 766,\n",
              " '[unused762]': 767,\n",
              " '[unused763]': 768,\n",
              " '[unused764]': 769,\n",
              " '[unused765]': 770,\n",
              " '[unused766]': 771,\n",
              " '[unused767]': 772,\n",
              " '[unused768]': 773,\n",
              " '[unused769]': 774,\n",
              " '[unused770]': 775,\n",
              " '[unused771]': 776,\n",
              " '[unused772]': 777,\n",
              " '[unused773]': 778,\n",
              " '[unused774]': 779,\n",
              " '[unused775]': 780,\n",
              " '[unused776]': 781,\n",
              " '[unused777]': 782,\n",
              " '[unused778]': 783,\n",
              " '[unused779]': 784,\n",
              " '[unused780]': 785,\n",
              " '[unused781]': 786,\n",
              " '[unused782]': 787,\n",
              " '[unused783]': 788,\n",
              " '[unused784]': 789,\n",
              " '[unused785]': 790,\n",
              " '[unused786]': 791,\n",
              " '[unused787]': 792,\n",
              " '[unused788]': 793,\n",
              " '[unused789]': 794,\n",
              " '[unused790]': 795,\n",
              " '[unused791]': 796,\n",
              " '[unused792]': 797,\n",
              " '[unused793]': 798,\n",
              " '[unused794]': 799,\n",
              " '[unused795]': 800,\n",
              " '[unused796]': 801,\n",
              " '[unused797]': 802,\n",
              " '[unused798]': 803,\n",
              " '[unused799]': 804,\n",
              " '[unused800]': 805,\n",
              " '[unused801]': 806,\n",
              " '[unused802]': 807,\n",
              " '[unused803]': 808,\n",
              " '[unused804]': 809,\n",
              " '[unused805]': 810,\n",
              " '[unused806]': 811,\n",
              " '[unused807]': 812,\n",
              " '[unused808]': 813,\n",
              " '[unused809]': 814,\n",
              " '[unused810]': 815,\n",
              " '[unused811]': 816,\n",
              " '[unused812]': 817,\n",
              " '[unused813]': 818,\n",
              " '[unused814]': 819,\n",
              " '[unused815]': 820,\n",
              " '[unused816]': 821,\n",
              " '[unused817]': 822,\n",
              " '[unused818]': 823,\n",
              " '[unused819]': 824,\n",
              " '[unused820]': 825,\n",
              " '[unused821]': 826,\n",
              " '[unused822]': 827,\n",
              " '[unused823]': 828,\n",
              " '[unused824]': 829,\n",
              " '[unused825]': 830,\n",
              " '[unused826]': 831,\n",
              " '[unused827]': 832,\n",
              " '[unused828]': 833,\n",
              " '[unused829]': 834,\n",
              " '[unused830]': 835,\n",
              " '[unused831]': 836,\n",
              " '[unused832]': 837,\n",
              " '[unused833]': 838,\n",
              " '[unused834]': 839,\n",
              " '[unused835]': 840,\n",
              " '[unused836]': 841,\n",
              " '[unused837]': 842,\n",
              " '[unused838]': 843,\n",
              " '[unused839]': 844,\n",
              " '[unused840]': 845,\n",
              " '[unused841]': 846,\n",
              " '[unused842]': 847,\n",
              " '[unused843]': 848,\n",
              " '[unused844]': 849,\n",
              " '[unused845]': 850,\n",
              " '[unused846]': 851,\n",
              " '[unused847]': 852,\n",
              " '[unused848]': 853,\n",
              " '[unused849]': 854,\n",
              " '[unused850]': 855,\n",
              " '[unused851]': 856,\n",
              " '[unused852]': 857,\n",
              " '[unused853]': 858,\n",
              " '[unused854]': 859,\n",
              " '[unused855]': 860,\n",
              " '[unused856]': 861,\n",
              " '[unused857]': 862,\n",
              " '[unused858]': 863,\n",
              " '[unused859]': 864,\n",
              " '[unused860]': 865,\n",
              " '[unused861]': 866,\n",
              " '[unused862]': 867,\n",
              " '[unused863]': 868,\n",
              " '[unused864]': 869,\n",
              " '[unused865]': 870,\n",
              " '[unused866]': 871,\n",
              " '[unused867]': 872,\n",
              " '[unused868]': 873,\n",
              " '[unused869]': 874,\n",
              " '[unused870]': 875,\n",
              " '[unused871]': 876,\n",
              " '[unused872]': 877,\n",
              " '[unused873]': 878,\n",
              " '[unused874]': 879,\n",
              " '[unused875]': 880,\n",
              " '[unused876]': 881,\n",
              " '[unused877]': 882,\n",
              " '[unused878]': 883,\n",
              " '[unused879]': 884,\n",
              " '[unused880]': 885,\n",
              " '[unused881]': 886,\n",
              " '[unused882]': 887,\n",
              " '[unused883]': 888,\n",
              " '[unused884]': 889,\n",
              " '[unused885]': 890,\n",
              " '[unused886]': 891,\n",
              " '[unused887]': 892,\n",
              " '[unused888]': 893,\n",
              " '[unused889]': 894,\n",
              " '[unused890]': 895,\n",
              " '[unused891]': 896,\n",
              " '[unused892]': 897,\n",
              " '[unused893]': 898,\n",
              " '[unused894]': 899,\n",
              " '[unused895]': 900,\n",
              " '[unused896]': 901,\n",
              " '[unused897]': 902,\n",
              " '[unused898]': 903,\n",
              " '[unused899]': 904,\n",
              " '[unused900]': 905,\n",
              " '[unused901]': 906,\n",
              " '[unused902]': 907,\n",
              " '[unused903]': 908,\n",
              " '[unused904]': 909,\n",
              " '[unused905]': 910,\n",
              " '[unused906]': 911,\n",
              " '[unused907]': 912,\n",
              " '[unused908]': 913,\n",
              " '[unused909]': 914,\n",
              " '[unused910]': 915,\n",
              " '[unused911]': 916,\n",
              " '[unused912]': 917,\n",
              " '[unused913]': 918,\n",
              " '[unused914]': 919,\n",
              " '[unused915]': 920,\n",
              " '[unused916]': 921,\n",
              " '[unused917]': 922,\n",
              " '[unused918]': 923,\n",
              " '[unused919]': 924,\n",
              " '[unused920]': 925,\n",
              " '[unused921]': 926,\n",
              " '[unused922]': 927,\n",
              " '[unused923]': 928,\n",
              " '[unused924]': 929,\n",
              " '[unused925]': 930,\n",
              " '[unused926]': 931,\n",
              " '[unused927]': 932,\n",
              " '[unused928]': 933,\n",
              " '[unused929]': 934,\n",
              " '[unused930]': 935,\n",
              " '[unused931]': 936,\n",
              " '[unused932]': 937,\n",
              " '[unused933]': 938,\n",
              " '[unused934]': 939,\n",
              " '[unused935]': 940,\n",
              " '[unused936]': 941,\n",
              " '[unused937]': 942,\n",
              " '[unused938]': 943,\n",
              " '[unused939]': 944,\n",
              " '[unused940]': 945,\n",
              " '[unused941]': 946,\n",
              " '[unused942]': 947,\n",
              " '[unused943]': 948,\n",
              " '[unused944]': 949,\n",
              " '[unused945]': 950,\n",
              " '[unused946]': 951,\n",
              " '[unused947]': 952,\n",
              " '[unused948]': 953,\n",
              " '[unused949]': 954,\n",
              " '[unused950]': 955,\n",
              " '[unused951]': 956,\n",
              " '[unused952]': 957,\n",
              " '[unused953]': 958,\n",
              " '[unused954]': 959,\n",
              " '[unused955]': 960,\n",
              " '[unused956]': 961,\n",
              " '[unused957]': 962,\n",
              " '[unused958]': 963,\n",
              " '[unused959]': 964,\n",
              " '[unused960]': 965,\n",
              " '[unused961]': 966,\n",
              " '[unused962]': 967,\n",
              " '[unused963]': 968,\n",
              " '[unused964]': 969,\n",
              " '[unused965]': 970,\n",
              " '[unused966]': 971,\n",
              " '[unused967]': 972,\n",
              " '[unused968]': 973,\n",
              " '[unused969]': 974,\n",
              " '[unused970]': 975,\n",
              " '[unused971]': 976,\n",
              " '[unused972]': 977,\n",
              " '[unused973]': 978,\n",
              " '[unused974]': 979,\n",
              " '[unused975]': 980,\n",
              " '[unused976]': 981,\n",
              " '[unused977]': 982,\n",
              " '[unused978]': 983,\n",
              " '[unused979]': 984,\n",
              " '[unused980]': 985,\n",
              " '[unused981]': 986,\n",
              " '[unused982]': 987,\n",
              " '[unused983]': 988,\n",
              " '[unused984]': 989,\n",
              " '[unused985]': 990,\n",
              " '[unused986]': 991,\n",
              " '[unused987]': 992,\n",
              " '[unused988]': 993,\n",
              " '[unused989]': 994,\n",
              " '[unused990]': 995,\n",
              " '[unused991]': 996,\n",
              " '[unused992]': 997,\n",
              " '[unused993]': 998,\n",
              " '!': 999,\n",
              " ...}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0WhiwaBCIXqv",
        "colab_type": "text"
      },
      "source": [
        "- BERT의 토큰화는 단어를 분리하는 토큰화 방식입니다. wordpiece(단어조각?) 방식이라고 하는데, 이는 한국어를 형태소로 꼭 변환해야 할 문제를 해결해주며, 의미가 있는 단어는 밀접하게 연관이 되게 하는 장점까지 갖추고 있습니다.\n",
        "- 단어의 첫 시작은 ##가 붙지 않지만, 단어에 포함되면서 단어의 시작이 아닌 부분에는 ##가 붙는 것이 특징입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "88T_cMA4tsk2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 단어와 넘버링된 숫자를 활용하여 Tokenizer 생성\n",
        "tokenizer = Tokenizer(token_dict)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0dmxrOhJ9vD",
        "colab_type": "text"
      },
      "source": [
        "토큰화가 잘 되었는지 확인해 봅니다.\n",
        "버트 모형은 문장 앞에 꼭 [CLS]라는 문자가 위치하고, [SEP]라는 문자가 끝에 위치합니다.  \n",
        "[CLS]는 문장의 시작, [SEP]는 문장의 끝을 의미합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-4EWrSTCMkzk",
        "colab_type": "code",
        "outputId": "c630149f-fed3-4b48-ba73-bf46d4975897",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "print(tokenizer.tokenize(\"Keras is Really fun.\"), tokenizer.tokenize(\"we can manipulate AI.\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'ke', '##ras', 'is', 'really', 'fun', '.', '[SEP]'] ['[CLS]', 'we', 'can', 'manipulate', 'ai', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCC_Z1TNSODT",
        "colab_type": "text"
      },
      "source": [
        "token dict의 key 값과 value 값을 바꾼 reverse_token_dict를 정의합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oF_7BhoWw6_E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 추후에 쓰기 위해 token_dict를 반대로 한 딕셔너리를 생성\n",
        "# {숫자 : 단어, 숫자 : 단어}의 방식임\n",
        "reverse_token_dict = {v : k for k, v in token_dict.items()}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jeYV0qlaxepy",
        "colab_type": "code",
        "outputId": "093b2130-2ef1-43d0-b9fb-2e2de6e90499",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "reverse_token_dict"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: '[PAD]',\n",
              " 1: '[unused0]',\n",
              " 2: '[unused1]',\n",
              " 3: '[unused2]',\n",
              " 4: '[unused3]',\n",
              " 5: '[unused4]',\n",
              " 6: '[unused5]',\n",
              " 7: '[unused6]',\n",
              " 8: '[unused7]',\n",
              " 9: '[unused8]',\n",
              " 10: '[unused9]',\n",
              " 11: '[unused10]',\n",
              " 12: '[unused11]',\n",
              " 13: '[unused12]',\n",
              " 14: '[unused13]',\n",
              " 15: '[unused14]',\n",
              " 16: '[unused15]',\n",
              " 17: '[unused16]',\n",
              " 18: '[unused17]',\n",
              " 19: '[unused18]',\n",
              " 20: '[unused19]',\n",
              " 21: '[unused20]',\n",
              " 22: '[unused21]',\n",
              " 23: '[unused22]',\n",
              " 24: '[unused23]',\n",
              " 25: '[unused24]',\n",
              " 26: '[unused25]',\n",
              " 27: '[unused26]',\n",
              " 28: '[unused27]',\n",
              " 29: '[unused28]',\n",
              " 30: '[unused29]',\n",
              " 31: '[unused30]',\n",
              " 32: '[unused31]',\n",
              " 33: '[unused32]',\n",
              " 34: '[unused33]',\n",
              " 35: '[unused34]',\n",
              " 36: '[unused35]',\n",
              " 37: '[unused36]',\n",
              " 38: '[unused37]',\n",
              " 39: '[unused38]',\n",
              " 40: '[unused39]',\n",
              " 41: '[unused40]',\n",
              " 42: '[unused41]',\n",
              " 43: '[unused42]',\n",
              " 44: '[unused43]',\n",
              " 45: '[unused44]',\n",
              " 46: '[unused45]',\n",
              " 47: '[unused46]',\n",
              " 48: '[unused47]',\n",
              " 49: '[unused48]',\n",
              " 50: '[unused49]',\n",
              " 51: '[unused50]',\n",
              " 52: '[unused51]',\n",
              " 53: '[unused52]',\n",
              " 54: '[unused53]',\n",
              " 55: '[unused54]',\n",
              " 56: '[unused55]',\n",
              " 57: '[unused56]',\n",
              " 58: '[unused57]',\n",
              " 59: '[unused58]',\n",
              " 60: '[unused59]',\n",
              " 61: '[unused60]',\n",
              " 62: '[unused61]',\n",
              " 63: '[unused62]',\n",
              " 64: '[unused63]',\n",
              " 65: '[unused64]',\n",
              " 66: '[unused65]',\n",
              " 67: '[unused66]',\n",
              " 68: '[unused67]',\n",
              " 69: '[unused68]',\n",
              " 70: '[unused69]',\n",
              " 71: '[unused70]',\n",
              " 72: '[unused71]',\n",
              " 73: '[unused72]',\n",
              " 74: '[unused73]',\n",
              " 75: '[unused74]',\n",
              " 76: '[unused75]',\n",
              " 77: '[unused76]',\n",
              " 78: '[unused77]',\n",
              " 79: '[unused78]',\n",
              " 80: '[unused79]',\n",
              " 81: '[unused80]',\n",
              " 82: '[unused81]',\n",
              " 83: '[unused82]',\n",
              " 84: '[unused83]',\n",
              " 85: '[unused84]',\n",
              " 86: '[unused85]',\n",
              " 87: '[unused86]',\n",
              " 88: '[unused87]',\n",
              " 89: '[unused88]',\n",
              " 90: '[unused89]',\n",
              " 91: '[unused90]',\n",
              " 92: '[unused91]',\n",
              " 93: '[unused92]',\n",
              " 94: '[unused93]',\n",
              " 95: '[unused94]',\n",
              " 96: '[unused95]',\n",
              " 97: '[unused96]',\n",
              " 98: '[unused97]',\n",
              " 99: '[unused98]',\n",
              " 100: '[UNK]',\n",
              " 101: '[CLS]',\n",
              " 102: '[SEP]',\n",
              " 103: '[MASK]',\n",
              " 104: '[unused99]',\n",
              " 105: '[unused100]',\n",
              " 106: '[unused101]',\n",
              " 107: '[unused102]',\n",
              " 108: '[unused103]',\n",
              " 109: '[unused104]',\n",
              " 110: '[unused105]',\n",
              " 111: '[unused106]',\n",
              " 112: '[unused107]',\n",
              " 113: '[unused108]',\n",
              " 114: '[unused109]',\n",
              " 115: '[unused110]',\n",
              " 116: '[unused111]',\n",
              " 117: '[unused112]',\n",
              " 118: '[unused113]',\n",
              " 119: '[unused114]',\n",
              " 120: '[unused115]',\n",
              " 121: '[unused116]',\n",
              " 122: '[unused117]',\n",
              " 123: '[unused118]',\n",
              " 124: '[unused119]',\n",
              " 125: '[unused120]',\n",
              " 126: '[unused121]',\n",
              " 127: '[unused122]',\n",
              " 128: '[unused123]',\n",
              " 129: '[unused124]',\n",
              " 130: '[unused125]',\n",
              " 131: '[unused126]',\n",
              " 132: '[unused127]',\n",
              " 133: '[unused128]',\n",
              " 134: '[unused129]',\n",
              " 135: '[unused130]',\n",
              " 136: '[unused131]',\n",
              " 137: '[unused132]',\n",
              " 138: '[unused133]',\n",
              " 139: '[unused134]',\n",
              " 140: '[unused135]',\n",
              " 141: '[unused136]',\n",
              " 142: '[unused137]',\n",
              " 143: '[unused138]',\n",
              " 144: '[unused139]',\n",
              " 145: '[unused140]',\n",
              " 146: '[unused141]',\n",
              " 147: '[unused142]',\n",
              " 148: '[unused143]',\n",
              " 149: '[unused144]',\n",
              " 150: '[unused145]',\n",
              " 151: '[unused146]',\n",
              " 152: '[unused147]',\n",
              " 153: '[unused148]',\n",
              " 154: '[unused149]',\n",
              " 155: '[unused150]',\n",
              " 156: '[unused151]',\n",
              " 157: '[unused152]',\n",
              " 158: '[unused153]',\n",
              " 159: '[unused154]',\n",
              " 160: '[unused155]',\n",
              " 161: '[unused156]',\n",
              " 162: '[unused157]',\n",
              " 163: '[unused158]',\n",
              " 164: '[unused159]',\n",
              " 165: '[unused160]',\n",
              " 166: '[unused161]',\n",
              " 167: '[unused162]',\n",
              " 168: '[unused163]',\n",
              " 169: '[unused164]',\n",
              " 170: '[unused165]',\n",
              " 171: '[unused166]',\n",
              " 172: '[unused167]',\n",
              " 173: '[unused168]',\n",
              " 174: '[unused169]',\n",
              " 175: '[unused170]',\n",
              " 176: '[unused171]',\n",
              " 177: '[unused172]',\n",
              " 178: '[unused173]',\n",
              " 179: '[unused174]',\n",
              " 180: '[unused175]',\n",
              " 181: '[unused176]',\n",
              " 182: '[unused177]',\n",
              " 183: '[unused178]',\n",
              " 184: '[unused179]',\n",
              " 185: '[unused180]',\n",
              " 186: '[unused181]',\n",
              " 187: '[unused182]',\n",
              " 188: '[unused183]',\n",
              " 189: '[unused184]',\n",
              " 190: '[unused185]',\n",
              " 191: '[unused186]',\n",
              " 192: '[unused187]',\n",
              " 193: '[unused188]',\n",
              " 194: '[unused189]',\n",
              " 195: '[unused190]',\n",
              " 196: '[unused191]',\n",
              " 197: '[unused192]',\n",
              " 198: '[unused193]',\n",
              " 199: '[unused194]',\n",
              " 200: '[unused195]',\n",
              " 201: '[unused196]',\n",
              " 202: '[unused197]',\n",
              " 203: '[unused198]',\n",
              " 204: '[unused199]',\n",
              " 205: '[unused200]',\n",
              " 206: '[unused201]',\n",
              " 207: '[unused202]',\n",
              " 208: '[unused203]',\n",
              " 209: '[unused204]',\n",
              " 210: '[unused205]',\n",
              " 211: '[unused206]',\n",
              " 212: '[unused207]',\n",
              " 213: '[unused208]',\n",
              " 214: '[unused209]',\n",
              " 215: '[unused210]',\n",
              " 216: '[unused211]',\n",
              " 217: '[unused212]',\n",
              " 218: '[unused213]',\n",
              " 219: '[unused214]',\n",
              " 220: '[unused215]',\n",
              " 221: '[unused216]',\n",
              " 222: '[unused217]',\n",
              " 223: '[unused218]',\n",
              " 224: '[unused219]',\n",
              " 225: '[unused220]',\n",
              " 226: '[unused221]',\n",
              " 227: '[unused222]',\n",
              " 228: '[unused223]',\n",
              " 229: '[unused224]',\n",
              " 230: '[unused225]',\n",
              " 231: '[unused226]',\n",
              " 232: '[unused227]',\n",
              " 233: '[unused228]',\n",
              " 234: '[unused229]',\n",
              " 235: '[unused230]',\n",
              " 236: '[unused231]',\n",
              " 237: '[unused232]',\n",
              " 238: '[unused233]',\n",
              " 239: '[unused234]',\n",
              " 240: '[unused235]',\n",
              " 241: '[unused236]',\n",
              " 242: '[unused237]',\n",
              " 243: '[unused238]',\n",
              " 244: '[unused239]',\n",
              " 245: '[unused240]',\n",
              " 246: '[unused241]',\n",
              " 247: '[unused242]',\n",
              " 248: '[unused243]',\n",
              " 249: '[unused244]',\n",
              " 250: '[unused245]',\n",
              " 251: '[unused246]',\n",
              " 252: '[unused247]',\n",
              " 253: '[unused248]',\n",
              " 254: '[unused249]',\n",
              " 255: '[unused250]',\n",
              " 256: '[unused251]',\n",
              " 257: '[unused252]',\n",
              " 258: '[unused253]',\n",
              " 259: '[unused254]',\n",
              " 260: '[unused255]',\n",
              " 261: '[unused256]',\n",
              " 262: '[unused257]',\n",
              " 263: '[unused258]',\n",
              " 264: '[unused259]',\n",
              " 265: '[unused260]',\n",
              " 266: '[unused261]',\n",
              " 267: '[unused262]',\n",
              " 268: '[unused263]',\n",
              " 269: '[unused264]',\n",
              " 270: '[unused265]',\n",
              " 271: '[unused266]',\n",
              " 272: '[unused267]',\n",
              " 273: '[unused268]',\n",
              " 274: '[unused269]',\n",
              " 275: '[unused270]',\n",
              " 276: '[unused271]',\n",
              " 277: '[unused272]',\n",
              " 278: '[unused273]',\n",
              " 279: '[unused274]',\n",
              " 280: '[unused275]',\n",
              " 281: '[unused276]',\n",
              " 282: '[unused277]',\n",
              " 283: '[unused278]',\n",
              " 284: '[unused279]',\n",
              " 285: '[unused280]',\n",
              " 286: '[unused281]',\n",
              " 287: '[unused282]',\n",
              " 288: '[unused283]',\n",
              " 289: '[unused284]',\n",
              " 290: '[unused285]',\n",
              " 291: '[unused286]',\n",
              " 292: '[unused287]',\n",
              " 293: '[unused288]',\n",
              " 294: '[unused289]',\n",
              " 295: '[unused290]',\n",
              " 296: '[unused291]',\n",
              " 297: '[unused292]',\n",
              " 298: '[unused293]',\n",
              " 299: '[unused294]',\n",
              " 300: '[unused295]',\n",
              " 301: '[unused296]',\n",
              " 302: '[unused297]',\n",
              " 303: '[unused298]',\n",
              " 304: '[unused299]',\n",
              " 305: '[unused300]',\n",
              " 306: '[unused301]',\n",
              " 307: '[unused302]',\n",
              " 308: '[unused303]',\n",
              " 309: '[unused304]',\n",
              " 310: '[unused305]',\n",
              " 311: '[unused306]',\n",
              " 312: '[unused307]',\n",
              " 313: '[unused308]',\n",
              " 314: '[unused309]',\n",
              " 315: '[unused310]',\n",
              " 316: '[unused311]',\n",
              " 317: '[unused312]',\n",
              " 318: '[unused313]',\n",
              " 319: '[unused314]',\n",
              " 320: '[unused315]',\n",
              " 321: '[unused316]',\n",
              " 322: '[unused317]',\n",
              " 323: '[unused318]',\n",
              " 324: '[unused319]',\n",
              " 325: '[unused320]',\n",
              " 326: '[unused321]',\n",
              " 327: '[unused322]',\n",
              " 328: '[unused323]',\n",
              " 329: '[unused324]',\n",
              " 330: '[unused325]',\n",
              " 331: '[unused326]',\n",
              " 332: '[unused327]',\n",
              " 333: '[unused328]',\n",
              " 334: '[unused329]',\n",
              " 335: '[unused330]',\n",
              " 336: '[unused331]',\n",
              " 337: '[unused332]',\n",
              " 338: '[unused333]',\n",
              " 339: '[unused334]',\n",
              " 340: '[unused335]',\n",
              " 341: '[unused336]',\n",
              " 342: '[unused337]',\n",
              " 343: '[unused338]',\n",
              " 344: '[unused339]',\n",
              " 345: '[unused340]',\n",
              " 346: '[unused341]',\n",
              " 347: '[unused342]',\n",
              " 348: '[unused343]',\n",
              " 349: '[unused344]',\n",
              " 350: '[unused345]',\n",
              " 351: '[unused346]',\n",
              " 352: '[unused347]',\n",
              " 353: '[unused348]',\n",
              " 354: '[unused349]',\n",
              " 355: '[unused350]',\n",
              " 356: '[unused351]',\n",
              " 357: '[unused352]',\n",
              " 358: '[unused353]',\n",
              " 359: '[unused354]',\n",
              " 360: '[unused355]',\n",
              " 361: '[unused356]',\n",
              " 362: '[unused357]',\n",
              " 363: '[unused358]',\n",
              " 364: '[unused359]',\n",
              " 365: '[unused360]',\n",
              " 366: '[unused361]',\n",
              " 367: '[unused362]',\n",
              " 368: '[unused363]',\n",
              " 369: '[unused364]',\n",
              " 370: '[unused365]',\n",
              " 371: '[unused366]',\n",
              " 372: '[unused367]',\n",
              " 373: '[unused368]',\n",
              " 374: '[unused369]',\n",
              " 375: '[unused370]',\n",
              " 376: '[unused371]',\n",
              " 377: '[unused372]',\n",
              " 378: '[unused373]',\n",
              " 379: '[unused374]',\n",
              " 380: '[unused375]',\n",
              " 381: '[unused376]',\n",
              " 382: '[unused377]',\n",
              " 383: '[unused378]',\n",
              " 384: '[unused379]',\n",
              " 385: '[unused380]',\n",
              " 386: '[unused381]',\n",
              " 387: '[unused382]',\n",
              " 388: '[unused383]',\n",
              " 389: '[unused384]',\n",
              " 390: '[unused385]',\n",
              " 391: '[unused386]',\n",
              " 392: '[unused387]',\n",
              " 393: '[unused388]',\n",
              " 394: '[unused389]',\n",
              " 395: '[unused390]',\n",
              " 396: '[unused391]',\n",
              " 397: '[unused392]',\n",
              " 398: '[unused393]',\n",
              " 399: '[unused394]',\n",
              " 400: '[unused395]',\n",
              " 401: '[unused396]',\n",
              " 402: '[unused397]',\n",
              " 403: '[unused398]',\n",
              " 404: '[unused399]',\n",
              " 405: '[unused400]',\n",
              " 406: '[unused401]',\n",
              " 407: '[unused402]',\n",
              " 408: '[unused403]',\n",
              " 409: '[unused404]',\n",
              " 410: '[unused405]',\n",
              " 411: '[unused406]',\n",
              " 412: '[unused407]',\n",
              " 413: '[unused408]',\n",
              " 414: '[unused409]',\n",
              " 415: '[unused410]',\n",
              " 416: '[unused411]',\n",
              " 417: '[unused412]',\n",
              " 418: '[unused413]',\n",
              " 419: '[unused414]',\n",
              " 420: '[unused415]',\n",
              " 421: '[unused416]',\n",
              " 422: '[unused417]',\n",
              " 423: '[unused418]',\n",
              " 424: '[unused419]',\n",
              " 425: '[unused420]',\n",
              " 426: '[unused421]',\n",
              " 427: '[unused422]',\n",
              " 428: '[unused423]',\n",
              " 429: '[unused424]',\n",
              " 430: '[unused425]',\n",
              " 431: '[unused426]',\n",
              " 432: '[unused427]',\n",
              " 433: '[unused428]',\n",
              " 434: '[unused429]',\n",
              " 435: '[unused430]',\n",
              " 436: '[unused431]',\n",
              " 437: '[unused432]',\n",
              " 438: '[unused433]',\n",
              " 439: '[unused434]',\n",
              " 440: '[unused435]',\n",
              " 441: '[unused436]',\n",
              " 442: '[unused437]',\n",
              " 443: '[unused438]',\n",
              " 444: '[unused439]',\n",
              " 445: '[unused440]',\n",
              " 446: '[unused441]',\n",
              " 447: '[unused442]',\n",
              " 448: '[unused443]',\n",
              " 449: '[unused444]',\n",
              " 450: '[unused445]',\n",
              " 451: '[unused446]',\n",
              " 452: '[unused447]',\n",
              " 453: '[unused448]',\n",
              " 454: '[unused449]',\n",
              " 455: '[unused450]',\n",
              " 456: '[unused451]',\n",
              " 457: '[unused452]',\n",
              " 458: '[unused453]',\n",
              " 459: '[unused454]',\n",
              " 460: '[unused455]',\n",
              " 461: '[unused456]',\n",
              " 462: '[unused457]',\n",
              " 463: '[unused458]',\n",
              " 464: '[unused459]',\n",
              " 465: '[unused460]',\n",
              " 466: '[unused461]',\n",
              " 467: '[unused462]',\n",
              " 468: '[unused463]',\n",
              " 469: '[unused464]',\n",
              " 470: '[unused465]',\n",
              " 471: '[unused466]',\n",
              " 472: '[unused467]',\n",
              " 473: '[unused468]',\n",
              " 474: '[unused469]',\n",
              " 475: '[unused470]',\n",
              " 476: '[unused471]',\n",
              " 477: '[unused472]',\n",
              " 478: '[unused473]',\n",
              " 479: '[unused474]',\n",
              " 480: '[unused475]',\n",
              " 481: '[unused476]',\n",
              " 482: '[unused477]',\n",
              " 483: '[unused478]',\n",
              " 484: '[unused479]',\n",
              " 485: '[unused480]',\n",
              " 486: '[unused481]',\n",
              " 487: '[unused482]',\n",
              " 488: '[unused483]',\n",
              " 489: '[unused484]',\n",
              " 490: '[unused485]',\n",
              " 491: '[unused486]',\n",
              " 492: '[unused487]',\n",
              " 493: '[unused488]',\n",
              " 494: '[unused489]',\n",
              " 495: '[unused490]',\n",
              " 496: '[unused491]',\n",
              " 497: '[unused492]',\n",
              " 498: '[unused493]',\n",
              " 499: '[unused494]',\n",
              " 500: '[unused495]',\n",
              " 501: '[unused496]',\n",
              " 502: '[unused497]',\n",
              " 503: '[unused498]',\n",
              " 504: '[unused499]',\n",
              " 505: '[unused500]',\n",
              " 506: '[unused501]',\n",
              " 507: '[unused502]',\n",
              " 508: '[unused503]',\n",
              " 509: '[unused504]',\n",
              " 510: '[unused505]',\n",
              " 511: '[unused506]',\n",
              " 512: '[unused507]',\n",
              " 513: '[unused508]',\n",
              " 514: '[unused509]',\n",
              " 515: '[unused510]',\n",
              " 516: '[unused511]',\n",
              " 517: '[unused512]',\n",
              " 518: '[unused513]',\n",
              " 519: '[unused514]',\n",
              " 520: '[unused515]',\n",
              " 521: '[unused516]',\n",
              " 522: '[unused517]',\n",
              " 523: '[unused518]',\n",
              " 524: '[unused519]',\n",
              " 525: '[unused520]',\n",
              " 526: '[unused521]',\n",
              " 527: '[unused522]',\n",
              " 528: '[unused523]',\n",
              " 529: '[unused524]',\n",
              " 530: '[unused525]',\n",
              " 531: '[unused526]',\n",
              " 532: '[unused527]',\n",
              " 533: '[unused528]',\n",
              " 534: '[unused529]',\n",
              " 535: '[unused530]',\n",
              " 536: '[unused531]',\n",
              " 537: '[unused532]',\n",
              " 538: '[unused533]',\n",
              " 539: '[unused534]',\n",
              " 540: '[unused535]',\n",
              " 541: '[unused536]',\n",
              " 542: '[unused537]',\n",
              " 543: '[unused538]',\n",
              " 544: '[unused539]',\n",
              " 545: '[unused540]',\n",
              " 546: '[unused541]',\n",
              " 547: '[unused542]',\n",
              " 548: '[unused543]',\n",
              " 549: '[unused544]',\n",
              " 550: '[unused545]',\n",
              " 551: '[unused546]',\n",
              " 552: '[unused547]',\n",
              " 553: '[unused548]',\n",
              " 554: '[unused549]',\n",
              " 555: '[unused550]',\n",
              " 556: '[unused551]',\n",
              " 557: '[unused552]',\n",
              " 558: '[unused553]',\n",
              " 559: '[unused554]',\n",
              " 560: '[unused555]',\n",
              " 561: '[unused556]',\n",
              " 562: '[unused557]',\n",
              " 563: '[unused558]',\n",
              " 564: '[unused559]',\n",
              " 565: '[unused560]',\n",
              " 566: '[unused561]',\n",
              " 567: '[unused562]',\n",
              " 568: '[unused563]',\n",
              " 569: '[unused564]',\n",
              " 570: '[unused565]',\n",
              " 571: '[unused566]',\n",
              " 572: '[unused567]',\n",
              " 573: '[unused568]',\n",
              " 574: '[unused569]',\n",
              " 575: '[unused570]',\n",
              " 576: '[unused571]',\n",
              " 577: '[unused572]',\n",
              " 578: '[unused573]',\n",
              " 579: '[unused574]',\n",
              " 580: '[unused575]',\n",
              " 581: '[unused576]',\n",
              " 582: '[unused577]',\n",
              " 583: '[unused578]',\n",
              " 584: '[unused579]',\n",
              " 585: '[unused580]',\n",
              " 586: '[unused581]',\n",
              " 587: '[unused582]',\n",
              " 588: '[unused583]',\n",
              " 589: '[unused584]',\n",
              " 590: '[unused585]',\n",
              " 591: '[unused586]',\n",
              " 592: '[unused587]',\n",
              " 593: '[unused588]',\n",
              " 594: '[unused589]',\n",
              " 595: '[unused590]',\n",
              " 596: '[unused591]',\n",
              " 597: '[unused592]',\n",
              " 598: '[unused593]',\n",
              " 599: '[unused594]',\n",
              " 600: '[unused595]',\n",
              " 601: '[unused596]',\n",
              " 602: '[unused597]',\n",
              " 603: '[unused598]',\n",
              " 604: '[unused599]',\n",
              " 605: '[unused600]',\n",
              " 606: '[unused601]',\n",
              " 607: '[unused602]',\n",
              " 608: '[unused603]',\n",
              " 609: '[unused604]',\n",
              " 610: '[unused605]',\n",
              " 611: '[unused606]',\n",
              " 612: '[unused607]',\n",
              " 613: '[unused608]',\n",
              " 614: '[unused609]',\n",
              " 615: '[unused610]',\n",
              " 616: '[unused611]',\n",
              " 617: '[unused612]',\n",
              " 618: '[unused613]',\n",
              " 619: '[unused614]',\n",
              " 620: '[unused615]',\n",
              " 621: '[unused616]',\n",
              " 622: '[unused617]',\n",
              " 623: '[unused618]',\n",
              " 624: '[unused619]',\n",
              " 625: '[unused620]',\n",
              " 626: '[unused621]',\n",
              " 627: '[unused622]',\n",
              " 628: '[unused623]',\n",
              " 629: '[unused624]',\n",
              " 630: '[unused625]',\n",
              " 631: '[unused626]',\n",
              " 632: '[unused627]',\n",
              " 633: '[unused628]',\n",
              " 634: '[unused629]',\n",
              " 635: '[unused630]',\n",
              " 636: '[unused631]',\n",
              " 637: '[unused632]',\n",
              " 638: '[unused633]',\n",
              " 639: '[unused634]',\n",
              " 640: '[unused635]',\n",
              " 641: '[unused636]',\n",
              " 642: '[unused637]',\n",
              " 643: '[unused638]',\n",
              " 644: '[unused639]',\n",
              " 645: '[unused640]',\n",
              " 646: '[unused641]',\n",
              " 647: '[unused642]',\n",
              " 648: '[unused643]',\n",
              " 649: '[unused644]',\n",
              " 650: '[unused645]',\n",
              " 651: '[unused646]',\n",
              " 652: '[unused647]',\n",
              " 653: '[unused648]',\n",
              " 654: '[unused649]',\n",
              " 655: '[unused650]',\n",
              " 656: '[unused651]',\n",
              " 657: '[unused652]',\n",
              " 658: '[unused653]',\n",
              " 659: '[unused654]',\n",
              " 660: '[unused655]',\n",
              " 661: '[unused656]',\n",
              " 662: '[unused657]',\n",
              " 663: '[unused658]',\n",
              " 664: '[unused659]',\n",
              " 665: '[unused660]',\n",
              " 666: '[unused661]',\n",
              " 667: '[unused662]',\n",
              " 668: '[unused663]',\n",
              " 669: '[unused664]',\n",
              " 670: '[unused665]',\n",
              " 671: '[unused666]',\n",
              " 672: '[unused667]',\n",
              " 673: '[unused668]',\n",
              " 674: '[unused669]',\n",
              " 675: '[unused670]',\n",
              " 676: '[unused671]',\n",
              " 677: '[unused672]',\n",
              " 678: '[unused673]',\n",
              " 679: '[unused674]',\n",
              " 680: '[unused675]',\n",
              " 681: '[unused676]',\n",
              " 682: '[unused677]',\n",
              " 683: '[unused678]',\n",
              " 684: '[unused679]',\n",
              " 685: '[unused680]',\n",
              " 686: '[unused681]',\n",
              " 687: '[unused682]',\n",
              " 688: '[unused683]',\n",
              " 689: '[unused684]',\n",
              " 690: '[unused685]',\n",
              " 691: '[unused686]',\n",
              " 692: '[unused687]',\n",
              " 693: '[unused688]',\n",
              " 694: '[unused689]',\n",
              " 695: '[unused690]',\n",
              " 696: '[unused691]',\n",
              " 697: '[unused692]',\n",
              " 698: '[unused693]',\n",
              " 699: '[unused694]',\n",
              " 700: '[unused695]',\n",
              " 701: '[unused696]',\n",
              " 702: '[unused697]',\n",
              " 703: '[unused698]',\n",
              " 704: '[unused699]',\n",
              " 705: '[unused700]',\n",
              " 706: '[unused701]',\n",
              " 707: '[unused702]',\n",
              " 708: '[unused703]',\n",
              " 709: '[unused704]',\n",
              " 710: '[unused705]',\n",
              " 711: '[unused706]',\n",
              " 712: '[unused707]',\n",
              " 713: '[unused708]',\n",
              " 714: '[unused709]',\n",
              " 715: '[unused710]',\n",
              " 716: '[unused711]',\n",
              " 717: '[unused712]',\n",
              " 718: '[unused713]',\n",
              " 719: '[unused714]',\n",
              " 720: '[unused715]',\n",
              " 721: '[unused716]',\n",
              " 722: '[unused717]',\n",
              " 723: '[unused718]',\n",
              " 724: '[unused719]',\n",
              " 725: '[unused720]',\n",
              " 726: '[unused721]',\n",
              " 727: '[unused722]',\n",
              " 728: '[unused723]',\n",
              " 729: '[unused724]',\n",
              " 730: '[unused725]',\n",
              " 731: '[unused726]',\n",
              " 732: '[unused727]',\n",
              " 733: '[unused728]',\n",
              " 734: '[unused729]',\n",
              " 735: '[unused730]',\n",
              " 736: '[unused731]',\n",
              " 737: '[unused732]',\n",
              " 738: '[unused733]',\n",
              " 739: '[unused734]',\n",
              " 740: '[unused735]',\n",
              " 741: '[unused736]',\n",
              " 742: '[unused737]',\n",
              " 743: '[unused738]',\n",
              " 744: '[unused739]',\n",
              " 745: '[unused740]',\n",
              " 746: '[unused741]',\n",
              " 747: '[unused742]',\n",
              " 748: '[unused743]',\n",
              " 749: '[unused744]',\n",
              " 750: '[unused745]',\n",
              " 751: '[unused746]',\n",
              " 752: '[unused747]',\n",
              " 753: '[unused748]',\n",
              " 754: '[unused749]',\n",
              " 755: '[unused750]',\n",
              " 756: '[unused751]',\n",
              " 757: '[unused752]',\n",
              " 758: '[unused753]',\n",
              " 759: '[unused754]',\n",
              " 760: '[unused755]',\n",
              " 761: '[unused756]',\n",
              " 762: '[unused757]',\n",
              " 763: '[unused758]',\n",
              " 764: '[unused759]',\n",
              " 765: '[unused760]',\n",
              " 766: '[unused761]',\n",
              " 767: '[unused762]',\n",
              " 768: '[unused763]',\n",
              " 769: '[unused764]',\n",
              " 770: '[unused765]',\n",
              " 771: '[unused766]',\n",
              " 772: '[unused767]',\n",
              " 773: '[unused768]',\n",
              " 774: '[unused769]',\n",
              " 775: '[unused770]',\n",
              " 776: '[unused771]',\n",
              " 777: '[unused772]',\n",
              " 778: '[unused773]',\n",
              " 779: '[unused774]',\n",
              " 780: '[unused775]',\n",
              " 781: '[unused776]',\n",
              " 782: '[unused777]',\n",
              " 783: '[unused778]',\n",
              " 784: '[unused779]',\n",
              " 785: '[unused780]',\n",
              " 786: '[unused781]',\n",
              " 787: '[unused782]',\n",
              " 788: '[unused783]',\n",
              " 789: '[unused784]',\n",
              " 790: '[unused785]',\n",
              " 791: '[unused786]',\n",
              " 792: '[unused787]',\n",
              " 793: '[unused788]',\n",
              " 794: '[unused789]',\n",
              " 795: '[unused790]',\n",
              " 796: '[unused791]',\n",
              " 797: '[unused792]',\n",
              " 798: '[unused793]',\n",
              " 799: '[unused794]',\n",
              " 800: '[unused795]',\n",
              " 801: '[unused796]',\n",
              " 802: '[unused797]',\n",
              " 803: '[unused798]',\n",
              " 804: '[unused799]',\n",
              " 805: '[unused800]',\n",
              " 806: '[unused801]',\n",
              " 807: '[unused802]',\n",
              " 808: '[unused803]',\n",
              " 809: '[unused804]',\n",
              " 810: '[unused805]',\n",
              " 811: '[unused806]',\n",
              " 812: '[unused807]',\n",
              " 813: '[unused808]',\n",
              " 814: '[unused809]',\n",
              " 815: '[unused810]',\n",
              " 816: '[unused811]',\n",
              " 817: '[unused812]',\n",
              " 818: '[unused813]',\n",
              " 819: '[unused814]',\n",
              " 820: '[unused815]',\n",
              " 821: '[unused816]',\n",
              " 822: '[unused817]',\n",
              " 823: '[unused818]',\n",
              " 824: '[unused819]',\n",
              " 825: '[unused820]',\n",
              " 826: '[unused821]',\n",
              " 827: '[unused822]',\n",
              " 828: '[unused823]',\n",
              " 829: '[unused824]',\n",
              " 830: '[unused825]',\n",
              " 831: '[unused826]',\n",
              " 832: '[unused827]',\n",
              " 833: '[unused828]',\n",
              " 834: '[unused829]',\n",
              " 835: '[unused830]',\n",
              " 836: '[unused831]',\n",
              " 837: '[unused832]',\n",
              " 838: '[unused833]',\n",
              " 839: '[unused834]',\n",
              " 840: '[unused835]',\n",
              " 841: '[unused836]',\n",
              " 842: '[unused837]',\n",
              " 843: '[unused838]',\n",
              " 844: '[unused839]',\n",
              " 845: '[unused840]',\n",
              " 846: '[unused841]',\n",
              " 847: '[unused842]',\n",
              " 848: '[unused843]',\n",
              " 849: '[unused844]',\n",
              " 850: '[unused845]',\n",
              " 851: '[unused846]',\n",
              " 852: '[unused847]',\n",
              " 853: '[unused848]',\n",
              " 854: '[unused849]',\n",
              " 855: '[unused850]',\n",
              " 856: '[unused851]',\n",
              " 857: '[unused852]',\n",
              " 858: '[unused853]',\n",
              " 859: '[unused854]',\n",
              " 860: '[unused855]',\n",
              " 861: '[unused856]',\n",
              " 862: '[unused857]',\n",
              " 863: '[unused858]',\n",
              " 864: '[unused859]',\n",
              " 865: '[unused860]',\n",
              " 866: '[unused861]',\n",
              " 867: '[unused862]',\n",
              " 868: '[unused863]',\n",
              " 869: '[unused864]',\n",
              " 870: '[unused865]',\n",
              " 871: '[unused866]',\n",
              " 872: '[unused867]',\n",
              " 873: '[unused868]',\n",
              " 874: '[unused869]',\n",
              " 875: '[unused870]',\n",
              " 876: '[unused871]',\n",
              " 877: '[unused872]',\n",
              " 878: '[unused873]',\n",
              " 879: '[unused874]',\n",
              " 880: '[unused875]',\n",
              " 881: '[unused876]',\n",
              " 882: '[unused877]',\n",
              " 883: '[unused878]',\n",
              " 884: '[unused879]',\n",
              " 885: '[unused880]',\n",
              " 886: '[unused881]',\n",
              " 887: '[unused882]',\n",
              " 888: '[unused883]',\n",
              " 889: '[unused884]',\n",
              " 890: '[unused885]',\n",
              " 891: '[unused886]',\n",
              " 892: '[unused887]',\n",
              " 893: '[unused888]',\n",
              " 894: '[unused889]',\n",
              " 895: '[unused890]',\n",
              " 896: '[unused891]',\n",
              " 897: '[unused892]',\n",
              " 898: '[unused893]',\n",
              " 899: '[unused894]',\n",
              " 900: '[unused895]',\n",
              " 901: '[unused896]',\n",
              " 902: '[unused897]',\n",
              " 903: '[unused898]',\n",
              " 904: '[unused899]',\n",
              " 905: '[unused900]',\n",
              " 906: '[unused901]',\n",
              " 907: '[unused902]',\n",
              " 908: '[unused903]',\n",
              " 909: '[unused904]',\n",
              " 910: '[unused905]',\n",
              " 911: '[unused906]',\n",
              " 912: '[unused907]',\n",
              " 913: '[unused908]',\n",
              " 914: '[unused909]',\n",
              " 915: '[unused910]',\n",
              " 916: '[unused911]',\n",
              " 917: '[unused912]',\n",
              " 918: '[unused913]',\n",
              " 919: '[unused914]',\n",
              " 920: '[unused915]',\n",
              " 921: '[unused916]',\n",
              " 922: '[unused917]',\n",
              " 923: '[unused918]',\n",
              " 924: '[unused919]',\n",
              " 925: '[unused920]',\n",
              " 926: '[unused921]',\n",
              " 927: '[unused922]',\n",
              " 928: '[unused923]',\n",
              " 929: '[unused924]',\n",
              " 930: '[unused925]',\n",
              " 931: '[unused926]',\n",
              " 932: '[unused927]',\n",
              " 933: '[unused928]',\n",
              " 934: '[unused929]',\n",
              " 935: '[unused930]',\n",
              " 936: '[unused931]',\n",
              " 937: '[unused932]',\n",
              " 938: '[unused933]',\n",
              " 939: '[unused934]',\n",
              " 940: '[unused935]',\n",
              " 941: '[unused936]',\n",
              " 942: '[unused937]',\n",
              " 943: '[unused938]',\n",
              " 944: '[unused939]',\n",
              " 945: '[unused940]',\n",
              " 946: '[unused941]',\n",
              " 947: '[unused942]',\n",
              " 948: '[unused943]',\n",
              " 949: '[unused944]',\n",
              " 950: '[unused945]',\n",
              " 951: '[unused946]',\n",
              " 952: '[unused947]',\n",
              " 953: '[unused948]',\n",
              " 954: '[unused949]',\n",
              " 955: '[unused950]',\n",
              " 956: '[unused951]',\n",
              " 957: '[unused952]',\n",
              " 958: '[unused953]',\n",
              " 959: '[unused954]',\n",
              " 960: '[unused955]',\n",
              " 961: '[unused956]',\n",
              " 962: '[unused957]',\n",
              " 963: '[unused958]',\n",
              " 964: '[unused959]',\n",
              " 965: '[unused960]',\n",
              " 966: '[unused961]',\n",
              " 967: '[unused962]',\n",
              " 968: '[unused963]',\n",
              " 969: '[unused964]',\n",
              " 970: '[unused965]',\n",
              " 971: '[unused966]',\n",
              " 972: '[unused967]',\n",
              " 973: '[unused968]',\n",
              " 974: '[unused969]',\n",
              " 975: '[unused970]',\n",
              " 976: '[unused971]',\n",
              " 977: '[unused972]',\n",
              " 978: '[unused973]',\n",
              " 979: '[unused974]',\n",
              " 980: '[unused975]',\n",
              " 981: '[unused976]',\n",
              " 982: '[unused977]',\n",
              " 983: '[unused978]',\n",
              " 984: '[unused979]',\n",
              " 985: '[unused980]',\n",
              " 986: '[unused981]',\n",
              " 987: '[unused982]',\n",
              " 988: '[unused983]',\n",
              " 989: '[unused984]',\n",
              " 990: '[unused985]',\n",
              " 991: '[unused986]',\n",
              " 992: '[unused987]',\n",
              " 993: '[unused988]',\n",
              " 994: '[unused989]',\n",
              " 995: '[unused990]',\n",
              " 996: '[unused991]',\n",
              " 997: '[unused992]',\n",
              " 998: '[unused993]',\n",
              " 999: '!',\n",
              " ...}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9sFKLjD-v2n",
        "colab_type": "text"
      },
      "source": [
        "**버트 모형에 들어갈 인풋은 토큰, 세그먼트, 포지션으로 구성됩니다.**  \n",
        "버트에 인풋으로 들어가는 토큰은 문장을 토크나이징 한 후, 인덱스 번호를 매긴 것입니다.  \n",
        "세그먼트는 예를 들어 문장이 두 개가 있다면, 앞의 문장과 뒤의 문장을 구분하는 것입니다.  \n",
        "포지션 임베딩은 단순히 단어의 위치를 말합니다.\n",
        "\n",
        "토큰, 세그먼트, 포지션을 인풋으로 버트 모형에 넣으면 기하학적인 문장 공간으로 임베딩이 됩니다.  \n",
        "그림을 보면 my dog is cute, he likes play ##ing 두 문장이 있는데요  \n",
        "SQUAD 문제에서는 첫번째 문장이 question, 두번째 문장이 context가 되고, 이 두 문장이 합쳐져서 하나의 문장으로 들어가게 됩니다.\n",
        "\n",
        "![대체 텍스트](https://i.imgur.com/l9BTao3.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7RgCdr0K9Uy",
        "colab_type": "text"
      },
      "source": [
        "토큰화 과정을 위의 문장인 my dog is cute, he likes playing을 가지고 실험해보도록 하겠습니다.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "psUA8JQ-K5rc",
        "colab_type": "code",
        "outputId": "838d236b-0725-4d4e-9e4c-7ce1c1c52cfa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(tokenizer.tokenize(\"my dog is cute\", \"he likes playing\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'my', 'dog', 'is', 'cute', '[SEP]', 'he', 'likes', 'playing', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hhOddobRA6BZ",
        "colab_type": "text"
      },
      "source": [
        "버트 인풋으로 들어가도록 토크나이즈 된 것을 숫자로 변환해보도록 하겠습니다.  \n",
        "버트 인풋으로 실제 들어가는 값이 되겠습니다.  \n",
        "SQUAD 문제에서는 Question으로 my dog is cute이 인풋으로 들어가고, Context로 he likes playing이 인풋으로 들어가게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yO5Bv2n-BBFY",
        "colab_type": "code",
        "outputId": "2b90277a-bdc2-4e26-ed32-2998293d2b37",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# 위에 토크나이즈 된 리스트가 숫자로 바뀜\n",
        "print(tokenizer.encode(\"my dog is cute\", \"he likes playing\"))\n",
        "# 0이 앞 문장에 대한 segment, 1이 뒷 문장에 대한 segment임\n",
        "# 버트 모델에 인풋 값으로 토크나이즈가 인덱스 된 리스트와 세그멘트 리스트가 들어가게 됨"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "([101, 2026, 3899, 2003, 10140, 102, 2002, 7777, 2652, 102], [0, 0, 0, 0, 0, 0, 1, 1, 1, 1])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXdjhy0cLlYA",
        "colab_type": "text"
      },
      "source": [
        "- 사전학습된 버트 모델의 인풋은 문장 토큰화가 숫자로 바뀐 것과, 앞문장인지 뒷문장인지 알려주는 문장 순서 벡터가 들어갑니다. **이번 SQUAD 문제에서는, 첫번째 문장에 해당하는 질문(Question)에 세그먼트가 0, 두번째 문장에 해당하는 Context에 세그먼트가1, 패딩 부분에 세그멘트가 0**이 들어가게 되어, 문장의 전후관계를 구분하게 됩니다.\n",
        "- 그리고 파인튜닝 시에는 문장 안에 일부 단어를 가리는 마스킹은 사용하지 않습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vBScUdNtJQoA",
        "colab_type": "code",
        "outputId": "286b2e06-bb9f-483e-b47b-bffb33abf7ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "# 토크나이징(평문장을 토큰화)\n",
        "print(tokenizer.tokenize(\"Keras is Really fun.\"), tokenizer.tokenize(\"we can manipulate AI.\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['[CLS]', 'ke', '##ras', 'is', 'really', 'fun', '.', '[SEP]'] ['[CLS]', 'we', 'can', 'manipulate', 'ai', '.', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fgOlJH_4ZQpS",
        "colab_type": "text"
      },
      "source": [
        "우리의 목표는, 질문(question)과 문장(context)를 받아서, 정답(text)를 맞추는 모델을 만드는 것입니다.  \n",
        "정답을 통째로 맞추는 것이 아니라, **토큰화된 것의 맨 앞 단어와, 맨 뒷 단어입니다.**  \n",
        "토큰화된 정답은 ['[CLS]', 'saint', 'bern', '##ade', '##tte', 'so', '##ub', '##iro', '##us', '[SEP]'] 인데, 여기서 **saint**에 해당하는 위치와 **##ub**에 해당하는 위치를 맞추는 버트 모형을 파인튜닝 하려 하는 것입니다.  \n",
        "\n",
        "  그래서 밑에 convert_data 함수에서, 정답(text) 길이만큼 문장(context)를 슬라이딩 하면서 만약에 문장이 정답을 포함하는 위치에 도달하면, 문장에서 정답의 맨 앞이 우리가 예측할 1번째 정답, 정답의 맨 뒤가 우리가 예측할 2번째 정답이 되게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "32gBWIqu_-De",
        "colab_type": "code",
        "outputId": "0dd8155b-e946-4c97-ca0a-b609c05483fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 523
        }
      },
      "source": [
        "train.head(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>index</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answer_start</th>\n",
              "      <th>text</th>\n",
              "      <th>c_id</th>\n",
              "      <th>context_len</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5733be284776f41900661182</td>\n",
              "      <td>To whom did the Virgin Mary allegedly appear i...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>515</td>\n",
              "      <td>Saint Bernadette Soubirous</td>\n",
              "      <td>0</td>\n",
              "      <td>695</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5733be284776f4190066117f</td>\n",
              "      <td>What is in front of the Notre Dame Main Building?</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>188</td>\n",
              "      <td>a copper statue of Christ</td>\n",
              "      <td>0</td>\n",
              "      <td>695</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5733be284776f41900661180</td>\n",
              "      <td>The Basilica of the Sacred heart at Notre Dame...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>279</td>\n",
              "      <td>the Main Building</td>\n",
              "      <td>0</td>\n",
              "      <td>695</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5733be284776f41900661181</td>\n",
              "      <td>What is the Grotto at Notre Dame?</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>381</td>\n",
              "      <td>a Marian place of prayer and reflection</td>\n",
              "      <td>0</td>\n",
              "      <td>695</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5733be284776f4190066117e</td>\n",
              "      <td>What sits on top of the Main Building at Notre...</td>\n",
              "      <td>Architecturally, the school has a Catholic cha...</td>\n",
              "      <td>92</td>\n",
              "      <td>a golden statue of the Virgin Mary</td>\n",
              "      <td>0</td>\n",
              "      <td>695</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                      index  ... context_len\n",
              "0  5733be284776f41900661182  ...         695\n",
              "1  5733be284776f4190066117f  ...         695\n",
              "2  5733be284776f41900661180  ...         695\n",
              "3  5733be284776f41900661181  ...         695\n",
              "4  5733be284776f4190066117e  ...         695\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FfTA6iNICKsq",
        "colab_type": "code",
        "outputId": "4dbaaadc-0025-4689-b091-3aa062f80bda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "print(\"Question : \", train.loc[0,'question'])\n",
        "print(\"Context : \", train.loc[0,'context'])\n",
        "print(\"Answer : \", train.loc[0,'text'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Question :  To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n",
            "Context :  Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.\n",
            "Answer :  Saint Bernadette Soubirous\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gCaZ5bSgDCcw",
        "colab_type": "code",
        "outputId": "b68297a9-c940-48d9-b71b-1ff0453c8bda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "print(\"Question : \", tokenizer.tokenize(train.loc[0,'question']))\n",
        "print(\"Context : \", tokenizer.tokenize(train.loc[0,'context']))\n",
        "print(\"Answer : \", tokenizer.tokenize(train.loc[0,'text']))\n",
        "\n",
        "# SQUAD 문제에서 예측하는 실제 정답은 맨 첫 토큰인 saint와 맨 끝 토큰인 ##us의 위치를 맞추는 것임"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Question :  ['[CLS]', 'to', 'whom', 'did', 'the', 'virgin', 'mary', 'allegedly', 'appear', 'in', '1858', 'in', 'lou', '##rdes', 'france', '?', '[SEP]']\n",
            "Context :  ['[CLS]', 'architectural', '##ly', ',', 'the', 'school', 'has', 'a', 'catholic', 'character', '.', 'atop', 'the', 'main', 'building', \"'\", 's', 'gold', 'dome', 'is', 'a', 'golden', 'statue', 'of', 'the', 'virgin', 'mary', '.', 'immediately', 'in', 'front', 'of', 'the', 'main', 'building', 'and', 'facing', 'it', ',', 'is', 'a', 'copper', 'statue', 'of', 'christ', 'with', 'arms', 'up', '##rai', '##sed', 'with', 'the', 'legend', '\"', 've', '##ni', '##te', 'ad', 'me', 'om', '##nes', '\"', '.', 'next', 'to', 'the', 'main', 'building', 'is', 'the', 'basilica', 'of', 'the', 'sacred', 'heart', '.', 'immediately', 'behind', 'the', 'basilica', 'is', 'the', 'gr', '##otto', ',', 'a', 'marian', 'place', 'of', 'prayer', 'and', 'reflection', '.', 'it', 'is', 'a', 'replica', 'of', 'the', 'gr', '##otto', 'at', 'lou', '##rdes', ',', 'france', 'where', 'the', 'virgin', 'mary', 'reputed', '##ly', 'appeared', 'to', 'saint', 'bern', '##ade', '##tte', 'so', '##ub', '##iro', '##us', 'in', '1858', '.', 'at', 'the', 'end', 'of', 'the', 'main', 'drive', '(', 'and', 'in', 'a', 'direct', 'line', 'that', 'connects', 'through', '3', 'statues', 'and', 'the', 'gold', 'dome', ')', ',', 'is', 'a', 'simple', ',', 'modern', 'stone', 'statue', 'of', 'mary', '.', '[SEP]']\n",
            "Answer :  ['[CLS]', 'saint', 'bern', '##ade', '##tte', 'so', '##ub', '##iro', '##us', '[SEP]']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5sTN9SwED6VZ",
        "colab_type": "text"
      },
      "source": [
        "정답이 있는 위치는 어떻게 맞추는지?  \n",
        "SQUAD 같은 경우에는 context에 정답(text)을 포함하고 있다.  \n",
        "따라서 context와 text 둘 다 토크나이징 한 다음에 context 안에서 text의 위치를 찾아내면 된다.  \n",
        "\n",
        "엄밀히 말하면, text의 시작 토큰의 위치와, 끝 토큰의 위치를 맞추면 되는 것이다. \n",
        "  \n",
        "  다만 **text에서 [CLS]와 [SEP]을 제외하고 생각해야 한다**.  \n",
        "[CLS]는 101, [SEP]은 102에 해당함."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PvhppkOSD-vY",
        "colab_type": "code",
        "outputId": "dbc61acd-ff05-4fe3-91d6-f255463f5162",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "#Context\n",
        "print(tokenizer.encode(train.loc[0, 'context']))\n",
        "#text(정답)\n",
        "print(tokenizer.encode(train.loc[0, 'text']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "([101, 6549, 2135, 1010, 1996, 2082, 2038, 1037, 3234, 2839, 1012, 10234, 1996, 2364, 2311, 1005, 1055, 2751, 8514, 2003, 1037, 3585, 6231, 1997, 1996, 6261, 2984, 1012, 3202, 1999, 2392, 1997, 1996, 2364, 2311, 1998, 5307, 2009, 1010, 2003, 1037, 6967, 6231, 1997, 4828, 2007, 2608, 2039, 14995, 6924, 2007, 1996, 5722, 1000, 2310, 3490, 2618, 4748, 2033, 18168, 5267, 1000, 1012, 2279, 2000, 1996, 2364, 2311, 2003, 1996, 13546, 1997, 1996, 6730, 2540, 1012, 3202, 2369, 1996, 13546, 2003, 1996, 24665, 23052, 1010, 1037, 14042, 2173, 1997, 7083, 1998, 9185, 1012, 2009, 2003, 1037, 15059, 1997, 1996, 24665, 23052, 2012, 10223, 26371, 1010, 2605, 2073, 1996, 6261, 2984, 22353, 2135, 2596, 2000, 3002, 16595, 9648, 4674, 2061, 12083, 9711, 2271, 1999, 8517, 1012, 2012, 1996, 2203, 1997, 1996, 2364, 3298, 1006, 1998, 1999, 1037, 3622, 2240, 2008, 8539, 2083, 1017, 11342, 1998, 1996, 2751, 8514, 1007, 1010, 2003, 1037, 3722, 1010, 2715, 2962, 6231, 1997, 2984, 1012, 102], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n",
            "([101, 3002, 16595, 9648, 4674, 2061, 12083, 9711, 2271, 102], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iR82bu7sNXTV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 세그먼트를 제외하고 토크나이즈 된 것만 추출\n",
        "context = tokenizer.encode(train.loc[0, 'context'])[0]\n",
        "text = tokenizer.encode(train.loc[0, 'text'])[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lYzw8t-zNd_q",
        "colab_type": "code",
        "outputId": "dd765c38-1a0d-4a59-f830-5f060781ee24",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# text에서 [CLS]와 [SEP]에 해당하는 부분 제거\n",
        "text.pop(0) # 첫 토크나이즈 제거\n",
        "text.pop(-1) # 마지막 토크나이즈 제거"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "102"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hg5fBM0qPVK0",
        "colab_type": "code",
        "outputId": "50a18987-1536-4be8-ef87-0517d0f06448",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "text"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[3002, 16595, 9648, 4674, 2061, 12083, 9711, 2271]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XCDZNh11PWu7",
        "colab_type": "code",
        "outputId": "31a3b8a4-f239-497e-e1a1-2582a48d9799",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# 텍스트(정답)의 길이 계산\n",
        "# context를 텍스트의 길이 만큼 슬라이딩 하면서 정답과 일치되는 부분을 찾는다\n",
        "text_slide_len = len(text)\n",
        "        \n",
        "for j in range(0,(len(context))):  \n",
        "  # exist_flag 변수는 context내에 정답이 존재하는지의 여부를 알려줌\n",
        "  exist_flag = 0\n",
        "  if text == context[j:j+text_slide_len]:\n",
        "    # context 내에 정답의 시작 토큰 위치\n",
        "    ans_start = j\n",
        "    # context 내에 정답의 끝 토큰 위치\n",
        "    ans_end = j + text_slide_len - 1\n",
        "    # 정답을 찾았으면 exist_flag 변수를 1로 바꿈\n",
        "    exist_flag = 1\n",
        "    break\n",
        "\n",
        "print(\"ans_start : {}, ans_end : {}\".format(ans_start, ans_end))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ans_start : 114, ans_end : 121\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HYa1qPkARfpV",
        "colab_type": "text"
      },
      "source": [
        "정답의 토큰상 context 내에 위치는 114, 121임을 알 수 있다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2g9jAZn8ROJJ",
        "colab_type": "code",
        "outputId": "ef6b0a8e-6d7a-44a2-cfec-49bdc3957fda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# context와 text가 일치함을 알 수 있다.\n",
        "print(context[114:121 + 1], text)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[3002, 16595, 9648, 4674, 2061, 12083, 9711, 2271] [3002, 16595, 9648, 4674, 2061, 12083, 9711, 2271]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fraw-fe9PL1k",
        "colab_type": "text"
      },
      "source": [
        "우리가 로드하였던 SQUAD 데이터를 **버트 모형의 입력에 맞게 변형해주는 함수**를 정의하도록 하겠습니다.  \n",
        "즉, BERT의 인풋으로 들어갈 함수를 정의하도록 하겠습니다.  \n",
        "87599개의 train 데이터를 한번에 입력할 수 있도록 bert의 인풋으로 바꿔줍니다.\n",
        "\n",
        "방금 보셨다시피, 함수 내부에 tokenizer.encode 함수가 버트 모형을 토큰화해주고 토큰화 된 단어를 인덱스에 맞게 숫자로 바꿔주게 됩니다.  \n",
        "**[CLS] 질문 [SEP] 문장 [SEP]** 이런 방식으로 질문과 문장이 인풋으로 들어가게 됩니다. SEQ_LEN이 384로 지정되어 있어서 길이가 384가 넘는 인풋은 문장 부분이 잘려서 들어가게 됩니다.  \n",
        "그리고 question의 총 길이를 최대 64로 제한하였습니다. 즉 64 미만이면 질문의 길이가 64가 되도록 잘리게 됨\n",
        "\n",
        "SQUAD 문제에서 문장(context) 내에 text(정답)이 포함된다고 미리 말씀 드렸는데요, 길이가 384가 넘는 인풋인 경우에 context 내에 정답을 포함하고 있는 context가 잘려서 정답을 포함하지 않는 경우가 생깁니다. 이번 실습에서는 이러한 경우의 인덱스를 del_list로 지정해서, 빼도록 하겠습니다. (완벽한 구현은 직접 숙제로 해 보시길 권장드립니다.) \n",
        "  \n",
        "    \n",
        "      \n",
        "        \n",
        "  **참고로 버트 인풋으로 들어가는 Question과 Context는   \n",
        "  [CLS] Question [SEP] Context [SEP]의 구조를 가져야 합니다.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3FdBW3-6MBOE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "# data_df(train 데이터의 pandas_dataframe)를 받아서 버트 인풋으로 변형해주는 함수 정의\n",
        "def convert_data(data_df):\n",
        "    global tokenizer\n",
        "    indices, segments, target_start, target_end = [], [], [], []\n",
        "    \n",
        "    for i in tqdm(range(len(data_df))):\n",
        "        # que : question을 버트 인풋으로 들어갈 수 있게 토큰화 한 변수, tokenizer.encode를 사용하면\n",
        "        # 토큰화 된 리스트와, 세그먼트가 같이 나오는데 일단 토큰화 된 리스트만 사용\n",
        "\n",
        "        # doc : context를 버트 인풋으로 들어갈 수 있게 토근화 한 변수\n",
        "        que, _ = tokenizer.encode(data_df[QUESTION_COLUMN][i])\n",
        "        doc, _ = tokenizer.encode(data_df[DATA_COLUMN][i])\n",
        "        \n",
        "        # 토큰화된 context의 맨 앞에 있는 [CLS]에 해당하는 101을 삭제\n",
        "        doc.pop(0)\n",
        "\n",
        "        # que_len, doc_len : 질문의 길이, context의 길이\n",
        "        que_len = len(que)\n",
        "        doc_len = len(doc)\n",
        "\n",
        "        # 만약 question의 길이가 64를 초과하면, 64로 잘라줌\n",
        "\n",
        "        if que_len > 64:\n",
        "          que = que[:63]\n",
        "          # 질문의 끝이 [SEP]이 되도록, [SEP]에 해당하는 102 추가\n",
        "          que.append(102)\n",
        "        \n",
        "        # 버트 인풋으로 들어가는 토큰화된 리스트가 최대 길이인 384가 넘지 않도록 만들어 줌\n",
        "        # 384 미만이면 context를 잘라줌\n",
        "        if len(que+doc) > SEQ_LEN:\n",
        "          while len(que+doc) != SEQ_LEN:\n",
        "            doc.pop(-1)\n",
        "          doc.pop(-1)\n",
        "          #context의 끝이 [SEP]가 되도록 [SEP]에 해당하는 102를 추가해 줌\n",
        "          doc.append(102)\n",
        "\n",
        "        # 문장의 전후관계를 구분해주는 segment는, question은 0이 되도록, context는 1이 되도록, 나머지 부분인 패딩 부분은\n",
        "        # 0이 되도록 만들어 줌\n",
        "        \n",
        "        ############################\n",
        "        ###### Segment 예시 ########\n",
        "        ############################\n",
        "        \n",
        "        # question, context, padding\n",
        "        # 00000000, 1111111, 0000000\n",
        "        \n",
        "        segment = [0]*len(que) + [1]*len(doc) + [0]*(SEQ_LEN-len(que)-len(doc))\n",
        "        \n",
        "        # 만약 question과 context를 합쳤을 때 그 길이가 384 미만이면\n",
        "        # padding 값인 0을 채워주도록 함\n",
        "        if len(que + doc) <= SEQ_LEN:\n",
        "          while len(que+doc) != SEQ_LEN:\n",
        "            doc.append(0)\n",
        "\n",
        "        # ids : question과 context를 합친 버트의 실질적인 인풋\n",
        "\n",
        "        ids = que + doc\n",
        "        \n",
        "        # text 길이만큼 context를 sliding 하면서, context 안에 일치하는 text를 찾았을 경우\n",
        "        # context 내에 text의 시작 위치와 끝 위치를 알려주는 부분 코딩\n",
        "        \n",
        "        text = tokenizer.encode(data_df[TEXT][i])[0]\n",
        "        text_slide_len = len(text[1:-1])\n",
        "        \n",
        "        # exist_flag : context 내에서 text를 찾았을 경우 0에서 1로 전환\n",
        "        for j in range(0,(len(doc))):  \n",
        "            exist_flag = 0\n",
        "            if text[1:-1] == doc[j:j+text_slide_len]:\n",
        "              ans_start = j + len(que)\n",
        "              ans_end = j + text_slide_len - 1 + len(que)\n",
        "              exist_flag = 1\n",
        "              break\n",
        "        \n",
        "        # 만약 context 내에서 text를 찾지 못해서 여전히 exist_flag 가 0인 경우\n",
        "        # 시작값과 끝 값을 SEQ_LEN(384로 지정)\n",
        "        # 향후 시작값과 끝 값이 384인 경우 이 목록은 삭제할 예정임\n",
        "        if exist_flag == 0:\n",
        "          ans_start = SEQ_LEN\n",
        "          ans_end = SEQ_LEN\n",
        "\n",
        "        # 버트 인풋으로 들어가는 ids, segments를 indices, segments에 각각 저장\n",
        "        indices.append(ids)\n",
        "        segments.append(segment)\n",
        "        # 정답에 해당하는 시작 위치인 ans_start와 ans_end를 target_start, target_end에 각각 저장\n",
        "        target_start.append(ans_start)\n",
        "        target_end.append(ans_end)\n",
        "\n",
        "    # indices, segments, ans_start, ans_end를 numpy array로 지정    \n",
        "    indices_x = np.array(indices)\n",
        "    segments = np.array(segments)\n",
        "    target_start = np.array(target_start)\n",
        "    target_end = np.array(target_end)\n",
        "    \n",
        "    # del_list를 지정하여 ans_start와 ans_end가 정답에 해당하지 않는 부분들을 삭제\n",
        "    del_list = np.where(target_start!=SEQ_LEN)[0]\n",
        "    not_del_list = np.where(target_start==SEQ_LEN)[0]\n",
        "    indices_x = indices_x[del_list]\n",
        "    segments = segments[del_list]\n",
        "    target_start = target_start[del_list]\n",
        "    target_end = target_end[del_list]\n",
        "\n",
        "    return [indices_x, segments], [target_start, target_end], not_del_list\n",
        "\n",
        "# 위에 정의한 convert_data 함수를 불러오는 함수를 정의\n",
        "def load_data(pandas_dataframe):\n",
        "    data_df = pandas_dataframe\n",
        "    data_df[DATA_COLUMN] = data_df[DATA_COLUMN].astype(str)\n",
        "    data_df[QUESTION_COLUMN] = data_df[QUESTION_COLUMN].astype(str)\n",
        "    data_x, data_y, del_list = convert_data(data_df)\n",
        "\n",
        "    return data_x, data_y, del_list"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rNUYX-Rntsk_",
        "colab_type": "code",
        "outputId": "2501ee06-0932-4aa7-947a-ca5d019777f0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# 버트에 사용할 수 있도록 train 데이터를 변환\n",
        "train_x, train_y, z = load_data(train)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 87589/87589 [02:59<00:00, 487.81it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLf9jPQtoHYH",
        "colab_type": "text"
      },
      "source": [
        "원래 데이터는 87,599개의 데이터였는데 변환 과정에서 87,353개로 200개정도의 데이터가 소실되었습니다.  \n",
        "question과 context를 합한 것이 384를 넘어서 context 가 정답인 text를 포함하지 않는 케이스일 수도 있고, 변환과정에서 정답을 잃어버린 경우도 있습니다.  \n",
        "이를 보완하기 위한 기법은 상당히 복잡하므로 생략하도록 하겠습니다.   "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JmpCrBEkuW04",
        "colab_type": "code",
        "outputId": "3718df22-2e08-4b33-8aa6-805bd614c431",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(train_x[0].shape, train_y[0].shape)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(87353, 384) (87353,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZ23W0s8MCBt",
        "colab_type": "text"
      },
      "source": [
        "이해가 되셨는지요?  \n",
        "구글 깃허브에서 다운받았던 사전학습된 모델을 colab으로 로드합니다.  \n",
        "Training을 False로 두어서 Bert 모델에서, 마지막 트랜스포머 계층까지만 모델이 로드되게 합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": false,
        "id": "8auSqxdHtslD",
        "colab_type": "code",
        "outputId": "4c765857-255f-468b-baec-4d99c86286b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Training False로 두어서 마지막 레이어의 출력 차원을 (batch_size, 384, 768)로 정함\n",
        "model = load_trained_model_from_checkpoint(\n",
        "    config_path,\n",
        "    checkpoint_path,\n",
        "    training=False,\n",
        "    trainable=True,\n",
        "    seq_len=SEQ_LEN,)\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_6\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Input-Token (InputLayer)        (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 23440896    Input-Token[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
            "                                                                 Embedding-Segment[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
            "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
            "==================================================================================================\n",
            "Total params: 108,793,344\n",
            "Trainable params: 108,793,344\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ptok2n0MHoV",
        "colab_type": "text"
      },
      "source": [
        "모델의 구조를 확인합니다.  \n",
        "총 12층의 트랜스포머 계층이 있음을 확인할 수 있습니다.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1N8hZvNMMWL",
        "colab_type": "text"
      },
      "source": [
        "Transfer learning을 위해 Custom Layer를 작성해 줍니다.  \n",
        "NonMasking 함수를 지정해서, Bert 모형의 자체 Masking 된 텐서들을 풀어줘야 합니다.  \n",
        "이번 튜토리얼에서 만약 NonMasking 클래스를 만들지 않는다면, Bert 모형을 훈련할 수 없습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H99O5l4X8xyb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class NonMasking(Layer):   \n",
        "    def __init__(self, **kwargs):   \n",
        "        self.supports_masking = True  \n",
        "        super(NonMasking, self).__init__(**kwargs)   \n",
        "  \n",
        "    def build(self, input_shape):   \n",
        "        input_shape = input_shape   \n",
        "  \n",
        "    def compute_mask(self, input, input_mask=None):   \n",
        "        return None   \n",
        "  \n",
        "    def call(self, x, mask=None):   \n",
        "        return x   \n",
        "  \n",
        "    def get_output_shape_for(self, input_shape):   \n",
        "        return input_shape  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0BgB9WtubcgQ",
        "colab_type": "text"
      },
      "source": [
        "Keras의 Custom Layer를 생성하도록 합니다.\n",
        "Squad_Prediction 클래스는 버트 모형의 레이어에 Squad 문제를 풀 수 있도록, 직접 레이어를 덧붙이는 클래스입니다.\n",
        "\n",
        "  \n",
        "  \n",
        "Bert 모형의 마지막 아웃풋을 이어서, (batch_size, 384, 768)의 텐서 모양을 (batch_size, 384, 2)로 만들어주는 텐서를 곱해줍니다.  \n",
        "이 다음에 i) (batch_size, 384), ii) (batch_size, 384)의 아웃풋을 출력할 수 있게 하나의 텐서를 두개로 잘라줍니다.  \n",
        "  \n",
        "왜 끝이 384냐면, 384개의 위치를 예측하기 때문입니다. 단어의 위치의 최대 개수는 384개로 앞서 지정하였습니다.(SEQ_LEN)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDUDN-nZpihk",
        "colab_type": "text"
      },
      "source": [
        "Relu보다 성능이 좋다고 알려진 Gelu 함수를 적용하도록 하겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WTDuSizb9IKO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def bert_gelu(x):\n",
        "    \"\"\"Gaussian Error Linear Unit.\n",
        "    This is a smoother version of the RELU.\n",
        "    Original paper: https://arxiv.org/abs/1606.08415\n",
        "    Args:\n",
        "        x: float Tensor to perform activation.\n",
        "    Returns:\n",
        "        `x` with the GELU activation applied.\n",
        "    \"\"\"\n",
        "    cdf = 0.5 * (1.0 + K.tanh(\n",
        "        (np.sqrt(2 / np.pi) * (x + 0.044715 * K.pow(x, 3)))))\n",
        "    return x * cdf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LxnNOSu_5b6Z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Start_End_Prediction(Layer):\n",
        "\n",
        "    def __init__(self,seq_len, **kwargs):\n",
        "        \n",
        "        self.seq_len = seq_len\n",
        "        self.supports_masking = True\n",
        "        super(Start_End_Prediction, self).__init__(**kwargs)\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        \n",
        "        # 버트 모델의 맨 마지막 출력 레이어인 (batch_size, 384, 768)에 곱해져서\n",
        "        # 아웃풋 차원을 (batch_size, 384, 2)로 만들어주는 텐서 self.W 정의\n",
        "        self.W = self.add_weight(name='kernel', \n",
        "                                 shape=(input_shape[2],2),\n",
        "                                 initializer='uniform',\n",
        "                                 trainable=True)\n",
        "        super(Start_End_Prediction, self).build(input_shape)\n",
        "\n",
        "    def call(self, x):\n",
        "        \n",
        "        x = K.reshape(x, shape=(-1,self.seq_len,K.shape(x)[2]))\n",
        "        # 출력 차원을 (batch_size, 384, 768)로 재정의\n",
        "        x = K.dot(x, self.W)\n",
        "        # 위에 정의한 self.W를 x에 행렬곱\n",
        "        # (batch_size, 384, 768) * (384, 2) = (batch_size, 384, 2)\n",
        "        x = K.permute_dimensions(x, (2,0,1))\n",
        "        # (batch_size, 384, 2) -> (2, batch_size, 384)\n",
        "\n",
        "        self.start_logits, self.end_logits = x[0], x[1]\n",
        "        # (2, batch_size, 384)를 각각 (batch_size, 384)로 쪼갬\n",
        "        # start_logit = (batch_size, 384)\n",
        "        # end_logit = (batch_size, 384)\n",
        "        \n",
        "        self.start_logits = bert_gelu(self.start_logits)\n",
        "        self.end_logits = bert_gelu(self.end_logits)\n",
        "        # gelu 레이어 통과\n",
        "\n",
        "        self.start_logits = K.softmax(self.start_logits, axis=-1)\n",
        "        self.end_logits = K.softmax(self.end_logits, axis=-1)\n",
        "        # softmax layer 통과 -> 384개의 아웃풋값이 확률값으로 변함\n",
        "        \n",
        "        return [self.start_logits, self.end_logits]\n",
        "\n",
        "    def compute_output_shape(self, input_shape):\n",
        "      # 케라스의 Custom Layer를 정의할 때에는 compute_output_shape에 output 차원을 정의해줘야 함\n",
        "        return [(input_shape[0], self.seq_len), (input_shape[0], self.seq_len)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3uYTT6WYgiBG",
        "colab_type": "text"
      },
      "source": [
        "BERT 모델을 출력하는 함수를 지정합니다.  \n",
        "start_answer, end_answer를 예측하게 됩니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NcbOzpEb0wxU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import merge, dot, concatenate\n",
        "from keras import metrics\n",
        "import numpy as np\n",
        "\n",
        "# 버트 모델을 불러오는 함수 정의\n",
        "def get_bert_finetuning_model(model):\n",
        "  inputs = model.inputs[:2]\n",
        "  # input은 token-embedding과 segment-embedding\n",
        "  \n",
        "  bert_transformer = model.layers[-1].output\n",
        "  # transformer의 아웃풋인 (batch_size, 384, 768)\n",
        "  \n",
        "  x = NonMasking()(bert_transformer)\n",
        "  # NoNMasking으로 마스킹 된 텐서 풀어주기\n",
        "\n",
        "  outputs_start, outputs_end = Start_End_Prediction(SEQ_LEN)(x)\n",
        "  # 최종적으로 (batch_size, 384), (batch_size,384)의 정답 토큰의 시작과 끝을 출력\n",
        "\n",
        "  bert_model = keras.models.Model(inputs, [outputs_start, outputs_end])\n",
        "\n",
        "  optimizer_warmup = RAdam(learning_rate = 1.5e-5, warmup_proportion=0.2, epsilon=1e-6, weight_decay=0.01)\n",
        "  # learning_rate을 0부터 1.5e-5까지 점진적으로 올려주는 optimizer 정의\n",
        "\n",
        "  bert_model.compile(\n",
        "      optimizer = optimizer_warmup,\n",
        "      loss = 'sparse_categorical_crossentropy',\n",
        "      metrics=['accuracy'])\n",
        "  # 최종 모델 compile!\n",
        "\n",
        "  return bert_model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H2LAk6d5dm4A",
        "colab_type": "text"
      },
      "source": [
        "**모델의 FLOW를 확인해 보도록 하겠습니다.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "scWw12ysbPZ9",
        "colab_type": "code",
        "outputId": "9a15d9da-80a9-4207-81da-3cd960a4ca96",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "from IPython.display import SVG\n",
        "from keras.utils import model_to_dot\n",
        "\n",
        "\n",
        "SVG(model_to_dot(get_bert_finetuning_model(model), dpi=65).create(prog='dot', format='svg'))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.SVG object>"
            ],
            "image/svg+xml": "<svg height=\"6829pt\" viewBox=\"0.00 0.00 549.50 7564.00\" width=\"496pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(.9028 .9028) rotate(0) translate(4 7560)\">\n<title>G</title>\n<polygon fill=\"#ffffff\" points=\"-4,4 -4,-7560 545.5,-7560 545.5,4 -4,4\" stroke=\"transparent\"/>\n<!-- 140680804097384 -->\n<g class=\"node\" id=\"node1\">\n<title>140680804097384</title>\n<polygon fill=\"none\" points=\"110.5,-7519.5 110.5,-7555.5 270.5,-7555.5 270.5,-7519.5 110.5,-7519.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.5\" y=\"-7533.8\">Input-Token: InputLayer</text>\n</g>\n<!-- 140684647895224 -->\n<g class=\"node\" id=\"node3\">\n<title>140684647895224</title>\n<polygon fill=\"none\" points=\"72,-7446.5 72,-7482.5 309,-7482.5 309,-7446.5 72,-7446.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"190.5\" y=\"-7460.8\">Embedding-Token: TokenEmbedding</text>\n</g>\n<!-- 140680804097384&#45;&gt;140684647895224 -->\n<g class=\"edge\" id=\"edge1\">\n<title>140680804097384-&gt;140684647895224</title>\n<path d=\"M190.5,-7519.4551C190.5,-7511.3828 190.5,-7501.6764 190.5,-7492.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"194.0001,-7492.5903 190.5,-7482.5904 187.0001,-7492.5904 194.0001,-7492.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680773830416 -->\n<g class=\"node\" id=\"node2\">\n<title>140680773830416</title>\n<polygon fill=\"none\" points=\"348,-7519.5 348,-7555.5 521,-7555.5 521,-7519.5 348,-7519.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"434.5\" y=\"-7533.8\">Input-Segment: InputLayer</text>\n</g>\n<!-- 140680804096488 -->\n<g class=\"node\" id=\"node4\">\n<title>140680804096488</title>\n<polygon fill=\"none\" points=\"327.5,-7446.5 327.5,-7482.5 541.5,-7482.5 541.5,-7446.5 327.5,-7446.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"434.5\" y=\"-7460.8\">Embedding-Segment: Embedding</text>\n</g>\n<!-- 140680773830416&#45;&gt;140680804096488 -->\n<g class=\"edge\" id=\"edge2\">\n<title>140680773830416-&gt;140680804096488</title>\n<path d=\"M434.5,-7519.4551C434.5,-7511.3828 434.5,-7501.6764 434.5,-7492.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"438.0001,-7492.5903 434.5,-7482.5904 431.0001,-7492.5904 438.0001,-7492.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680804097832 -->\n<g class=\"node\" id=\"node5\">\n<title>140680804097832</title>\n<polygon fill=\"none\" points=\"206.5,-7373.5 206.5,-7409.5 418.5,-7409.5 418.5,-7373.5 206.5,-7373.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7387.8\">Embedding-Token-Segment: Add</text>\n</g>\n<!-- 140684647895224&#45;&gt;140680804097832 -->\n<g class=\"edge\" id=\"edge3\">\n<title>140684647895224-&gt;140680804097832</title>\n<path d=\"M220.6573,-7446.4551C236.6407,-7436.8912 256.4579,-7425.0334 273.5882,-7414.7833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"275.4828,-7417.7284 282.2668,-7409.5904 271.8885,-7411.7216 275.4828,-7417.7284\" stroke=\"#000000\"/>\n</g>\n<!-- 140680804096488&#45;&gt;140680804097832 -->\n<g class=\"edge\" id=\"edge4\">\n<title>140680804096488-&gt;140680804097832</title>\n<path d=\"M404.3427,-7446.4551C388.3593,-7436.8912 368.5421,-7425.0334 351.4118,-7414.7833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"353.1115,-7411.7216 342.7332,-7409.5904 349.5172,-7417.7284 353.1115,-7411.7216\" stroke=\"#000000\"/>\n</g>\n<!-- 140680773395792 -->\n<g class=\"node\" id=\"node6\">\n<title>140680773395792</title>\n<polygon fill=\"none\" points=\"183.5,-7300.5 183.5,-7336.5 441.5,-7336.5 441.5,-7300.5 183.5,-7300.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7314.8\">Embedding-Position: PositionEmbedding</text>\n</g>\n<!-- 140680804097832&#45;&gt;140680773395792 -->\n<g class=\"edge\" id=\"edge5\">\n<title>140680804097832-&gt;140680773395792</title>\n<path d=\"M312.5,-7373.4551C312.5,-7365.3828 312.5,-7355.6764 312.5,-7346.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.0001,-7346.5903 312.5,-7336.5904 309.0001,-7346.5904 316.0001,-7346.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680735956496 -->\n<g class=\"node\" id=\"node7\">\n<title>140680735956496</title>\n<polygon fill=\"none\" points=\"215.5,-7227.5 215.5,-7263.5 409.5,-7263.5 409.5,-7227.5 215.5,-7227.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7241.8\">Embedding-Dropout: Dropout</text>\n</g>\n<!-- 140680773395792&#45;&gt;140680735956496 -->\n<g class=\"edge\" id=\"edge6\">\n<title>140680773395792-&gt;140680735956496</title>\n<path d=\"M312.5,-7300.4551C312.5,-7292.3828 312.5,-7282.6764 312.5,-7273.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.0001,-7273.5903 312.5,-7263.5904 309.0001,-7273.5904 316.0001,-7273.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680735917392 -->\n<g class=\"node\" id=\"node8\">\n<title>140680735917392</title>\n<polygon fill=\"none\" points=\"190,-7154.5 190,-7190.5 435,-7190.5 435,-7154.5 190,-7154.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-7168.8\">Embedding-Norm: LayerNormalization</text>\n</g>\n<!-- 140680735956496&#45;&gt;140680735917392 -->\n<g class=\"edge\" id=\"edge7\">\n<title>140680735956496-&gt;140680735917392</title>\n<path d=\"M312.5,-7227.4551C312.5,-7219.3828 312.5,-7209.6764 312.5,-7200.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.0001,-7200.5903 312.5,-7190.5904 309.0001,-7200.5904 316.0001,-7200.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680735807584 -->\n<g class=\"node\" id=\"node9\">\n<title>140680735807584</title>\n<polygon fill=\"none\" points=\"42.5,-7081.5 42.5,-7117.5 384.5,-7117.5 384.5,-7081.5 42.5,-7081.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"213.5\" y=\"-7095.8\">Encoder-1-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680735917392&#45;&gt;140680735807584 -->\n<g class=\"edge\" id=\"edge8\">\n<title>140680735917392-&gt;140680735807584</title>\n<path d=\"M288.0281,-7154.4551C275.4149,-7145.1545 259.8597,-7133.6844 246.2286,-7123.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"248.1592,-7120.7082 238.0335,-7117.5904 244.0049,-7126.3421 248.1592,-7120.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734092032 -->\n<g class=\"node\" id=\"node11\">\n<title>140680734092032</title>\n<polygon fill=\"none\" points=\"173,-6935.5 173,-6971.5 454,-6971.5 454,-6935.5 173,-6935.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6949.8\">Encoder-1-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680735917392&#45;&gt;140680734092032 -->\n<g class=\"edge\" id=\"edge10\">\n<title>140680735917392-&gt;140680734092032</title>\n<path d=\"M355.0685,-7154.3804C370.0065,-7145.7 385.1359,-7133.7088 393.5,-7118 401.1793,-7103.5774 400.9885,-7027.2923 390.5,-7008 383.7016,-6995.4953 372.6452,-6985.1998 361.0052,-6977.0548\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"362.7688,-6974.0266 352.4788,-6971.5031 358.9492,-6979.8927 362.7688,-6974.0266\" stroke=\"#000000\"/>\n</g>\n<!-- 140680735239360 -->\n<g class=\"node\" id=\"node10\">\n<title>140680735239360</title>\n<polygon fill=\"none\" points=\"53,-7008.5 53,-7044.5 382,-7044.5 382,-7008.5 53,-7008.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217.5\" y=\"-7022.8\">Encoder-1-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680735807584&#45;&gt;140680735239360 -->\n<g class=\"edge\" id=\"edge9\">\n<title>140680735807584-&gt;140680735239360</title>\n<path d=\"M214.4888,-7081.4551C214.9311,-7073.3828 215.4629,-7063.6764 215.9558,-7054.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"219.4563,-7054.7669 216.5087,-7044.5904 212.4668,-7054.3839 219.4563,-7054.7669\" stroke=\"#000000\"/>\n</g>\n<!-- 140680735239360&#45;&gt;140680734092032 -->\n<g class=\"edge\" id=\"edge11\">\n<title>140680735239360-&gt;140680734092032</title>\n<path d=\"M241.2303,-7008.4551C253.3459,-6999.2422 268.2608,-6987.9006 281.3886,-6977.918\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"283.8684,-6980.4293 289.7099,-6971.5904 279.6314,-6974.8573 283.8684,-6980.4293\" stroke=\"#000000\"/>\n</g>\n<!-- 140680740564384 -->\n<g class=\"node\" id=\"node12\">\n<title>140680740564384</title>\n<polygon fill=\"none\" points=\"123.5,-6862.5 123.5,-6898.5 503.5,-6898.5 503.5,-6862.5 123.5,-6862.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6876.8\">Encoder-1-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680734092032&#45;&gt;140680740564384 -->\n<g class=\"edge\" id=\"edge12\">\n<title>140680734092032-&gt;140680740564384</title>\n<path d=\"M313.5,-6935.4551C313.5,-6927.3828 313.5,-6917.6764 313.5,-6908.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"317.0001,-6908.5903 313.5,-6898.5904 310.0001,-6908.5904 317.0001,-6908.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680740562592 -->\n<g class=\"node\" id=\"node13\">\n<title>140680740562592</title>\n<polygon fill=\"none\" points=\"118.5,-6789.5 118.5,-6825.5 360.5,-6825.5 360.5,-6789.5 118.5,-6789.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"239.5\" y=\"-6803.8\">Encoder-1-FeedForward: FeedForward</text>\n</g>\n<!-- 140680740564384&#45;&gt;140680740562592 -->\n<g class=\"edge\" id=\"edge13\">\n<title>140680740564384-&gt;140680740562592</title>\n<path d=\"M295.2079,-6862.4551C286.1356,-6853.5054 275.0271,-6842.547 265.1232,-6832.7769\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"267.4152,-6830.1215 257.8382,-6825.5904 262.4992,-6835.1049 267.4152,-6830.1215\" stroke=\"#000000\"/>\n</g>\n<!-- 140680742391424 -->\n<g class=\"node\" id=\"node15\">\n<title>140680742391424</title>\n<polygon fill=\"none\" points=\"203.5,-6643.5 203.5,-6679.5 423.5,-6679.5 423.5,-6643.5 203.5,-6643.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6657.8\">Encoder-1-FeedForward-Add: Add</text>\n</g>\n<!-- 140680740564384&#45;&gt;140680742391424 -->\n<g class=\"edge\" id=\"edge15\">\n<title>140680740564384-&gt;140680742391424</title>\n<path d=\"M339.1147,-6862.4444C350.4534,-6852.9752 362.7322,-6840.3575 369.5,-6826 390.3764,-6781.7122 395.6809,-6760.609 375.5,-6716 370.2852,-6704.4728 361.2939,-6694.4147 351.751,-6686.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"353.6666,-6683.2331 343.676,-6679.7064 349.2868,-6688.6936 353.6666,-6683.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680741242248 -->\n<g class=\"node\" id=\"node14\">\n<title>140680741242248</title>\n<polygon fill=\"none\" points=\"99,-6716.5 99,-6752.5 366,-6752.5 366,-6716.5 99,-6716.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.5\" y=\"-6730.8\">Encoder-1-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680740562592&#45;&gt;140680741242248 -->\n<g class=\"edge\" id=\"edge14\">\n<title>140680740562592-&gt;140680741242248</title>\n<path d=\"M237.7697,-6789.4551C236.9956,-6781.3828 236.0649,-6771.6764 235.2024,-6762.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"238.6733,-6762.2106 234.2347,-6752.5904 231.7053,-6762.8788 238.6733,-6762.2106\" stroke=\"#000000\"/>\n</g>\n<!-- 140680741242248&#45;&gt;140680742391424 -->\n<g class=\"edge\" id=\"edge16\">\n<title>140680741242248-&gt;140680742391424</title>\n<path d=\"M252.5225,-6716.4551C262.5502,-6707.4177 274.8509,-6696.3319 285.7714,-6686.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"288.3419,-6688.885 293.4271,-6679.5904 283.6556,-6683.6852 288.3419,-6688.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680743415704 -->\n<g class=\"node\" id=\"node16\">\n<title>140680743415704</title>\n<polygon fill=\"none\" points=\"154.5,-6570.5 154.5,-6606.5 472.5,-6606.5 472.5,-6570.5 154.5,-6570.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"313.5\" y=\"-6584.8\">Encoder-1-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680742391424&#45;&gt;140680743415704 -->\n<g class=\"edge\" id=\"edge17\">\n<title>140680742391424-&gt;140680743415704</title>\n<path d=\"M313.5,-6643.4551C313.5,-6635.3828 313.5,-6625.6764 313.5,-6616.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"317.0001,-6616.5903 313.5,-6606.5904 310.0001,-6616.5904 317.0001,-6616.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680743415536 -->\n<g class=\"node\" id=\"node17\">\n<title>140680743415536</title>\n<polygon fill=\"none\" points=\"43.5,-6497.5 43.5,-6533.5 385.5,-6533.5 385.5,-6497.5 43.5,-6497.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"214.5\" y=\"-6511.8\">Encoder-2-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680743415704&#45;&gt;140680743415536 -->\n<g class=\"edge\" id=\"edge18\">\n<title>140680743415704-&gt;140680743415536</title>\n<path d=\"M289.0281,-6570.4551C276.4149,-6561.1545 260.8597,-6549.6844 247.2286,-6539.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"249.1592,-6536.7082 239.0335,-6533.5904 245.0049,-6542.3421 249.1592,-6536.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680745388856 -->\n<g class=\"node\" id=\"node19\">\n<title>140680745388856</title>\n<polygon fill=\"none\" points=\"175,-6351.5 175,-6387.5 456,-6387.5 456,-6351.5 175,-6351.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"315.5\" y=\"-6365.8\">Encoder-2-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680743415704&#45;&gt;140680745388856 -->\n<g class=\"edge\" id=\"edge20\">\n<title>140680743415704-&gt;140680745388856</title>\n<path d=\"M356.0685,-6570.3804C371.0065,-6561.7 386.1359,-6549.7088 394.5,-6534 402.5894,-6518.8071 400.2511,-6441.9102 390.5,-6424 383.7494,-6411.601 372.8204,-6401.3006 361.3687,-6393.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"363.2521,-6390.1654 352.99,-6387.531 359.3694,-6395.9899 363.2521,-6390.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 140680744783208 -->\n<g class=\"node\" id=\"node18\">\n<title>140680744783208</title>\n<polygon fill=\"none\" points=\"53,-6424.5 53,-6460.5 382,-6460.5 382,-6424.5 53,-6424.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217.5\" y=\"-6438.8\">Encoder-2-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680743415536&#45;&gt;140680744783208 -->\n<g class=\"edge\" id=\"edge19\">\n<title>140680743415536-&gt;140680744783208</title>\n<path d=\"M215.2416,-6497.4551C215.5733,-6489.3828 215.9722,-6479.6764 216.3418,-6470.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"219.8429,-6470.7257 216.7566,-6460.5904 212.8488,-6470.4382 219.8429,-6470.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680744783208&#45;&gt;140680745388856 -->\n<g class=\"edge\" id=\"edge21\">\n<title>140680744783208-&gt;140680745388856</title>\n<path d=\"M241.7247,-6424.4551C254.2105,-6415.1545 269.6086,-6403.6844 283.102,-6393.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"285.2855,-6396.371 291.2143,-6387.5904 281.1039,-6390.7573 285.2855,-6396.371\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734662104 -->\n<g class=\"node\" id=\"node20\">\n<title>140680734662104</title>\n<polygon fill=\"none\" points=\"125.5,-6278.5 125.5,-6314.5 505.5,-6314.5 505.5,-6278.5 125.5,-6278.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"315.5\" y=\"-6292.8\">Encoder-2-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680745388856&#45;&gt;140680734662104 -->\n<g class=\"edge\" id=\"edge22\">\n<title>140680745388856-&gt;140680734662104</title>\n<path d=\"M315.5,-6351.4551C315.5,-6343.3828 315.5,-6333.6764 315.5,-6324.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"319.0001,-6324.5903 315.5,-6314.5904 312.0001,-6324.5904 319.0001,-6324.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734661768 -->\n<g class=\"node\" id=\"node21\">\n<title>140680734661768</title>\n<polygon fill=\"none\" points=\"113.5,-6205.5 113.5,-6241.5 355.5,-6241.5 355.5,-6205.5 113.5,-6205.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"234.5\" y=\"-6219.8\">Encoder-2-FeedForward: FeedForward</text>\n</g>\n<!-- 140680734662104&#45;&gt;140680734661768 -->\n<g class=\"edge\" id=\"edge23\">\n<title>140680734662104-&gt;140680734661768</title>\n<path d=\"M295.4775,-6278.4551C285.4498,-6269.4177 273.1491,-6258.3319 262.2286,-6248.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"264.3444,-6245.6852 254.5729,-6241.5904 259.6581,-6250.885 264.3444,-6245.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680733674016 -->\n<g class=\"node\" id=\"node23\">\n<title>140680733674016</title>\n<polygon fill=\"none\" points=\"199.5,-6059.5 199.5,-6095.5 419.5,-6095.5 419.5,-6059.5 199.5,-6059.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"309.5\" y=\"-6073.8\">Encoder-2-FeedForward-Add: Add</text>\n</g>\n<!-- 140680734662104&#45;&gt;140680733674016 -->\n<g class=\"edge\" id=\"edge25\">\n<title>140680734662104-&gt;140680733674016</title>\n<path d=\"M337.3471,-6278.291C347.3936,-6268.6134 358.4097,-6255.8624 364.5,-6242 384.2044,-6197.1498 391.6917,-6176.6329 371.5,-6132 366.2852,-6120.4728 357.2939,-6110.4147 347.751,-6102.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"349.6666,-6099.2331 339.676,-6095.7064 345.2868,-6104.6936 349.6666,-6099.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680733367544 -->\n<g class=\"node\" id=\"node22\">\n<title>140680733367544</title>\n<polygon fill=\"none\" points=\"95,-6132.5 95,-6168.5 362,-6168.5 362,-6132.5 95,-6132.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"228.5\" y=\"-6146.8\">Encoder-2-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680734661768&#45;&gt;140680733367544 -->\n<g class=\"edge\" id=\"edge24\">\n<title>140680734661768-&gt;140680733367544</title>\n<path d=\"M233.0169,-6205.4551C232.3534,-6197.3828 231.5556,-6187.6764 230.8163,-6178.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"234.2944,-6178.27 229.9869,-6168.5904 227.3179,-6178.8435 234.2944,-6178.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680733367544&#45;&gt;140680733674016 -->\n<g class=\"edge\" id=\"edge26\">\n<title>140680733367544-&gt;140680733674016</title>\n<path d=\"M248.5225,-6132.4551C258.5502,-6123.4177 270.8509,-6112.3319 281.7714,-6102.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"284.3419,-6104.885 289.4271,-6095.5904 279.6556,-6099.6852 284.3419,-6104.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734248296 -->\n<g class=\"node\" id=\"node24\">\n<title>140680734248296</title>\n<polygon fill=\"none\" points=\"150.5,-5986.5 150.5,-6022.5 468.5,-6022.5 468.5,-5986.5 150.5,-5986.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"309.5\" y=\"-6000.8\">Encoder-2-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680733674016&#45;&gt;140680734248296 -->\n<g class=\"edge\" id=\"edge27\">\n<title>140680733674016-&gt;140680734248296</title>\n<path d=\"M309.5,-6059.4551C309.5,-6051.3828 309.5,-6041.6764 309.5,-6032.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"313.0001,-6032.5903 309.5,-6022.5904 306.0001,-6032.5904 313.0001,-6032.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734247512 -->\n<g class=\"node\" id=\"node25\">\n<title>140680734247512</title>\n<polygon fill=\"none\" points=\"39.5,-5913.5 39.5,-5949.5 381.5,-5949.5 381.5,-5913.5 39.5,-5913.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"210.5\" y=\"-5927.8\">Encoder-3-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680734248296&#45;&gt;140680734247512 -->\n<g class=\"edge\" id=\"edge28\">\n<title>140680734248296-&gt;140680734247512</title>\n<path d=\"M285.0281,-5986.4551C272.4149,-5977.1545 256.8597,-5965.6844 243.2286,-5955.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"245.1592,-5952.7082 235.0335,-5949.5904 241.0049,-5958.3421 245.1592,-5952.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680735188976 -->\n<g class=\"node\" id=\"node27\">\n<title>140680735188976</title>\n<polygon fill=\"none\" points=\"171,-5767.5 171,-5803.5 452,-5803.5 452,-5767.5 171,-5767.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"311.5\" y=\"-5781.8\">Encoder-3-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680734248296&#45;&gt;140680735188976 -->\n<g class=\"edge\" id=\"edge30\">\n<title>140680734248296-&gt;140680735188976</title>\n<path d=\"M352.0685,-5986.3804C367.0065,-5977.7 382.1359,-5965.7088 390.5,-5950 398.5894,-5934.8071 396.2511,-5857.9102 386.5,-5840 379.7494,-5827.601 368.8204,-5817.3006 357.3687,-5809.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"359.2521,-5806.1654 348.99,-5803.531 355.3694,-5811.9899 359.2521,-5806.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734743352 -->\n<g class=\"node\" id=\"node26\">\n<title>140680734743352</title>\n<polygon fill=\"none\" points=\"49,-5840.5 49,-5876.5 378,-5876.5 378,-5840.5 49,-5840.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"213.5\" y=\"-5854.8\">Encoder-3-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680734247512&#45;&gt;140680734743352 -->\n<g class=\"edge\" id=\"edge29\">\n<title>140680734247512-&gt;140680734743352</title>\n<path d=\"M211.2416,-5913.4551C211.5733,-5905.3828 211.9722,-5895.6764 212.3418,-5886.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"215.8429,-5886.7257 212.7566,-5876.5904 208.8488,-5886.4382 215.8429,-5886.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680734743352&#45;&gt;140680735188976 -->\n<g class=\"edge\" id=\"edge31\">\n<title>140680734743352-&gt;140680735188976</title>\n<path d=\"M237.7247,-5840.4551C250.2105,-5831.1545 265.6086,-5819.6844 279.102,-5809.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"281.2855,-5812.371 287.2143,-5803.5904 277.1039,-5806.7573 281.2855,-5812.371\" stroke=\"#000000\"/>\n</g>\n<!-- 140680737332640 -->\n<g class=\"node\" id=\"node28\">\n<title>140680737332640</title>\n<polygon fill=\"none\" points=\"121.5,-5694.5 121.5,-5730.5 501.5,-5730.5 501.5,-5694.5 121.5,-5694.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"311.5\" y=\"-5708.8\">Encoder-3-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680735188976&#45;&gt;140680737332640 -->\n<g class=\"edge\" id=\"edge32\">\n<title>140680735188976-&gt;140680737332640</title>\n<path d=\"M311.5,-5767.4551C311.5,-5759.3828 311.5,-5749.6764 311.5,-5740.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"315.0001,-5740.5903 311.5,-5730.5904 308.0001,-5740.5904 315.0001,-5740.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680737975040 -->\n<g class=\"node\" id=\"node29\">\n<title>140680737975040</title>\n<polygon fill=\"none\" points=\"109.5,-5621.5 109.5,-5657.5 351.5,-5657.5 351.5,-5621.5 109.5,-5621.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"230.5\" y=\"-5635.8\">Encoder-3-FeedForward: FeedForward</text>\n</g>\n<!-- 140680737332640&#45;&gt;140680737975040 -->\n<g class=\"edge\" id=\"edge33\">\n<title>140680737332640-&gt;140680737975040</title>\n<path d=\"M291.4775,-5694.4551C281.4498,-5685.4177 269.1491,-5674.3319 258.2286,-5664.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"260.3444,-5661.6852 250.5729,-5657.5904 255.6581,-5666.885 260.3444,-5661.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680738469760 -->\n<g class=\"node\" id=\"node31\">\n<title>140680738469760</title>\n<polygon fill=\"none\" points=\"195.5,-5475.5 195.5,-5511.5 415.5,-5511.5 415.5,-5475.5 195.5,-5475.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"305.5\" y=\"-5489.8\">Encoder-3-FeedForward-Add: Add</text>\n</g>\n<!-- 140680737332640&#45;&gt;140680738469760 -->\n<g class=\"edge\" id=\"edge35\">\n<title>140680737332640-&gt;140680738469760</title>\n<path d=\"M333.3471,-5694.291C343.3936,-5684.6134 354.4097,-5671.8624 360.5,-5658 380.2044,-5613.1498 387.6917,-5592.6329 367.5,-5548 362.2852,-5536.4728 353.2939,-5526.4147 343.751,-5518.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"345.6666,-5515.2331 335.676,-5511.7064 341.2868,-5520.6936 345.6666,-5515.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680737864392 -->\n<g class=\"node\" id=\"node30\">\n<title>140680737864392</title>\n<polygon fill=\"none\" points=\"91,-5548.5 91,-5584.5 358,-5584.5 358,-5548.5 91,-5548.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"224.5\" y=\"-5562.8\">Encoder-3-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680737975040&#45;&gt;140680737864392 -->\n<g class=\"edge\" id=\"edge34\">\n<title>140680737975040-&gt;140680737864392</title>\n<path d=\"M229.0169,-5621.4551C228.3534,-5613.3828 227.5556,-5603.6764 226.8163,-5594.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"230.2944,-5594.27 225.9869,-5584.5904 223.3179,-5594.8435 230.2944,-5594.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680737864392&#45;&gt;140680738469760 -->\n<g class=\"edge\" id=\"edge36\">\n<title>140680737864392-&gt;140680738469760</title>\n<path d=\"M244.5225,-5548.4551C254.5502,-5539.4177 266.8509,-5528.3319 277.7714,-5518.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"280.3419,-5520.885 285.4271,-5511.5904 275.6556,-5515.6852 280.3419,-5520.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680739572480 -->\n<g class=\"node\" id=\"node32\">\n<title>140680739572480</title>\n<polygon fill=\"none\" points=\"146.5,-5402.5 146.5,-5438.5 464.5,-5438.5 464.5,-5402.5 146.5,-5402.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"305.5\" y=\"-5416.8\">Encoder-3-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680738469760&#45;&gt;140680739572480 -->\n<g class=\"edge\" id=\"edge37\">\n<title>140680738469760-&gt;140680739572480</title>\n<path d=\"M305.5,-5475.4551C305.5,-5467.3828 305.5,-5457.6764 305.5,-5448.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"309.0001,-5448.5903 305.5,-5438.5904 302.0001,-5448.5904 309.0001,-5448.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680739571696 -->\n<g class=\"node\" id=\"node33\">\n<title>140680739571696</title>\n<polygon fill=\"none\" points=\"35.5,-5329.5 35.5,-5365.5 377.5,-5365.5 377.5,-5329.5 35.5,-5329.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"206.5\" y=\"-5343.8\">Encoder-4-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680739572480&#45;&gt;140680739571696 -->\n<g class=\"edge\" id=\"edge38\">\n<title>140680739572480-&gt;140680739571696</title>\n<path d=\"M281.0281,-5402.4551C268.4149,-5393.1545 252.8597,-5381.6844 239.2286,-5371.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"241.1592,-5368.7082 231.0335,-5365.5904 237.0049,-5374.3421 241.1592,-5368.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680739906784 -->\n<g class=\"node\" id=\"node35\">\n<title>140680739906784</title>\n<polygon fill=\"none\" points=\"167,-5183.5 167,-5219.5 448,-5219.5 448,-5183.5 167,-5183.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"307.5\" y=\"-5197.8\">Encoder-4-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680739572480&#45;&gt;140680739906784 -->\n<g class=\"edge\" id=\"edge40\">\n<title>140680739572480-&gt;140680739906784</title>\n<path d=\"M348.0685,-5402.3804C363.0065,-5393.7 378.1359,-5381.7088 386.5,-5366 394.5894,-5350.8071 392.2511,-5273.9102 382.5,-5256 375.7494,-5243.601 364.8204,-5233.3006 353.3687,-5225.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"355.2521,-5222.1654 344.99,-5219.531 351.3694,-5227.9899 355.2521,-5222.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 140680739229424 -->\n<g class=\"node\" id=\"node34\">\n<title>140680739229424</title>\n<polygon fill=\"none\" points=\"45,-5256.5 45,-5292.5 374,-5292.5 374,-5256.5 45,-5256.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"209.5\" y=\"-5270.8\">Encoder-4-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680739571696&#45;&gt;140680739229424 -->\n<g class=\"edge\" id=\"edge39\">\n<title>140680739571696-&gt;140680739229424</title>\n<path d=\"M207.2416,-5329.4551C207.5733,-5321.3828 207.9722,-5311.6764 208.3418,-5302.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"211.8429,-5302.7257 208.7566,-5292.5904 204.8488,-5302.4382 211.8429,-5302.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680739229424&#45;&gt;140680739906784 -->\n<g class=\"edge\" id=\"edge41\">\n<title>140680739229424-&gt;140680739906784</title>\n<path d=\"M233.7247,-5256.4551C246.2105,-5247.1545 261.6086,-5235.6844 275.102,-5225.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"277.2855,-5228.371 283.2143,-5219.5904 273.1039,-5222.7573 277.2855,-5228.371\" stroke=\"#000000\"/>\n</g>\n<!-- 140680742087480 -->\n<g class=\"node\" id=\"node36\">\n<title>140680742087480</title>\n<polygon fill=\"none\" points=\"117.5,-5110.5 117.5,-5146.5 497.5,-5146.5 497.5,-5110.5 117.5,-5110.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"307.5\" y=\"-5124.8\">Encoder-4-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680739906784&#45;&gt;140680742087480 -->\n<g class=\"edge\" id=\"edge42\">\n<title>140680739906784-&gt;140680742087480</title>\n<path d=\"M307.5,-5183.4551C307.5,-5175.3828 307.5,-5165.6764 307.5,-5156.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"311.0001,-5156.5903 307.5,-5146.5904 304.0001,-5156.5904 311.0001,-5156.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680742088376 -->\n<g class=\"node\" id=\"node37\">\n<title>140680742088376</title>\n<polygon fill=\"none\" points=\"105.5,-5037.5 105.5,-5073.5 347.5,-5073.5 347.5,-5037.5 105.5,-5037.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"226.5\" y=\"-5051.8\">Encoder-4-FeedForward: FeedForward</text>\n</g>\n<!-- 140680742087480&#45;&gt;140680742088376 -->\n<g class=\"edge\" id=\"edge43\">\n<title>140680742087480-&gt;140680742088376</title>\n<path d=\"M287.4775,-5110.4551C277.4498,-5101.4177 265.1491,-5090.3319 254.2286,-5080.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"256.3444,-5077.6852 246.5729,-5073.5904 251.6581,-5082.885 256.3444,-5077.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680743683240 -->\n<g class=\"node\" id=\"node39\">\n<title>140680743683240</title>\n<polygon fill=\"none\" points=\"191.5,-4891.5 191.5,-4927.5 411.5,-4927.5 411.5,-4891.5 191.5,-4891.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"301.5\" y=\"-4905.8\">Encoder-4-FeedForward-Add: Add</text>\n</g>\n<!-- 140680742087480&#45;&gt;140680743683240 -->\n<g class=\"edge\" id=\"edge45\">\n<title>140680742087480-&gt;140680743683240</title>\n<path d=\"M329.3471,-5110.291C339.3936,-5100.6134 350.4097,-5087.8624 356.5,-5074 376.2044,-5029.1498 383.6917,-5008.6329 363.5,-4964 358.2852,-4952.4728 349.2939,-4942.4147 339.751,-4934.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"341.6666,-4931.2331 331.676,-4927.7064 337.2868,-4936.6936 341.6666,-4931.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680742583992 -->\n<g class=\"node\" id=\"node38\">\n<title>140680742583992</title>\n<polygon fill=\"none\" points=\"87,-4964.5 87,-5000.5 354,-5000.5 354,-4964.5 87,-4964.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"220.5\" y=\"-4978.8\">Encoder-4-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680742088376&#45;&gt;140680742583992 -->\n<g class=\"edge\" id=\"edge44\">\n<title>140680742088376-&gt;140680742583992</title>\n<path d=\"M225.0169,-5037.4551C224.3534,-5029.3828 223.5556,-5019.6764 222.8163,-5010.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"226.2944,-5010.27 221.9869,-5000.5904 219.3179,-5010.8435 226.2944,-5010.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680742583992&#45;&gt;140680743683240 -->\n<g class=\"edge\" id=\"edge46\">\n<title>140680742583992-&gt;140680743683240</title>\n<path d=\"M240.5225,-4964.4551C250.5502,-4955.4177 262.8509,-4944.3319 273.7714,-4934.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.3419,-4936.885 281.4271,-4927.5904 271.6556,-4931.6852 276.3419,-4936.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680743983256 -->\n<g class=\"node\" id=\"node40\">\n<title>140680743983256</title>\n<polygon fill=\"none\" points=\"142.5,-4818.5 142.5,-4854.5 460.5,-4854.5 460.5,-4818.5 142.5,-4818.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"301.5\" y=\"-4832.8\">Encoder-4-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680743683240&#45;&gt;140680743983256 -->\n<g class=\"edge\" id=\"edge47\">\n<title>140680743683240-&gt;140680743983256</title>\n<path d=\"M301.5,-4891.4551C301.5,-4883.3828 301.5,-4873.6764 301.5,-4864.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"305.0001,-4864.5903 301.5,-4854.5904 298.0001,-4864.5904 305.0001,-4864.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680743982472 -->\n<g class=\"node\" id=\"node41\">\n<title>140680743982472</title>\n<polygon fill=\"none\" points=\"31.5,-4745.5 31.5,-4781.5 373.5,-4781.5 373.5,-4745.5 31.5,-4745.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.5\" y=\"-4759.8\">Encoder-5-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680743983256&#45;&gt;140680743982472 -->\n<g class=\"edge\" id=\"edge48\">\n<title>140680743983256-&gt;140680743982472</title>\n<path d=\"M277.0281,-4818.4551C264.4149,-4809.1545 248.8597,-4797.6844 235.2286,-4787.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"237.1592,-4784.7082 227.0335,-4781.5904 233.0049,-4790.3421 237.1592,-4784.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680745156848 -->\n<g class=\"node\" id=\"node43\">\n<title>140680745156848</title>\n<polygon fill=\"none\" points=\"163,-4599.5 163,-4635.5 444,-4635.5 444,-4599.5 163,-4599.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"303.5\" y=\"-4613.8\">Encoder-5-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680743983256&#45;&gt;140680745156848 -->\n<g class=\"edge\" id=\"edge50\">\n<title>140680743983256-&gt;140680745156848</title>\n<path d=\"M344.0685,-4818.3804C359.0065,-4809.7 374.1359,-4797.7088 382.5,-4782 390.5894,-4766.8071 388.2511,-4689.9102 378.5,-4672 371.7494,-4659.601 360.8204,-4649.3006 349.3687,-4641.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"351.2521,-4638.1654 340.99,-4635.531 347.3694,-4643.9899 351.2521,-4638.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 140680744521680 -->\n<g class=\"node\" id=\"node42\">\n<title>140680744521680</title>\n<polygon fill=\"none\" points=\"41,-4672.5 41,-4708.5 370,-4708.5 370,-4672.5 41,-4672.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"205.5\" y=\"-4686.8\">Encoder-5-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680743982472&#45;&gt;140680744521680 -->\n<g class=\"edge\" id=\"edge49\">\n<title>140680743982472-&gt;140680744521680</title>\n<path d=\"M203.2416,-4745.4551C203.5733,-4737.3828 203.9722,-4727.6764 204.3418,-4718.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"207.8429,-4718.7257 204.7566,-4708.5904 200.8488,-4718.4382 207.8429,-4718.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680744521680&#45;&gt;140680745156848 -->\n<g class=\"edge\" id=\"edge51\">\n<title>140680744521680-&gt;140680745156848</title>\n<path d=\"M229.7247,-4672.4551C242.2105,-4663.1545 257.6086,-4651.6844 271.102,-4641.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"273.2855,-4644.371 279.2143,-4635.5904 269.1039,-4638.7573 273.2855,-4644.371\" stroke=\"#000000\"/>\n</g>\n<!-- 140680821477048 -->\n<g class=\"node\" id=\"node44\">\n<title>140680821477048</title>\n<polygon fill=\"none\" points=\"113.5,-4526.5 113.5,-4562.5 493.5,-4562.5 493.5,-4526.5 113.5,-4526.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"303.5\" y=\"-4540.8\">Encoder-5-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680745156848&#45;&gt;140680821477048 -->\n<g class=\"edge\" id=\"edge52\">\n<title>140680745156848-&gt;140680821477048</title>\n<path d=\"M303.5,-4599.4551C303.5,-4591.3828 303.5,-4581.6764 303.5,-4572.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"307.0001,-4572.5903 303.5,-4562.5904 300.0001,-4572.5904 307.0001,-4572.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680821476376 -->\n<g class=\"node\" id=\"node45\">\n<title>140680821476376</title>\n<polygon fill=\"none\" points=\"101.5,-4453.5 101.5,-4489.5 343.5,-4489.5 343.5,-4453.5 101.5,-4453.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222.5\" y=\"-4467.8\">Encoder-5-FeedForward: FeedForward</text>\n</g>\n<!-- 140680821477048&#45;&gt;140680821476376 -->\n<g class=\"edge\" id=\"edge53\">\n<title>140680821477048-&gt;140680821476376</title>\n<path d=\"M283.4775,-4526.4551C273.4498,-4517.4177 261.1491,-4506.3319 250.2286,-4496.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"252.3444,-4493.6852 242.5729,-4489.5904 247.6581,-4498.885 252.3444,-4493.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680819130320 -->\n<g class=\"node\" id=\"node47\">\n<title>140680819130320</title>\n<polygon fill=\"none\" points=\"187.5,-4307.5 187.5,-4343.5 407.5,-4343.5 407.5,-4307.5 187.5,-4307.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-4321.8\">Encoder-5-FeedForward-Add: Add</text>\n</g>\n<!-- 140680821477048&#45;&gt;140680819130320 -->\n<g class=\"edge\" id=\"edge55\">\n<title>140680821477048-&gt;140680819130320</title>\n<path d=\"M325.3471,-4526.291C335.3936,-4516.6134 346.4097,-4503.8624 352.5,-4490 372.2044,-4445.1498 379.6917,-4424.6329 359.5,-4380 354.2852,-4368.4728 345.2939,-4358.4147 335.751,-4350.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"337.6666,-4347.2331 327.676,-4343.7064 333.2868,-4352.6936 337.6666,-4347.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680821610536 -->\n<g class=\"node\" id=\"node46\">\n<title>140680821610536</title>\n<polygon fill=\"none\" points=\"83,-4380.5 83,-4416.5 350,-4416.5 350,-4380.5 83,-4380.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"216.5\" y=\"-4394.8\">Encoder-5-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680821476376&#45;&gt;140680821610536 -->\n<g class=\"edge\" id=\"edge54\">\n<title>140680821476376-&gt;140680821610536</title>\n<path d=\"M221.0169,-4453.4551C220.3534,-4445.3828 219.5556,-4435.6764 218.8163,-4426.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"222.2944,-4426.27 217.9869,-4416.5904 215.3179,-4426.8435 222.2944,-4426.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680821610536&#45;&gt;140680819130320 -->\n<g class=\"edge\" id=\"edge56\">\n<title>140680821610536-&gt;140680819130320</title>\n<path d=\"M236.5225,-4380.4551C246.5502,-4371.4177 258.8509,-4360.3319 269.7714,-4350.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"272.3419,-4352.885 277.4271,-4343.5904 267.6556,-4347.6852 272.3419,-4352.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680818593352 -->\n<g class=\"node\" id=\"node48\">\n<title>140680818593352</title>\n<polygon fill=\"none\" points=\"138.5,-4234.5 138.5,-4270.5 456.5,-4270.5 456.5,-4234.5 138.5,-4234.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"297.5\" y=\"-4248.8\">Encoder-5-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680819130320&#45;&gt;140680818593352 -->\n<g class=\"edge\" id=\"edge57\">\n<title>140680819130320-&gt;140680818593352</title>\n<path d=\"M297.5,-4307.4551C297.5,-4299.3828 297.5,-4289.6764 297.5,-4280.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"301.0001,-4280.5903 297.5,-4270.5904 294.0001,-4280.5904 301.0001,-4280.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680818592960 -->\n<g class=\"node\" id=\"node49\">\n<title>140680818592960</title>\n<polygon fill=\"none\" points=\"27.5,-4161.5 27.5,-4197.5 369.5,-4197.5 369.5,-4161.5 27.5,-4161.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"198.5\" y=\"-4175.8\">Encoder-6-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680818593352&#45;&gt;140680818592960 -->\n<g class=\"edge\" id=\"edge58\">\n<title>140680818593352-&gt;140680818592960</title>\n<path d=\"M273.0281,-4234.4551C260.4149,-4225.1545 244.8597,-4213.6844 231.2286,-4203.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"233.1592,-4200.7082 223.0335,-4197.5904 229.0049,-4206.3421 233.1592,-4200.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680818838328 -->\n<g class=\"node\" id=\"node51\">\n<title>140680818838328</title>\n<polygon fill=\"none\" points=\"158,-4015.5 158,-4051.5 439,-4051.5 439,-4015.5 158,-4015.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.5\" y=\"-4029.8\">Encoder-6-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680818593352&#45;&gt;140680818838328 -->\n<g class=\"edge\" id=\"edge60\">\n<title>140680818593352-&gt;140680818838328</title>\n<path d=\"M340.0685,-4234.3804C355.0065,-4225.7 370.1359,-4213.7088 378.5,-4198 386.1793,-4183.5774 385.9885,-4107.2923 375.5,-4088 368.7016,-4075.4953 357.6452,-4065.1998 346.0052,-4057.0548\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"347.7688,-4054.0266 337.4788,-4051.5031 343.9492,-4059.8927 347.7688,-4054.0266\" stroke=\"#000000\"/>\n</g>\n<!-- 140680818788000 -->\n<g class=\"node\" id=\"node50\">\n<title>140680818788000</title>\n<polygon fill=\"none\" points=\"38,-4088.5 38,-4124.5 367,-4124.5 367,-4088.5 38,-4088.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.5\" y=\"-4102.8\">Encoder-6-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680818592960&#45;&gt;140680818788000 -->\n<g class=\"edge\" id=\"edge59\">\n<title>140680818592960-&gt;140680818788000</title>\n<path d=\"M199.4888,-4161.4551C199.9311,-4153.3828 200.4629,-4143.6764 200.9558,-4134.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"204.4563,-4134.7669 201.5087,-4124.5904 197.4668,-4134.3839 204.4563,-4134.7669\" stroke=\"#000000\"/>\n</g>\n<!-- 140680818788000&#45;&gt;140680818838328 -->\n<g class=\"edge\" id=\"edge61\">\n<title>140680818788000-&gt;140680818838328</title>\n<path d=\"M226.2303,-4088.4551C238.3459,-4079.2422 253.2608,-4067.9006 266.3886,-4057.918\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"268.8684,-4060.4293 274.7099,-4051.5904 264.6314,-4054.8573 268.8684,-4060.4293\" stroke=\"#000000\"/>\n</g>\n<!-- 140680802778752 -->\n<g class=\"node\" id=\"node52\">\n<title>140680802778752</title>\n<polygon fill=\"none\" points=\"108.5,-3942.5 108.5,-3978.5 488.5,-3978.5 488.5,-3942.5 108.5,-3942.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"298.5\" y=\"-3956.8\">Encoder-6-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680818838328&#45;&gt;140680802778752 -->\n<g class=\"edge\" id=\"edge62\">\n<title>140680818838328-&gt;140680802778752</title>\n<path d=\"M298.5,-4015.4551C298.5,-4007.3828 298.5,-3997.6764 298.5,-3988.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"302.0001,-3988.5903 298.5,-3978.5904 295.0001,-3988.5904 302.0001,-3988.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680802777520 -->\n<g class=\"node\" id=\"node53\">\n<title>140680802777520</title>\n<polygon fill=\"none\" points=\"96.5,-3869.5 96.5,-3905.5 338.5,-3905.5 338.5,-3869.5 96.5,-3869.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"217.5\" y=\"-3883.8\">Encoder-6-FeedForward: FeedForward</text>\n</g>\n<!-- 140680802778752&#45;&gt;140680802777520 -->\n<g class=\"edge\" id=\"edge63\">\n<title>140680802778752-&gt;140680802777520</title>\n<path d=\"M278.4775,-3942.4551C268.4498,-3933.4177 256.1491,-3922.3319 245.2286,-3912.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"247.3444,-3909.6852 237.5729,-3905.5904 242.6581,-3914.885 247.3444,-3909.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680800430848 -->\n<g class=\"node\" id=\"node55\">\n<title>140680800430848</title>\n<polygon fill=\"none\" points=\"182.5,-3723.5 182.5,-3759.5 402.5,-3759.5 402.5,-3723.5 182.5,-3723.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.5\" y=\"-3737.8\">Encoder-6-FeedForward-Add: Add</text>\n</g>\n<!-- 140680802778752&#45;&gt;140680800430848 -->\n<g class=\"edge\" id=\"edge65\">\n<title>140680802778752-&gt;140680800430848</title>\n<path d=\"M320.3471,-3942.291C330.3936,-3932.6134 341.4097,-3919.8624 347.5,-3906 367.2044,-3861.1498 374.6917,-3840.6329 354.5,-3796 349.2852,-3784.4728 340.2939,-3774.4147 330.751,-3766.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"332.6666,-3763.2331 322.676,-3759.7064 328.2868,-3768.6936 332.6666,-3763.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680800359984 -->\n<g class=\"node\" id=\"node54\">\n<title>140680800359984</title>\n<polygon fill=\"none\" points=\"78,-3796.5 78,-3832.5 345,-3832.5 345,-3796.5 78,-3796.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"211.5\" y=\"-3810.8\">Encoder-6-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680802777520&#45;&gt;140680800359984 -->\n<g class=\"edge\" id=\"edge64\">\n<title>140680802777520-&gt;140680800359984</title>\n<path d=\"M216.0169,-3869.4551C215.3534,-3861.3828 214.5556,-3851.6764 213.8163,-3842.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"217.2944,-3842.27 212.9869,-3832.5904 210.3179,-3842.8435 217.2944,-3842.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680800359984&#45;&gt;140680800430848 -->\n<g class=\"edge\" id=\"edge66\">\n<title>140680800359984-&gt;140680800430848</title>\n<path d=\"M231.5225,-3796.4551C241.5502,-3787.4177 253.8509,-3776.3319 264.7714,-3766.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"267.3419,-3768.885 272.4271,-3759.5904 262.6556,-3763.6852 267.3419,-3768.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680800138744 -->\n<g class=\"node\" id=\"node56\">\n<title>140680800138744</title>\n<polygon fill=\"none\" points=\"133.5,-3650.5 133.5,-3686.5 451.5,-3686.5 451.5,-3650.5 133.5,-3650.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"292.5\" y=\"-3664.8\">Encoder-6-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680800430848&#45;&gt;140680800138744 -->\n<g class=\"edge\" id=\"edge67\">\n<title>140680800430848-&gt;140680800138744</title>\n<path d=\"M292.5,-3723.4551C292.5,-3715.3828 292.5,-3705.6764 292.5,-3696.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"296.0001,-3696.5903 292.5,-3686.5904 289.0001,-3696.5904 296.0001,-3696.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680800235416 -->\n<g class=\"node\" id=\"node57\">\n<title>140680800235416</title>\n<polygon fill=\"none\" points=\"22.5,-3577.5 22.5,-3613.5 364.5,-3613.5 364.5,-3577.5 22.5,-3577.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"193.5\" y=\"-3591.8\">Encoder-7-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680800138744&#45;&gt;140680800235416 -->\n<g class=\"edge\" id=\"edge68\">\n<title>140680800138744-&gt;140680800235416</title>\n<path d=\"M268.0281,-3650.4551C255.4149,-3641.1545 239.8597,-3629.6844 226.2286,-3619.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"228.1592,-3616.7082 218.0335,-3613.5904 224.0049,-3622.3421 228.1592,-3616.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680785923600 -->\n<g class=\"node\" id=\"node59\">\n<title>140680785923600</title>\n<polygon fill=\"none\" points=\"153,-3431.5 153,-3467.5 434,-3467.5 434,-3431.5 153,-3431.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"293.5\" y=\"-3445.8\">Encoder-7-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680800138744&#45;&gt;140680785923600 -->\n<g class=\"edge\" id=\"edge70\">\n<title>140680800138744-&gt;140680785923600</title>\n<path d=\"M335.0685,-3650.3804C350.0065,-3641.7 365.1359,-3629.7088 373.5,-3614 381.1793,-3599.5774 380.9885,-3523.2923 370.5,-3504 363.7016,-3491.4953 352.6452,-3481.1998 341.0052,-3473.0548\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"342.7688,-3470.0266 332.4788,-3467.5031 338.9492,-3475.8927 342.7688,-3470.0266\" stroke=\"#000000\"/>\n</g>\n<!-- 140680785888592 -->\n<g class=\"node\" id=\"node58\">\n<title>140680785888592</title>\n<polygon fill=\"none\" points=\"33,-3504.5 33,-3540.5 362,-3540.5 362,-3504.5 33,-3504.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"197.5\" y=\"-3518.8\">Encoder-7-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680800235416&#45;&gt;140680785888592 -->\n<g class=\"edge\" id=\"edge69\">\n<title>140680800235416-&gt;140680785888592</title>\n<path d=\"M194.4888,-3577.4551C194.9311,-3569.3828 195.4629,-3559.6764 195.9558,-3550.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"199.4563,-3550.7669 196.5087,-3540.5904 192.4668,-3550.3839 199.4563,-3550.7669\" stroke=\"#000000\"/>\n</g>\n<!-- 140680785888592&#45;&gt;140680785923600 -->\n<g class=\"edge\" id=\"edge71\">\n<title>140680785888592-&gt;140680785923600</title>\n<path d=\"M221.2303,-3504.4551C233.3459,-3495.2422 248.2608,-3483.9006 261.3886,-3473.918\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"263.8684,-3476.4293 269.7099,-3467.5904 259.6314,-3470.8573 263.8684,-3476.4293\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803479392 -->\n<g class=\"node\" id=\"node60\">\n<title>140680803479392</title>\n<polygon fill=\"none\" points=\"103.5,-3358.5 103.5,-3394.5 483.5,-3394.5 483.5,-3358.5 103.5,-3358.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"293.5\" y=\"-3372.8\">Encoder-7-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680785923600&#45;&gt;140680803479392 -->\n<g class=\"edge\" id=\"edge72\">\n<title>140680785923600-&gt;140680803479392</title>\n<path d=\"M293.5,-3431.4551C293.5,-3423.3828 293.5,-3413.6764 293.5,-3404.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"297.0001,-3404.5903 293.5,-3394.5904 290.0001,-3404.5904 297.0001,-3404.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803565352 -->\n<g class=\"node\" id=\"node61\">\n<title>140680803565352</title>\n<polygon fill=\"none\" points=\"91.5,-3285.5 91.5,-3321.5 333.5,-3321.5 333.5,-3285.5 91.5,-3285.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"212.5\" y=\"-3299.8\">Encoder-7-FeedForward: FeedForward</text>\n</g>\n<!-- 140680803479392&#45;&gt;140680803565352 -->\n<g class=\"edge\" id=\"edge73\">\n<title>140680803479392-&gt;140680803565352</title>\n<path d=\"M273.4775,-3358.4551C263.4498,-3349.4177 251.1491,-3338.3319 240.2286,-3328.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"242.3444,-3325.6852 232.5729,-3321.5904 237.6581,-3330.885 242.3444,-3325.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680818109912 -->\n<g class=\"node\" id=\"node63\">\n<title>140680818109912</title>\n<polygon fill=\"none\" points=\"177.5,-3139.5 177.5,-3175.5 397.5,-3175.5 397.5,-3139.5 177.5,-3139.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"287.5\" y=\"-3153.8\">Encoder-7-FeedForward-Add: Add</text>\n</g>\n<!-- 140680803479392&#45;&gt;140680818109912 -->\n<g class=\"edge\" id=\"edge75\">\n<title>140680803479392-&gt;140680818109912</title>\n<path d=\"M315.3471,-3358.291C325.3936,-3348.6134 336.4097,-3335.8624 342.5,-3322 362.2044,-3277.1498 369.6917,-3256.6329 349.5,-3212 344.2852,-3200.4728 335.2939,-3190.4147 325.751,-3182.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"327.6666,-3179.2331 317.676,-3175.7064 323.2868,-3184.6936 327.6666,-3179.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803628256 -->\n<g class=\"node\" id=\"node62\">\n<title>140680803628256</title>\n<polygon fill=\"none\" points=\"73,-3212.5 73,-3248.5 340,-3248.5 340,-3212.5 73,-3212.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"206.5\" y=\"-3226.8\">Encoder-7-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680803565352&#45;&gt;140680803628256 -->\n<g class=\"edge\" id=\"edge74\">\n<title>140680803565352-&gt;140680803628256</title>\n<path d=\"M211.0169,-3285.4551C210.3534,-3277.3828 209.5556,-3267.6764 208.8163,-3258.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"212.2944,-3258.27 207.9869,-3248.5904 205.3179,-3258.8435 212.2944,-3258.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803628256&#45;&gt;140680818109912 -->\n<g class=\"edge\" id=\"edge76\">\n<title>140680803628256-&gt;140680818109912</title>\n<path d=\"M226.5225,-3212.4551C236.5502,-3203.4177 248.8509,-3192.3319 259.7714,-3182.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"262.3419,-3184.885 267.4271,-3175.5904 257.6556,-3179.6852 262.3419,-3184.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680817569296 -->\n<g class=\"node\" id=\"node64\">\n<title>140680817569296</title>\n<polygon fill=\"none\" points=\"128.5,-3066.5 128.5,-3102.5 446.5,-3102.5 446.5,-3066.5 128.5,-3066.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"287.5\" y=\"-3080.8\">Encoder-7-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680818109912&#45;&gt;140680817569296 -->\n<g class=\"edge\" id=\"edge77\">\n<title>140680818109912-&gt;140680817569296</title>\n<path d=\"M287.5,-3139.4551C287.5,-3131.3828 287.5,-3121.6764 287.5,-3112.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"291.0001,-3112.5903 287.5,-3102.5904 284.0001,-3112.5904 291.0001,-3112.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680817679664 -->\n<g class=\"node\" id=\"node65\">\n<title>140680817679664</title>\n<polygon fill=\"none\" points=\"17.5,-2993.5 17.5,-3029.5 359.5,-3029.5 359.5,-2993.5 17.5,-2993.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"188.5\" y=\"-3007.8\">Encoder-8-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680817569296&#45;&gt;140680817679664 -->\n<g class=\"edge\" id=\"edge78\">\n<title>140680817569296-&gt;140680817679664</title>\n<path d=\"M263.0281,-3066.4551C250.4149,-3057.1545 234.8597,-3045.6844 221.2286,-3035.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"223.1592,-3032.7082 213.0335,-3029.5904 219.0049,-3038.3421 223.1592,-3032.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680817785768 -->\n<g class=\"node\" id=\"node67\">\n<title>140680817785768</title>\n<polygon fill=\"none\" points=\"149,-2847.5 149,-2883.5 430,-2883.5 430,-2847.5 149,-2847.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"289.5\" y=\"-2861.8\">Encoder-8-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680817569296&#45;&gt;140680817785768 -->\n<g class=\"edge\" id=\"edge80\">\n<title>140680817569296-&gt;140680817785768</title>\n<path d=\"M330.0685,-3066.3804C345.0065,-3057.7 360.1359,-3045.7088 368.5,-3030 376.5894,-3014.8071 374.2511,-2937.9102 364.5,-2920 357.7494,-2907.601 346.8204,-2897.3006 335.3687,-2889.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"337.2521,-2886.1654 326.99,-2883.531 333.3694,-2891.9899 337.2521,-2886.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 140680817740880 -->\n<g class=\"node\" id=\"node66\">\n<title>140680817740880</title>\n<polygon fill=\"none\" points=\"27,-2920.5 27,-2956.5 356,-2956.5 356,-2920.5 27,-2920.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"191.5\" y=\"-2934.8\">Encoder-8-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680817679664&#45;&gt;140680817740880 -->\n<g class=\"edge\" id=\"edge79\">\n<title>140680817679664-&gt;140680817740880</title>\n<path d=\"M189.2416,-2993.4551C189.5733,-2985.3828 189.9722,-2975.6764 190.3418,-2966.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"193.8429,-2966.7257 190.7566,-2956.5904 186.8488,-2966.4382 193.8429,-2966.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680817740880&#45;&gt;140680817785768 -->\n<g class=\"edge\" id=\"edge81\">\n<title>140680817740880-&gt;140680817785768</title>\n<path d=\"M215.7247,-2920.4551C228.2105,-2911.1545 243.6086,-2899.6844 257.102,-2889.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"259.2855,-2892.371 265.2143,-2883.5904 255.1039,-2886.7573 259.2855,-2892.371\" stroke=\"#000000\"/>\n</g>\n<!-- 140680798571936 -->\n<g class=\"node\" id=\"node68\">\n<title>140680798571936</title>\n<polygon fill=\"none\" points=\"99.5,-2774.5 99.5,-2810.5 479.5,-2810.5 479.5,-2774.5 99.5,-2774.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"289.5\" y=\"-2788.8\">Encoder-8-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680817785768&#45;&gt;140680798571936 -->\n<g class=\"edge\" id=\"edge82\">\n<title>140680817785768-&gt;140680798571936</title>\n<path d=\"M289.5,-2847.4551C289.5,-2839.3828 289.5,-2829.6764 289.5,-2820.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"293.0001,-2820.5903 289.5,-2810.5904 286.0001,-2820.5904 293.0001,-2820.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680798620920 -->\n<g class=\"node\" id=\"node69\">\n<title>140680798620920</title>\n<polygon fill=\"none\" points=\"87.5,-2701.5 87.5,-2737.5 329.5,-2737.5 329.5,-2701.5 87.5,-2701.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"208.5\" y=\"-2715.8\">Encoder-8-FeedForward: FeedForward</text>\n</g>\n<!-- 140680798571936&#45;&gt;140680798620920 -->\n<g class=\"edge\" id=\"edge83\">\n<title>140680798571936-&gt;140680798620920</title>\n<path d=\"M269.4775,-2774.4551C259.4498,-2765.4177 247.1491,-2754.3319 236.2286,-2744.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"238.3444,-2741.6852 228.5729,-2737.5904 233.6581,-2746.885 238.3444,-2741.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140680796142448 -->\n<g class=\"node\" id=\"node71\">\n<title>140680796142448</title>\n<polygon fill=\"none\" points=\"173.5,-2555.5 173.5,-2591.5 393.5,-2591.5 393.5,-2555.5 173.5,-2555.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"283.5\" y=\"-2569.8\">Encoder-8-FeedForward-Add: Add</text>\n</g>\n<!-- 140680798571936&#45;&gt;140680796142448 -->\n<g class=\"edge\" id=\"edge85\">\n<title>140680798571936-&gt;140680796142448</title>\n<path d=\"M311.3471,-2774.291C321.3936,-2764.6134 332.4097,-2751.8624 338.5,-2738 358.2044,-2693.1498 365.6917,-2672.6329 345.5,-2628 340.2852,-2616.4728 331.2939,-2606.4147 321.751,-2598.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"323.6666,-2595.2331 313.676,-2591.7064 319.2868,-2600.6936 323.6666,-2595.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140680796093464 -->\n<g class=\"node\" id=\"node70\">\n<title>140680796093464</title>\n<polygon fill=\"none\" points=\"69,-2628.5 69,-2664.5 336,-2664.5 336,-2628.5 69,-2628.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"202.5\" y=\"-2642.8\">Encoder-8-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680798620920&#45;&gt;140680796093464 -->\n<g class=\"edge\" id=\"edge84\">\n<title>140680798620920-&gt;140680796093464</title>\n<path d=\"M207.0169,-2701.4551C206.3534,-2693.3828 205.5556,-2683.6764 204.8163,-2674.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"208.2944,-2674.27 203.9869,-2664.5904 201.3179,-2674.8435 208.2944,-2674.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680796093464&#45;&gt;140680796142448 -->\n<g class=\"edge\" id=\"edge86\">\n<title>140680796093464-&gt;140680796142448</title>\n<path d=\"M222.5225,-2628.4551C232.5502,-2619.4177 244.8509,-2608.3319 255.7714,-2598.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"258.3419,-2600.885 263.4271,-2591.5904 253.6556,-2595.6852 258.3419,-2600.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140680794500792 -->\n<g class=\"node\" id=\"node72\">\n<title>140680794500792</title>\n<polygon fill=\"none\" points=\"124.5,-2482.5 124.5,-2518.5 442.5,-2518.5 442.5,-2482.5 124.5,-2482.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"283.5\" y=\"-2496.8\">Encoder-8-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680796142448&#45;&gt;140680794500792 -->\n<g class=\"edge\" id=\"edge87\">\n<title>140680796142448-&gt;140680794500792</title>\n<path d=\"M283.5,-2555.4551C283.5,-2547.3828 283.5,-2537.6764 283.5,-2528.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"287.0001,-2528.5903 283.5,-2518.5904 280.0001,-2528.5904 287.0001,-2528.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680794500008 -->\n<g class=\"node\" id=\"node73\">\n<title>140680794500008</title>\n<polygon fill=\"none\" points=\"13.5,-2409.5 13.5,-2445.5 355.5,-2445.5 355.5,-2409.5 13.5,-2409.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"184.5\" y=\"-2423.8\">Encoder-9-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680794500792&#45;&gt;140680794500008 -->\n<g class=\"edge\" id=\"edge88\">\n<title>140680794500792-&gt;140680794500008</title>\n<path d=\"M259.0281,-2482.4551C246.4149,-2473.1545 230.8597,-2461.6844 217.2286,-2451.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"219.1592,-2448.7082 209.0335,-2445.5904 215.0049,-2454.3421 219.1592,-2448.7082\" stroke=\"#000000\"/>\n</g>\n<!-- 140680791370048 -->\n<g class=\"node\" id=\"node75\">\n<title>140680791370048</title>\n<polygon fill=\"none\" points=\"145,-2263.5 145,-2299.5 426,-2299.5 426,-2263.5 145,-2263.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.5\" y=\"-2277.8\">Encoder-9-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680794500792&#45;&gt;140680791370048 -->\n<g class=\"edge\" id=\"edge90\">\n<title>140680794500792-&gt;140680791370048</title>\n<path d=\"M326.0685,-2482.3804C341.0065,-2473.7 356.1359,-2461.7088 364.5,-2446 372.5894,-2430.8071 370.2511,-2353.9102 360.5,-2336 353.7494,-2323.601 342.8204,-2313.3006 331.3687,-2305.1162\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"333.2521,-2302.1654 322.99,-2299.531 329.3694,-2307.9899 333.2521,-2302.1654\" stroke=\"#000000\"/>\n</g>\n<!-- 140680794664072 -->\n<g class=\"node\" id=\"node74\">\n<title>140680794664072</title>\n<polygon fill=\"none\" points=\"23,-2336.5 23,-2372.5 352,-2372.5 352,-2336.5 23,-2336.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"187.5\" y=\"-2350.8\">Encoder-9-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680794500008&#45;&gt;140680794664072 -->\n<g class=\"edge\" id=\"edge89\">\n<title>140680794500008-&gt;140680794664072</title>\n<path d=\"M185.2416,-2409.4551C185.5733,-2401.3828 185.9722,-2391.6764 186.3418,-2382.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"189.8429,-2382.7257 186.7566,-2372.5904 182.8488,-2382.4382 189.8429,-2382.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680794664072&#45;&gt;140680791370048 -->\n<g class=\"edge\" id=\"edge91\">\n<title>140680794664072-&gt;140680791370048</title>\n<path d=\"M211.7247,-2336.4551C224.2105,-2327.1545 239.6086,-2315.6844 253.102,-2305.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"255.2855,-2308.371 261.2143,-2299.5904 251.1039,-2302.7573 255.2855,-2308.371\" stroke=\"#000000\"/>\n</g>\n<!-- 140681120480840 -->\n<g class=\"node\" id=\"node76\">\n<title>140681120480840</title>\n<polygon fill=\"none\" points=\"95.5,-2190.5 95.5,-2226.5 475.5,-2226.5 475.5,-2190.5 95.5,-2190.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"285.5\" y=\"-2204.8\">Encoder-9-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680791370048&#45;&gt;140681120480840 -->\n<g class=\"edge\" id=\"edge92\">\n<title>140680791370048-&gt;140681120480840</title>\n<path d=\"M285.5,-2263.4551C285.5,-2255.3828 285.5,-2245.6764 285.5,-2236.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"289.0001,-2236.5903 285.5,-2226.5904 282.0001,-2236.5904 289.0001,-2236.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140681120562088 -->\n<g class=\"node\" id=\"node77\">\n<title>140681120562088</title>\n<polygon fill=\"none\" points=\"83.5,-2117.5 83.5,-2153.5 325.5,-2153.5 325.5,-2117.5 83.5,-2117.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"204.5\" y=\"-2131.8\">Encoder-9-FeedForward: FeedForward</text>\n</g>\n<!-- 140681120480840&#45;&gt;140681120562088 -->\n<g class=\"edge\" id=\"edge93\">\n<title>140681120480840-&gt;140681120562088</title>\n<path d=\"M265.4775,-2190.4551C255.4498,-2181.4177 243.1491,-2170.3319 232.2286,-2160.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"234.3444,-2157.6852 224.5729,-2153.5904 229.6581,-2162.885 234.3444,-2157.6852\" stroke=\"#000000\"/>\n</g>\n<!-- 140681121978352 -->\n<g class=\"node\" id=\"node79\">\n<title>140681121978352</title>\n<polygon fill=\"none\" points=\"169.5,-1971.5 169.5,-2007.5 389.5,-2007.5 389.5,-1971.5 169.5,-1971.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"279.5\" y=\"-1985.8\">Encoder-9-FeedForward-Add: Add</text>\n</g>\n<!-- 140681120480840&#45;&gt;140681121978352 -->\n<g class=\"edge\" id=\"edge95\">\n<title>140681120480840-&gt;140681121978352</title>\n<path d=\"M307.3471,-2190.291C317.3936,-2180.6134 328.4097,-2167.8624 334.5,-2154 354.2044,-2109.1498 361.6917,-2088.6329 341.5,-2044 336.2852,-2032.4728 327.2939,-2022.4147 317.751,-2014.1833\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"319.6666,-2011.2331 309.676,-2007.7064 315.2868,-2016.6936 319.6666,-2011.2331\" stroke=\"#000000\"/>\n</g>\n<!-- 140681119501168 -->\n<g class=\"node\" id=\"node78\">\n<title>140681119501168</title>\n<polygon fill=\"none\" points=\"65,-2044.5 65,-2080.5 332,-2080.5 332,-2044.5 65,-2044.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"198.5\" y=\"-2058.8\">Encoder-9-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140681120562088&#45;&gt;140681119501168 -->\n<g class=\"edge\" id=\"edge94\">\n<title>140681120562088-&gt;140681119501168</title>\n<path d=\"M203.0169,-2117.4551C202.3534,-2109.3828 201.5556,-2099.6764 200.8163,-2090.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"204.2944,-2090.27 199.9869,-2080.5904 197.3179,-2090.8435 204.2944,-2090.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140681119501168&#45;&gt;140681121978352 -->\n<g class=\"edge\" id=\"edge96\">\n<title>140681119501168-&gt;140681121978352</title>\n<path d=\"M218.5225,-2044.4551C228.5502,-2035.4177 240.8509,-2024.3319 251.7714,-2014.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"254.3419,-2016.885 259.4271,-2007.5904 249.6556,-2011.6852 254.3419,-2016.885\" stroke=\"#000000\"/>\n</g>\n<!-- 140681121053552 -->\n<g class=\"node\" id=\"node80\">\n<title>140681121053552</title>\n<polygon fill=\"none\" points=\"120.5,-1898.5 120.5,-1934.5 438.5,-1934.5 438.5,-1898.5 120.5,-1898.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"279.5\" y=\"-1912.8\">Encoder-9-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140681121978352&#45;&gt;140681121053552 -->\n<g class=\"edge\" id=\"edge97\">\n<title>140681121978352-&gt;140681121053552</title>\n<path d=\"M279.5,-1971.4551C279.5,-1963.3828 279.5,-1953.6764 279.5,-1944.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"283.0001,-1944.5903 279.5,-1934.5904 276.0001,-1944.5904 283.0001,-1944.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140681121052768 -->\n<g class=\"node\" id=\"node81\">\n<title>140681121052768</title>\n<polygon fill=\"none\" points=\"3.5,-1825.5 3.5,-1861.5 353.5,-1861.5 353.5,-1825.5 3.5,-1825.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"178.5\" y=\"-1839.8\">Encoder-10-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140681121053552&#45;&gt;140681121052768 -->\n<g class=\"edge\" id=\"edge98\">\n<title>140681121053552-&gt;140681121052768</title>\n<path d=\"M254.5337,-1898.4551C241.6658,-1889.1545 225.7962,-1877.6844 211.8898,-1867.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"213.6841,-1864.6116 203.5291,-1861.5904 209.5836,-1870.2849 213.6841,-1864.6116\" stroke=\"#000000\"/>\n</g>\n<!-- 140681120265720 -->\n<g class=\"node\" id=\"node83\">\n<title>140681120265720</title>\n<polygon fill=\"none\" points=\"137,-1679.5 137,-1715.5 426,-1715.5 426,-1679.5 137,-1679.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.5\" y=\"-1693.8\">Encoder-10-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140681121053552&#45;&gt;140681120265720 -->\n<g class=\"edge\" id=\"edge100\">\n<title>140681121053552-&gt;140681120265720</title>\n<path d=\"M323.5299,-1898.3618C338.6906,-1889.7299 353.9854,-1877.7771 362.5,-1862 370.6814,-1846.8405 368.4328,-1769.9762 358.5,-1752 351.6163,-1739.542 340.5325,-1729.2616 328.8935,-1721.116\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"330.6612,-1718.0903 320.3726,-1715.5614 326.8385,-1723.9544 330.6612,-1718.0903\" stroke=\"#000000\"/>\n</g>\n<!-- 140681120811888 -->\n<g class=\"node\" id=\"node82\">\n<title>140681120811888</title>\n<polygon fill=\"none\" points=\"13.5,-1752.5 13.5,-1788.5 349.5,-1788.5 349.5,-1752.5 13.5,-1752.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"181.5\" y=\"-1766.8\">Encoder-10-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140681121052768&#45;&gt;140681120811888 -->\n<g class=\"edge\" id=\"edge99\">\n<title>140681121052768-&gt;140681120811888</title>\n<path d=\"M179.2416,-1825.4551C179.5733,-1817.3828 179.9722,-1807.6764 180.3418,-1798.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"183.8429,-1798.7257 180.7566,-1788.5904 176.8488,-1798.4382 183.8429,-1798.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140681120811888&#45;&gt;140681120265720 -->\n<g class=\"edge\" id=\"edge101\">\n<title>140681120811888-&gt;140681120265720</title>\n<path d=\"M206.2191,-1752.4551C218.9597,-1743.1545 234.6721,-1731.6844 248.4408,-1721.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"250.7055,-1724.3134 256.7187,-1715.5904 246.5781,-1718.6596 250.7055,-1724.3134\" stroke=\"#000000\"/>\n</g>\n<!-- 140680787032944 -->\n<g class=\"node\" id=\"node84\">\n<title>140680787032944</title>\n<polygon fill=\"none\" points=\"88,-1606.5 88,-1642.5 475,-1642.5 475,-1606.5 88,-1606.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"281.5\" y=\"-1620.8\">Encoder-10-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140681120265720&#45;&gt;140680787032944 -->\n<g class=\"edge\" id=\"edge102\">\n<title>140681120265720-&gt;140680787032944</title>\n<path d=\"M281.5,-1679.4551C281.5,-1671.3828 281.5,-1661.6764 281.5,-1652.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"285.0001,-1652.5903 281.5,-1642.5904 278.0001,-1652.5904 285.0001,-1652.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680787033840 -->\n<g class=\"node\" id=\"node85\">\n<title>140680787033840</title>\n<polygon fill=\"none\" points=\"74,-1533.5 74,-1569.5 323,-1569.5 323,-1533.5 74,-1533.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"198.5\" y=\"-1547.8\">Encoder-10-FeedForward: FeedForward</text>\n</g>\n<!-- 140680787032944&#45;&gt;140680787033840 -->\n<g class=\"edge\" id=\"edge103\">\n<title>140680787032944-&gt;140680787033840</title>\n<path d=\"M260.9831,-1606.4551C250.608,-1597.3299 237.8584,-1586.1165 226.5877,-1576.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"228.8889,-1573.5665 219.0685,-1569.5904 224.2659,-1578.8228 228.8889,-1573.5665\" stroke=\"#000000\"/>\n</g>\n<!-- 140680779764960 -->\n<g class=\"node\" id=\"node87\">\n<title>140680779764960</title>\n<polygon fill=\"none\" points=\"162,-1387.5 162,-1423.5 389,-1423.5 389,-1387.5 162,-1387.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"275.5\" y=\"-1401.8\">Encoder-10-FeedForward-Add: Add</text>\n</g>\n<!-- 140680787032944&#45;&gt;140680779764960 -->\n<g class=\"edge\" id=\"edge105\">\n<title>140680787032944-&gt;140680779764960</title>\n<path d=\"M303.7102,-1606.3371C313.9414,-1596.6709 325.1891,-1583.916 331.5,-1570 351.745,-1525.358 360.0543,-1504.5004 339.5,-1460 334.0436,-1448.1867 324.6242,-1437.9911 314.6643,-1429.7193\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"316.678,-1426.8521 306.6262,-1423.5038 312.396,-1432.3897 316.678,-1426.8521\" stroke=\"#000000\"/>\n</g>\n<!-- 140680779620080 -->\n<g class=\"node\" id=\"node86\">\n<title>140680779620080</title>\n<polygon fill=\"none\" points=\"55,-1460.5 55,-1496.5 330,-1496.5 330,-1460.5 55,-1460.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"192.5\" y=\"-1474.8\">Encoder-10-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680787033840&#45;&gt;140680779620080 -->\n<g class=\"edge\" id=\"edge104\">\n<title>140680787033840-&gt;140680779620080</title>\n<path d=\"M197.0169,-1533.4551C196.3534,-1525.3828 195.5556,-1515.6764 194.8163,-1506.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"198.2944,-1506.27 193.9869,-1496.5904 191.3179,-1506.8435 198.2944,-1506.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680779620080&#45;&gt;140680779764960 -->\n<g class=\"edge\" id=\"edge106\">\n<title>140680779620080-&gt;140680779764960</title>\n<path d=\"M213.0169,-1460.4551C223.392,-1451.3299 236.1416,-1440.1165 247.4123,-1430.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"249.7341,-1432.8228 254.9315,-1423.5904 245.1111,-1427.5665 249.7341,-1432.8228\" stroke=\"#000000\"/>\n</g>\n<!-- 140680777357576 -->\n<g class=\"node\" id=\"node88\">\n<title>140680777357576</title>\n<polygon fill=\"none\" points=\"112.5,-1314.5 112.5,-1350.5 438.5,-1350.5 438.5,-1314.5 112.5,-1314.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"275.5\" y=\"-1328.8\">Encoder-10-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680779764960&#45;&gt;140680777357576 -->\n<g class=\"edge\" id=\"edge107\">\n<title>140680779764960-&gt;140680777357576</title>\n<path d=\"M275.5,-1387.4551C275.5,-1379.3828 275.5,-1369.6764 275.5,-1360.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"279.0001,-1360.5903 275.5,-1350.5904 272.0001,-1360.5904 279.0001,-1360.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680777356792 -->\n<g class=\"node\" id=\"node89\">\n<title>140680777356792</title>\n<polygon fill=\"none\" points=\"0,-1241.5 0,-1277.5 349,-1277.5 349,-1241.5 0,-1241.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"174.5\" y=\"-1255.8\">Encoder-11-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680777357576&#45;&gt;140680777356792 -->\n<g class=\"edge\" id=\"edge108\">\n<title>140680777357576-&gt;140680777356792</title>\n<path d=\"M250.5337,-1314.4551C237.6658,-1305.1545 221.7962,-1293.6844 207.8898,-1283.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"209.6841,-1280.6116 199.5291,-1277.5904 205.5836,-1286.2849 209.6841,-1280.6116\" stroke=\"#000000\"/>\n</g>\n<!-- 140680813204200 -->\n<g class=\"node\" id=\"node91\">\n<title>140680813204200</title>\n<polygon fill=\"none\" points=\"133.5,-1095.5 133.5,-1131.5 421.5,-1131.5 421.5,-1095.5 133.5,-1095.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"277.5\" y=\"-1109.8\">Encoder-11-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680777357576&#45;&gt;140680813204200 -->\n<g class=\"edge\" id=\"edge110\">\n<title>140680777357576-&gt;140680813204200</title>\n<path d=\"M319.5299,-1314.3618C334.6906,-1305.7299 349.9854,-1293.7771 358.5,-1278 366.6814,-1262.8405 364.4328,-1185.9762 354.5,-1168 347.6163,-1155.542 336.5325,-1145.2616 324.8935,-1137.116\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"326.6612,-1134.0903 316.3726,-1131.5614 322.8385,-1139.9544 326.6612,-1134.0903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680813112568 -->\n<g class=\"node\" id=\"node90\">\n<title>140680813112568</title>\n<polygon fill=\"none\" points=\"10,-1168.5 10,-1204.5 345,-1204.5 345,-1168.5 10,-1168.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"177.5\" y=\"-1182.8\">Encoder-11-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680777356792&#45;&gt;140680813112568 -->\n<g class=\"edge\" id=\"edge109\">\n<title>140680777356792-&gt;140680813112568</title>\n<path d=\"M175.2416,-1241.4551C175.5733,-1233.3828 175.9722,-1223.6764 176.3418,-1214.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"179.8429,-1214.7257 176.7566,-1204.5904 172.8488,-1214.4382 179.8429,-1214.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680813112568&#45;&gt;140680813204200 -->\n<g class=\"edge\" id=\"edge111\">\n<title>140680813112568-&gt;140680813204200</title>\n<path d=\"M202.2191,-1168.4551C214.9597,-1159.1545 230.6721,-1147.6844 244.4408,-1137.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"246.7055,-1140.3134 252.7187,-1131.5904 242.5781,-1134.6596 246.7055,-1140.3134\" stroke=\"#000000\"/>\n</g>\n<!-- 140680811743552 -->\n<g class=\"node\" id=\"node92\">\n<title>140680811743552</title>\n<polygon fill=\"none\" points=\"84.5,-1022.5 84.5,-1058.5 470.5,-1058.5 470.5,-1022.5 84.5,-1022.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"277.5\" y=\"-1036.8\">Encoder-11-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680813204200&#45;&gt;140680811743552 -->\n<g class=\"edge\" id=\"edge112\">\n<title>140680813204200-&gt;140680811743552</title>\n<path d=\"M277.5,-1095.4551C277.5,-1087.3828 277.5,-1077.6764 277.5,-1068.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"281.0001,-1068.5903 277.5,-1058.5904 274.0001,-1068.5904 281.0001,-1068.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680811744448 -->\n<g class=\"node\" id=\"node93\">\n<title>140680811744448</title>\n<polygon fill=\"none\" points=\"77.5,-949.5 77.5,-985.5 325.5,-985.5 325.5,-949.5 77.5,-949.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"201.5\" y=\"-963.8\">Encoder-11-FeedForward: FeedForward</text>\n</g>\n<!-- 140680811743552&#45;&gt;140680811744448 -->\n<g class=\"edge\" id=\"edge113\">\n<title>140680811743552-&gt;140680811744448</title>\n<path d=\"M258.7135,-1022.4551C249.3961,-1013.5054 237.9873,-1002.547 227.8157,-992.7769\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"229.9704,-989.9935 220.3338,-985.5904 225.1212,-995.0419 229.9704,-989.9935\" stroke=\"#000000\"/>\n</g>\n<!-- 140680808337984 -->\n<g class=\"node\" id=\"node95\">\n<title>140680808337984</title>\n<polygon fill=\"none\" points=\"163,-803.5 163,-839.5 390,-839.5 390,-803.5 163,-803.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"276.5\" y=\"-817.8\">Encoder-11-FeedForward-Add: Add</text>\n</g>\n<!-- 140680811743552&#45;&gt;140680808337984 -->\n<g class=\"edge\" id=\"edge115\">\n<title>140680811743552-&gt;140680808337984</title>\n<path d=\"M303.8634,-1022.2916C315.3206,-1012.8699 327.6647,-1000.3358 334.5,-986 355.572,-941.8049 361.0306,-920.4492 340.5,-876 335.0436,-864.1867 325.6242,-853.9911 315.6643,-845.7193\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"317.678,-842.8521 307.6262,-839.5038 313.396,-848.3897 317.678,-842.8521\" stroke=\"#000000\"/>\n</g>\n<!-- 140680808242368 -->\n<g class=\"node\" id=\"node94\">\n<title>140680808242368</title>\n<polygon fill=\"none\" points=\"57.5,-876.5 57.5,-912.5 331.5,-912.5 331.5,-876.5 57.5,-876.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"194.5\" y=\"-890.8\">Encoder-11-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680811744448&#45;&gt;140680808242368 -->\n<g class=\"edge\" id=\"edge114\">\n<title>140680811744448-&gt;140680808242368</title>\n<path d=\"M199.7697,-949.4551C198.9956,-941.3828 198.0649,-931.6764 197.2024,-922.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"200.6733,-922.2106 196.2347,-912.5904 193.7053,-922.8788 200.6733,-922.2106\" stroke=\"#000000\"/>\n</g>\n<!-- 140680808242368&#45;&gt;140680808337984 -->\n<g class=\"edge\" id=\"edge116\">\n<title>140680808242368-&gt;140680808337984</title>\n<path d=\"M214.7697,-876.4551C224.9212,-867.4177 237.3738,-856.3319 248.4291,-846.4899\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"251.0375,-848.8539 256.1793,-839.5904 246.383,-843.6255 251.0375,-848.8539\" stroke=\"#000000\"/>\n</g>\n<!-- 140680807740864 -->\n<g class=\"node\" id=\"node96\">\n<title>140680807740864</title>\n<polygon fill=\"none\" points=\"114,-730.5 114,-766.5 439,-766.5 439,-730.5 114,-730.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"276.5\" y=\"-744.8\">Encoder-11-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680808337984&#45;&gt;140680807740864 -->\n<g class=\"edge\" id=\"edge117\">\n<title>140680808337984-&gt;140680807740864</title>\n<path d=\"M276.5,-803.4551C276.5,-795.3828 276.5,-785.6764 276.5,-776.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"280.0001,-776.5903 276.5,-766.5904 273.0001,-776.5904 280.0001,-776.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680807742992 -->\n<g class=\"node\" id=\"node97\">\n<title>140680807742992</title>\n<polygon fill=\"none\" points=\".5,-657.5 .5,-693.5 350.5,-693.5 350.5,-657.5 .5,-657.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"175.5\" y=\"-671.8\">Encoder-12-MultiHeadSelfAttention: MultiHeadAttention</text>\n</g>\n<!-- 140680807740864&#45;&gt;140680807742992 -->\n<g class=\"edge\" id=\"edge118\">\n<title>140680807740864-&gt;140680807742992</title>\n<path d=\"M251.5337,-730.4551C238.6658,-721.1545 222.7962,-709.6844 208.8898,-699.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"210.6841,-696.6116 200.5291,-693.5904 206.5836,-702.2849 210.6841,-696.6116\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803741032 -->\n<g class=\"node\" id=\"node99\">\n<title>140680803741032</title>\n<polygon fill=\"none\" points=\"134,-511.5 134,-547.5 423,-547.5 423,-511.5 134,-511.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"278.5\" y=\"-525.8\">Encoder-12-MultiHeadSelfAttention-Add: Add</text>\n</g>\n<!-- 140680807740864&#45;&gt;140680803741032 -->\n<g class=\"edge\" id=\"edge120\">\n<title>140680807740864-&gt;140680803741032</title>\n<path d=\"M320.5299,-730.3618C335.6906,-721.7299 350.9854,-709.7771 359.5,-694 367.6814,-678.8405 365.4328,-601.9762 355.5,-584 348.6163,-571.542 337.5325,-561.2616 325.8935,-553.116\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"327.6612,-550.0903 317.3726,-547.5614 323.8385,-555.9544 327.6612,-550.0903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803656816 -->\n<g class=\"node\" id=\"node98\">\n<title>140680803656816</title>\n<polygon fill=\"none\" points=\"10.5,-584.5 10.5,-620.5 346.5,-620.5 346.5,-584.5 10.5,-584.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"178.5\" y=\"-598.8\">Encoder-12-MultiHeadSelfAttention-Dropout: Dropout</text>\n</g>\n<!-- 140680807742992&#45;&gt;140680803656816 -->\n<g class=\"edge\" id=\"edge119\">\n<title>140680807742992-&gt;140680803656816</title>\n<path d=\"M176.2416,-657.4551C176.5733,-649.3828 176.9722,-639.6764 177.3418,-630.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"180.8429,-630.7257 177.7566,-620.5904 173.8488,-630.4382 180.8429,-630.7257\" stroke=\"#000000\"/>\n</g>\n<!-- 140680803656816&#45;&gt;140680803741032 -->\n<g class=\"edge\" id=\"edge121\">\n<title>140680803656816-&gt;140680803741032</title>\n<path d=\"M203.2191,-584.4551C215.9597,-575.1545 231.6721,-563.6844 245.4408,-553.6332\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"247.7055,-556.3134 253.7187,-547.5904 243.5781,-550.6596 247.7055,-556.3134\" stroke=\"#000000\"/>\n</g>\n<!-- 140680798326512 -->\n<g class=\"node\" id=\"node100\">\n<title>140680798326512</title>\n<polygon fill=\"none\" points=\"85,-438.5 85,-474.5 472,-474.5 472,-438.5 85,-438.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"278.5\" y=\"-452.8\">Encoder-12-MultiHeadSelfAttention-Norm: LayerNormalization</text>\n</g>\n<!-- 140680803741032&#45;&gt;140680798326512 -->\n<g class=\"edge\" id=\"edge122\">\n<title>140680803741032-&gt;140680798326512</title>\n<path d=\"M278.5,-511.4551C278.5,-503.3828 278.5,-493.6764 278.5,-484.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"282.0001,-484.5903 278.5,-474.5904 275.0001,-484.5904 282.0001,-484.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680798325280 -->\n<g class=\"node\" id=\"node101\">\n<title>140680798325280</title>\n<polygon fill=\"none\" points=\"71,-365.5 71,-401.5 320,-401.5 320,-365.5 71,-365.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"195.5\" y=\"-379.8\">Encoder-12-FeedForward: FeedForward</text>\n</g>\n<!-- 140680798326512&#45;&gt;140680798325280 -->\n<g class=\"edge\" id=\"edge123\">\n<title>140680798326512-&gt;140680798325280</title>\n<path d=\"M257.9831,-438.4551C247.608,-429.3299 234.8584,-418.1165 223.5877,-408.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"225.8889,-405.5665 216.0685,-401.5904 221.2659,-410.8228 225.8889,-405.5665\" stroke=\"#000000\"/>\n</g>\n<!-- 140680795422056 -->\n<g class=\"node\" id=\"node103\">\n<title>140680795422056</title>\n<polygon fill=\"none\" points=\"159,-219.5 159,-255.5 386,-255.5 386,-219.5 159,-219.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-233.8\">Encoder-12-FeedForward-Add: Add</text>\n</g>\n<!-- 140680798326512&#45;&gt;140680795422056 -->\n<g class=\"edge\" id=\"edge125\">\n<title>140680798326512-&gt;140680795422056</title>\n<path d=\"M300.7102,-438.3371C310.9414,-428.6709 322.1891,-415.916 328.5,-402 348.745,-357.358 357.0543,-336.5004 336.5,-292 331.0436,-280.1867 321.6242,-269.9911 311.6643,-261.7193\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"313.678,-258.8521 303.6262,-255.5038 309.396,-264.3897 313.678,-258.8521\" stroke=\"#000000\"/>\n</g>\n<!-- 140680795350688 -->\n<g class=\"node\" id=\"node102\">\n<title>140680795350688</title>\n<polygon fill=\"none\" points=\"52,-292.5 52,-328.5 327,-328.5 327,-292.5 52,-292.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"189.5\" y=\"-306.8\">Encoder-12-FeedForward-Dropout: Dropout</text>\n</g>\n<!-- 140680798325280&#45;&gt;140680795350688 -->\n<g class=\"edge\" id=\"edge124\">\n<title>140680798325280-&gt;140680795350688</title>\n<path d=\"M194.0169,-365.4551C193.3534,-357.3828 192.5556,-347.6764 191.8163,-338.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"195.2944,-338.27 190.9869,-328.5904 188.3179,-338.8435 195.2944,-338.27\" stroke=\"#000000\"/>\n</g>\n<!-- 140680795350688&#45;&gt;140680795422056 -->\n<g class=\"edge\" id=\"edge126\">\n<title>140680795350688-&gt;140680795422056</title>\n<path d=\"M210.0169,-292.4551C220.392,-283.3299 233.1416,-272.1165 244.4123,-262.2036\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"246.7341,-264.8228 251.9315,-255.5904 242.1111,-259.5665 246.7341,-264.8228\" stroke=\"#000000\"/>\n</g>\n<!-- 140680792249848 -->\n<g class=\"node\" id=\"node104\">\n<title>140680792249848</title>\n<polygon fill=\"none\" points=\"109.5,-146.5 109.5,-182.5 435.5,-182.5 435.5,-146.5 109.5,-146.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-160.8\">Encoder-12-FeedForward-Norm: LayerNormalization</text>\n</g>\n<!-- 140680795422056&#45;&gt;140680792249848 -->\n<g class=\"edge\" id=\"edge127\">\n<title>140680795422056-&gt;140680792249848</title>\n<path d=\"M272.5,-219.4551C272.5,-211.3828 272.5,-201.6764 272.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.0001,-192.5903 272.5,-182.5904 269.0001,-192.5904 276.0001,-192.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680812382360 -->\n<g class=\"node\" id=\"node105\">\n<title>140680812382360</title>\n<polygon fill=\"none\" points=\"175.5,-73.5 175.5,-109.5 369.5,-109.5 369.5,-73.5 175.5,-73.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-87.8\">non_masking_3: NonMasking</text>\n</g>\n<!-- 140680792249848&#45;&gt;140680812382360 -->\n<g class=\"edge\" id=\"edge128\">\n<title>140680792249848-&gt;140680812382360</title>\n<path d=\"M272.5,-146.4551C272.5,-138.3828 272.5,-128.6764 272.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.0001,-119.5903 272.5,-109.5904 269.0001,-119.5904 276.0001,-119.5903\" stroke=\"#000000\"/>\n</g>\n<!-- 140680812382472 -->\n<g class=\"node\" id=\"node106\">\n<title>140680812382472</title>\n<polygon fill=\"none\" points=\"125,-.5 125,-36.5 420,-36.5 420,-.5 125,-.5\" stroke=\"#000000\"/>\n<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.5\" y=\"-14.8\">start__end__prediction_3: Start_End_Prediction</text>\n</g>\n<!-- 140680812382360&#45;&gt;140680812382472 -->\n<g class=\"edge\" id=\"edge129\">\n<title>140680812382360-&gt;140680812382472</title>\n<path d=\"M272.5,-73.4551C272.5,-65.3828 272.5,-55.6764 272.5,-46.6817\" fill=\"none\" stroke=\"#000000\"/>\n<polygon fill=\"#000000\" points=\"276.0001,-46.5903 272.5,-36.5904 269.0001,-46.5904 276.0001,-46.5903\" stroke=\"#000000\"/>\n</g>\n</g>\n</svg>"
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gLLayijHNEjz",
        "colab_type": "text"
      },
      "source": [
        "훈련을 시작합니다.  \n",
        "1epoch을 훈련해 보고, 결과를 확인하고 훈련을 다시 시작할 예정입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1YZVDl7jOAnP",
        "colab_type": "code",
        "outputId": "972dd028-aa7d-424b-b0d6-32db86d06704",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sat Feb 29 08:23:19 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 440.48.02    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   47C    P0    37W / 250W |   1415MiB / 16280MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-7On1hY3I7o1",
        "colab_type": "code",
        "outputId": "e328f3e1-910e-4ea5-de8d-76f3d0fdb0d7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "sess = K.get_session()\n",
        "uninitialized_variables = set([i.decode('ascii') for i in sess.run(tf.report_uninitialized_variables())])\n",
        "init = tf.variables_initializer([v for v in tf.global_variables() if v.name.split(':')[0] in uninitialized_variables])\n",
        "sess.run(init)\n",
        "\n",
        "bert_model = get_bert_finetuning_model(model)\n",
        "bert_model.summary()\n",
        "# 훈련 시작\n",
        "history = bert_model.fit(train_x, train_y, batch_size=12, shuffle=True, verbose=1, epochs=2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_5\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "Input-Token (InputLayer)        (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Input-Segment (InputLayer)      (None, 384)          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token (TokenEmbedding [(None, 384, 768), ( 23440896    Input-Token[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Segment (Embedding)   (None, 384, 768)     1536        Input-Segment[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Token-Segment (Add)   (None, 384, 768)     0           Embedding-Token[0][0]            \n",
            "                                                                 Embedding-Segment[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Position (PositionEmb (None, 384, 768)     294912      Embedding-Token-Segment[0][0]    \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Dropout (Dropout)     (None, 384, 768)     0           Embedding-Position[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "Embedding-Norm (LayerNormalizat (None, 384, 768)     1536        Embedding-Dropout[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     2362368     Embedding-Norm[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     0           Embedding-Norm[0][0]             \n",
            "                                                                 Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-1-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-1-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-1-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-1-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-1-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-1-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-1-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-1-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-2-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-2-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-2-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-2-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-2-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-2-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-2-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-2-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-3-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-3-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-3-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-3-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-3-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-3-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-3-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-3-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-4-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-4-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-4-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-4-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-4-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-4-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-4-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-4-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-5-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-5-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-5-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-5-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-5-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-5-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-5-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-5-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-6-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-6-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-6-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-6-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-6-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-6-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-6-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-6-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-7-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-7-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-7-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-7-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-7-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-7-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-7-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-7-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-8-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-8-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-8-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-8-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-8-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-8-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     2362368     Encoder-8-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention[\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     0           Encoder-8-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-MultiHeadSelfAttentio (None, 384, 768)     1536        Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward (FeedForw (None, 384, 768)     4722432     Encoder-9-MultiHeadSelfAttention-\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Dropout ( (None, 384, 768)     0           Encoder-9-FeedForward[0][0]      \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Add (Add) (None, 384, 768)     0           Encoder-9-MultiHeadSelfAttention-\n",
            "                                                                 Encoder-9-FeedForward-Dropout[0][\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-9-FeedForward-Norm (Lay (None, 384, 768)     1536        Encoder-9-FeedForward-Add[0][0]  \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-9-FeedForward-Norm[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-9-FeedForward-Norm[0][0] \n",
            "                                                                 Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-10-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Dropout  (None, 384, 768)     0           Encoder-10-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Add (Add (None, 384, 768)     0           Encoder-10-MultiHeadSelfAttention\n",
            "                                                                 Encoder-10-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-10-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-10-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-10-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-10-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-11-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Dropout  (None, 384, 768)     0           Encoder-11-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Add (Add (None, 384, 768)     0           Encoder-11-MultiHeadSelfAttention\n",
            "                                                                 Encoder-11-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-11-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-11-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     2362368     Encoder-11-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     0           Encoder-11-FeedForward-Norm[0][0]\n",
            "                                                                 Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-MultiHeadSelfAttenti (None, 384, 768)     1536        Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward (FeedFor (None, 384, 768)     4722432     Encoder-12-MultiHeadSelfAttention\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Dropout  (None, 384, 768)     0           Encoder-12-FeedForward[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Add (Add (None, 384, 768)     0           Encoder-12-MultiHeadSelfAttention\n",
            "                                                                 Encoder-12-FeedForward-Dropout[0]\n",
            "__________________________________________________________________________________________________\n",
            "Encoder-12-FeedForward-Norm (La (None, 384, 768)     1536        Encoder-12-FeedForward-Add[0][0] \n",
            "__________________________________________________________________________________________________\n",
            "non_masking_3 (NonMasking)      (None, 384, 768)     0           Encoder-12-FeedForward-Norm[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "start__end__prediction_3 (Start [(None, 384), (None, 1536        non_masking_3[0][0]              \n",
            "==================================================================================================\n",
            "Total params: 108,794,880\n",
            "Trainable params: 108,794,880\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "Epoch 1/2\n",
            "87353/87353 [==============================] - 4692s 54ms/step - loss: 2.7465 - start__end__prediction_3_loss: 1.3255 - start__end__prediction_3_acc: 0.6047 - start__end__prediction_3_acc_1: 0.6463\n",
            "Epoch 2/2\n",
            "87353/87353 [==============================] - 4685s 54ms/step - loss: 1.5561 - start__end__prediction_3_loss: 0.7341 - start__end__prediction_3_acc: 0.7402 - start__end__prediction_3_acc_1: 0.7797\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kGMBuKa1y83w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.metrics import classification_report\n",
        "preds = bert_model.predict(train_x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Eogv_Nlv4gXU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "start_indexes = np.argmax(preds[0], axis=-1)\n",
        "end_indexes = np.argmax(preds[1], axis=-1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tO7BAAjZz-rt",
        "colab_type": "code",
        "outputId": "c83d5264-3095-4897-e5ea-9ab16eb4acbf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# start_index의 f1_score 85%\n",
        "print(classification_report(train_y[0], start_indexes))\n",
        "\n",
        "# end_index의 f1_score 88%\n",
        "print(classification_report(train_y[1], end_indexes))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           4       1.00      1.00      1.00         1\n",
            "           6       1.00      0.75      0.86         4\n",
            "           7       0.67      0.82      0.73        22\n",
            "           8       0.78      0.83      0.81        65\n",
            "           9       0.82      0.87      0.84       173\n",
            "          10       0.85      0.89      0.87       328\n",
            "          11       0.86      0.86      0.86       464\n",
            "          12       0.84      0.86      0.85       607\n",
            "          13       0.85      0.86      0.86       748\n",
            "          14       0.88      0.87      0.87       849\n",
            "          15       0.87      0.87      0.87       907\n",
            "          16       0.89      0.88      0.88       957\n",
            "          17       0.88      0.89      0.88       963\n",
            "          18       0.89      0.86      0.87       992\n",
            "          19       0.86      0.86      0.86      1001\n",
            "          20       0.87      0.89      0.88       972\n",
            "          21       0.86      0.87      0.87       988\n",
            "          22       0.86      0.88      0.87       986\n",
            "          23       0.85      0.87      0.86       938\n",
            "          24       0.87      0.87      0.87       931\n",
            "          25       0.87      0.87      0.87       959\n",
            "          26       0.87      0.86      0.86       911\n",
            "          27       0.85      0.84      0.85       909\n",
            "          28       0.86      0.87      0.86       894\n",
            "          29       0.87      0.86      0.86       901\n",
            "          30       0.85      0.85      0.85       869\n",
            "          31       0.85      0.83      0.84       822\n",
            "          32       0.89      0.85      0.87       871\n",
            "          33       0.85      0.85      0.85       820\n",
            "          34       0.87      0.87      0.87       798\n",
            "          35       0.87      0.86      0.86       819\n",
            "          36       0.87      0.86      0.87       826\n",
            "          37       0.86      0.86      0.86       810\n",
            "          38       0.86      0.83      0.85       776\n",
            "          39       0.83      0.84      0.84       787\n",
            "          40       0.87      0.83      0.85       794\n",
            "          41       0.84      0.84      0.84       729\n",
            "          42       0.86      0.82      0.84       749\n",
            "          43       0.85      0.84      0.84       776\n",
            "          44       0.84      0.84      0.84       736\n",
            "          45       0.86      0.84      0.85       760\n",
            "          46       0.84      0.83      0.84       772\n",
            "          47       0.86      0.86      0.86       735\n",
            "          48       0.85      0.87      0.86       752\n",
            "          49       0.84      0.88      0.86       720\n",
            "          50       0.86      0.87      0.87       719\n",
            "          51       0.83      0.86      0.84       674\n",
            "          52       0.85      0.85      0.85       720\n",
            "          53       0.86      0.86      0.86       672\n",
            "          54       0.86      0.86      0.86       701\n",
            "          55       0.85      0.86      0.86       667\n",
            "          56       0.84      0.83      0.84       690\n",
            "          57       0.83      0.82      0.83       630\n",
            "          58       0.86      0.86      0.86       657\n",
            "          59       0.84      0.85      0.85       701\n",
            "          60       0.85      0.82      0.84       673\n",
            "          61       0.84      0.85      0.85       645\n",
            "          62       0.85      0.84      0.85       658\n",
            "          63       0.86      0.85      0.86       615\n",
            "          64       0.84      0.83      0.84       596\n",
            "          65       0.85      0.85      0.85       658\n",
            "          66       0.86      0.83      0.84       617\n",
            "          67       0.87      0.82      0.84       586\n",
            "          68       0.84      0.85      0.85       576\n",
            "          69       0.87      0.86      0.86       604\n",
            "          70       0.85      0.84      0.84       584\n",
            "          71       0.85      0.83      0.84       602\n",
            "          72       0.85      0.84      0.85       604\n",
            "          73       0.87      0.84      0.85       605\n",
            "          74       0.86      0.88      0.87       569\n",
            "          75       0.83      0.84      0.84       589\n",
            "          76       0.85      0.85      0.85       544\n",
            "          77       0.81      0.85      0.83       521\n",
            "          78       0.85      0.86      0.85       557\n",
            "          79       0.84      0.88      0.86       548\n",
            "          80       0.84      0.87      0.86       542\n",
            "          81       0.87      0.84      0.85       509\n",
            "          82       0.84      0.85      0.84       527\n",
            "          83       0.89      0.84      0.86       571\n",
            "          84       0.83      0.86      0.84       507\n",
            "          85       0.82      0.87      0.85       490\n",
            "          86       0.85      0.84      0.84       514\n",
            "          87       0.86      0.86      0.86       515\n",
            "          88       0.85      0.83      0.84       486\n",
            "          89       0.86      0.88      0.87       474\n",
            "          90       0.84      0.86      0.85       503\n",
            "          91       0.87      0.84      0.85       461\n",
            "          92       0.87      0.87      0.87       498\n",
            "          93       0.84      0.84      0.84       471\n",
            "          94       0.86      0.85      0.85       484\n",
            "          95       0.87      0.87      0.87       485\n",
            "          96       0.85      0.86      0.85       429\n",
            "          97       0.84      0.84      0.84       465\n",
            "          98       0.82      0.83      0.83       455\n",
            "          99       0.85      0.84      0.84       456\n",
            "         100       0.86      0.84      0.85       490\n",
            "         101       0.84      0.84      0.84       451\n",
            "         102       0.83      0.84      0.84       485\n",
            "         103       0.84      0.87      0.85       453\n",
            "         104       0.83      0.84      0.83       456\n",
            "         105       0.86      0.87      0.87       447\n",
            "         106       0.86      0.86      0.86       433\n",
            "         107       0.85      0.83      0.84       432\n",
            "         108       0.81      0.84      0.83       407\n",
            "         109       0.85      0.85      0.85       433\n",
            "         110       0.83      0.84      0.83       422\n",
            "         111       0.82      0.86      0.84       401\n",
            "         112       0.85      0.85      0.85       399\n",
            "         113       0.83      0.86      0.84       417\n",
            "         114       0.84      0.83      0.83       387\n",
            "         115       0.81      0.82      0.82       370\n",
            "         116       0.84      0.84      0.84       388\n",
            "         117       0.82      0.85      0.84       372\n",
            "         118       0.82      0.83      0.83       413\n",
            "         119       0.84      0.85      0.84       370\n",
            "         120       0.87      0.85      0.86       382\n",
            "         121       0.85      0.84      0.84       356\n",
            "         122       0.83      0.80      0.82       341\n",
            "         123       0.83      0.84      0.84       350\n",
            "         124       0.82      0.82      0.82       364\n",
            "         125       0.83      0.87      0.85       298\n",
            "         126       0.83      0.82      0.83       344\n",
            "         127       0.84      0.86      0.85       316\n",
            "         128       0.84      0.85      0.85       320\n",
            "         129       0.83      0.82      0.82       300\n",
            "         130       0.83      0.81      0.82       284\n",
            "         131       0.85      0.82      0.84       311\n",
            "         132       0.80      0.84      0.82       296\n",
            "         133       0.84      0.87      0.85       298\n",
            "         134       0.81      0.81      0.81       299\n",
            "         135       0.84      0.83      0.83       281\n",
            "         136       0.85      0.88      0.87       303\n",
            "         137       0.83      0.83      0.83       279\n",
            "         138       0.86      0.85      0.86       244\n",
            "         139       0.87      0.86      0.86       256\n",
            "         140       0.85      0.88      0.86       246\n",
            "         141       0.85      0.85      0.85       236\n",
            "         142       0.86      0.84      0.85       250\n",
            "         143       0.85      0.87      0.86       221\n",
            "         144       0.84      0.85      0.85       205\n",
            "         145       0.87      0.83      0.85       210\n",
            "         146       0.86      0.78      0.82       180\n",
            "         147       0.86      0.82      0.84       209\n",
            "         148       0.83      0.85      0.84       195\n",
            "         149       0.82      0.88      0.85       185\n",
            "         150       0.86      0.82      0.83       217\n",
            "         151       0.82      0.84      0.83       200\n",
            "         152       0.82      0.82      0.82       176\n",
            "         153       0.84      0.84      0.84       196\n",
            "         154       0.83      0.82      0.83       182\n",
            "         155       0.84      0.87      0.85       166\n",
            "         156       0.87      0.84      0.86       165\n",
            "         157       0.85      0.86      0.85       173\n",
            "         158       0.90      0.80      0.84       142\n",
            "         159       0.85      0.83      0.84       153\n",
            "         160       0.88      0.82      0.85       157\n",
            "         161       0.84      0.85      0.85       143\n",
            "         162       0.82      0.85      0.83       148\n",
            "         163       0.89      0.84      0.87       161\n",
            "         164       0.87      0.89      0.88       141\n",
            "         165       0.79      0.80      0.79       135\n",
            "         166       0.78      0.79      0.79       114\n",
            "         167       0.87      0.85      0.86       156\n",
            "         168       0.84      0.84      0.84       146\n",
            "         169       0.82      0.81      0.81       122\n",
            "         170       0.88      0.83      0.86       133\n",
            "         171       0.80      0.86      0.83       105\n",
            "         172       0.85      0.81      0.83       116\n",
            "         173       0.85      0.86      0.86       110\n",
            "         174       0.87      0.85      0.86       101\n",
            "         175       0.87      0.87      0.87       112\n",
            "         176       0.83      0.83      0.83        98\n",
            "         177       0.82      0.85      0.83       110\n",
            "         178       0.79      0.85      0.82       102\n",
            "         179       0.82      0.85      0.83        92\n",
            "         180       0.81      0.77      0.79        92\n",
            "         181       0.86      0.88      0.87        95\n",
            "         182       0.87      0.90      0.88       101\n",
            "         183       0.83      0.85      0.84        91\n",
            "         184       0.86      0.88      0.87       100\n",
            "         185       0.86      0.83      0.84       105\n",
            "         186       0.81      0.91      0.86        78\n",
            "         187       0.85      0.77      0.81        91\n",
            "         188       0.86      0.92      0.89        93\n",
            "         189       0.85      0.80      0.82        76\n",
            "         190       0.82      0.85      0.84        78\n",
            "         191       0.79      0.80      0.79        89\n",
            "         192       0.84      0.91      0.87        86\n",
            "         193       0.92      0.87      0.89        75\n",
            "         194       0.81      0.82      0.82        79\n",
            "         195       0.81      0.81      0.81        81\n",
            "         196       0.79      0.82      0.81        51\n",
            "         197       0.83      0.82      0.82        83\n",
            "         198       0.81      0.86      0.84        65\n",
            "         199       0.80      0.90      0.85        68\n",
            "         200       0.83      0.82      0.83        61\n",
            "         201       0.75      0.79      0.77        48\n",
            "         202       0.81      0.77      0.79        62\n",
            "         203       0.87      0.82      0.84        66\n",
            "         204       0.80      0.84      0.82        57\n",
            "         205       0.79      0.87      0.83        62\n",
            "         206       0.77      0.82      0.80        50\n",
            "         207       0.81      0.79      0.80        43\n",
            "         208       0.86      0.87      0.86        55\n",
            "         209       0.81      0.82      0.81        61\n",
            "         210       0.84      0.84      0.84        57\n",
            "         211       0.92      0.84      0.88        56\n",
            "         212       0.81      0.79      0.80        43\n",
            "         213       0.77      0.87      0.82        46\n",
            "         214       0.89      0.93      0.91        44\n",
            "         215       0.89      0.86      0.87        36\n",
            "         216       0.72      0.90      0.80        31\n",
            "         217       0.89      0.85      0.87        39\n",
            "         218       0.95      0.93      0.94        40\n",
            "         219       0.84      0.80      0.82        51\n",
            "         220       0.82      0.84      0.83        38\n",
            "         221       0.79      0.79      0.79        33\n",
            "         222       0.88      0.88      0.88        42\n",
            "         223       0.79      0.76      0.77        45\n",
            "         224       0.84      0.77      0.81        35\n",
            "         225       0.92      0.83      0.88        42\n",
            "         226       0.88      0.88      0.88        34\n",
            "         227       0.83      0.89      0.86        28\n",
            "         228       0.86      0.82      0.84        22\n",
            "         229       0.88      0.94      0.91        32\n",
            "         230       0.97      0.80      0.88        41\n",
            "         231       0.81      0.86      0.83        35\n",
            "         232       0.87      0.82      0.84        33\n",
            "         233       0.86      0.92      0.89        26\n",
            "         234       0.88      0.83      0.85        42\n",
            "         235       0.89      0.94      0.91        33\n",
            "         236       0.86      0.86      0.86        37\n",
            "         237       0.81      0.85      0.83        20\n",
            "         238       0.92      0.77      0.84        30\n",
            "         239       0.90      0.87      0.88        30\n",
            "         240       0.89      0.96      0.92        25\n",
            "         241       0.81      0.92      0.86        24\n",
            "         242       0.89      0.69      0.78        36\n",
            "         243       0.80      0.82      0.81        40\n",
            "         244       0.85      0.85      0.85        26\n",
            "         245       0.91      0.88      0.89        33\n",
            "         246       0.83      0.96      0.89        26\n",
            "         247       0.85      1.00      0.92        28\n",
            "         248       0.87      0.95      0.91        21\n",
            "         249       0.87      0.81      0.84        16\n",
            "         250       0.77      0.83      0.80        12\n",
            "         251       0.86      0.92      0.89        13\n",
            "         252       1.00      0.89      0.94         9\n",
            "         253       1.00      1.00      1.00        17\n",
            "         254       0.86      1.00      0.92        18\n",
            "         255       1.00      0.64      0.78        22\n",
            "         256       0.73      0.85      0.79        13\n",
            "         257       0.91      0.81      0.86        26\n",
            "         258       0.82      0.90      0.86        10\n",
            "         259       0.75      0.86      0.80        14\n",
            "         260       0.88      0.82      0.85        17\n",
            "         261       0.88      0.93      0.90        15\n",
            "         262       0.92      0.85      0.88        13\n",
            "         263       0.83      1.00      0.91        10\n",
            "         264       0.92      0.85      0.88        13\n",
            "         265       0.93      0.76      0.84        17\n",
            "         266       0.87      0.93      0.90        14\n",
            "         267       0.89      0.80      0.84        10\n",
            "         268       1.00      0.60      0.75         5\n",
            "         269       0.91      0.83      0.87        12\n",
            "         270       0.72      1.00      0.84        13\n",
            "         271       0.79      0.73      0.76        15\n",
            "         272       0.88      0.74      0.80        19\n",
            "         273       0.85      0.73      0.79        15\n",
            "         274       0.76      0.93      0.84        14\n",
            "         275       1.00      0.50      0.67         8\n",
            "         276       0.79      1.00      0.88        11\n",
            "         277       1.00      0.64      0.78        11\n",
            "         278       1.00      0.89      0.94         9\n",
            "         279       0.55      0.86      0.67         7\n",
            "         280       0.50      0.71      0.59         7\n",
            "         281       0.82      0.82      0.82        11\n",
            "         282       0.75      0.43      0.55         7\n",
            "         283       0.92      0.92      0.92        12\n",
            "         284       0.92      0.86      0.89        14\n",
            "         285       0.88      0.70      0.78        10\n",
            "         286       0.88      0.88      0.88         8\n",
            "         287       0.78      0.88      0.82         8\n",
            "         288       0.77      0.83      0.80        12\n",
            "         289       0.70      0.70      0.70        10\n",
            "         290       0.75      0.90      0.82        10\n",
            "         291       1.00      1.00      1.00         1\n",
            "         292       1.00      0.83      0.91         6\n",
            "         293       0.86      1.00      0.92         6\n",
            "         294       1.00      0.80      0.89         5\n",
            "         295       0.75      1.00      0.86         6\n",
            "         296       0.88      1.00      0.93         7\n",
            "         297       1.00      1.00      1.00         6\n",
            "         298       1.00      0.71      0.83         7\n",
            "         299       0.83      0.83      0.83         6\n",
            "         300       0.50      0.50      0.50         2\n",
            "         301       1.00      0.82      0.90        11\n",
            "         302       1.00      0.67      0.80         9\n",
            "         303       0.89      0.89      0.89         9\n",
            "         304       0.25      0.50      0.33         2\n",
            "         305       1.00      0.57      0.73         7\n",
            "         306       0.71      0.83      0.77         6\n",
            "         307       1.00      0.67      0.80         3\n",
            "         308       0.50      1.00      0.67         2\n",
            "         309       0.67      1.00      0.80         2\n",
            "         310       0.75      0.75      0.75         4\n",
            "         311       0.88      1.00      0.93         7\n",
            "         312       0.60      1.00      0.75         3\n",
            "         313       0.90      0.90      0.90        10\n",
            "         314       1.00      0.67      0.80         6\n",
            "         315       1.00      1.00      1.00         6\n",
            "         316       1.00      0.83      0.91         6\n",
            "         317       0.60      1.00      0.75         3\n",
            "         318       0.80      1.00      0.89         4\n",
            "         319       1.00      0.91      0.95        11\n",
            "         320       0.50      0.50      0.50         2\n",
            "         321       1.00      1.00      1.00         3\n",
            "         322       0.83      0.83      0.83         6\n",
            "         323       1.00      0.67      0.80         3\n",
            "         324       0.71      0.83      0.77         6\n",
            "         325       1.00      1.00      1.00         2\n",
            "         326       1.00      1.00      1.00         7\n",
            "         327       1.00      1.00      1.00         2\n",
            "         328       0.80      0.80      0.80         5\n",
            "         329       0.75      1.00      0.86         3\n",
            "         330       1.00      1.00      1.00         5\n",
            "         331       0.33      1.00      0.50         2\n",
            "         332       1.00      0.80      0.89         5\n",
            "         333       0.80      1.00      0.89         4\n",
            "         334       0.75      0.60      0.67         5\n",
            "         335       1.00      0.67      0.80         6\n",
            "         336       1.00      1.00      1.00         2\n",
            "         337       1.00      1.00      1.00         4\n",
            "         338       0.75      1.00      0.86         3\n",
            "         339       1.00      0.50      0.67         4\n",
            "         340       0.67      0.67      0.67         3\n",
            "         341       0.71      0.71      0.71         7\n",
            "         342       1.00      1.00      1.00         1\n",
            "         343       0.50      1.00      0.67         1\n",
            "         344       1.00      1.00      1.00         1\n",
            "         346       1.00      1.00      1.00         3\n",
            "         347       1.00      1.00      1.00         2\n",
            "         348       0.80      1.00      0.89         4\n",
            "         349       1.00      1.00      1.00         2\n",
            "         350       1.00      1.00      1.00         2\n",
            "         351       0.00      0.00      0.00         1\n",
            "         352       1.00      0.75      0.86         4\n",
            "         353       0.80      1.00      0.89         4\n",
            "         354       1.00      0.75      0.86         4\n",
            "         355       0.75      0.60      0.67         5\n",
            "         356       1.00      0.50      0.67         2\n",
            "         357       1.00      1.00      1.00         2\n",
            "         358       0.33      0.50      0.40         2\n",
            "         359       0.33      0.33      0.33         3\n",
            "         360       1.00      0.50      0.67         2\n",
            "         361       0.50      0.50      0.50         2\n",
            "         362       1.00      0.50      0.67         2\n",
            "         363       0.67      0.67      0.67         3\n",
            "         364       0.67      1.00      0.80         4\n",
            "         365       1.00      0.33      0.50         3\n",
            "         366       1.00      1.00      1.00         3\n",
            "         367       1.00      1.00      1.00         3\n",
            "         368       0.67      1.00      0.80         2\n",
            "         370       1.00      1.00      1.00         1\n",
            "         371       0.00      0.00      0.00         1\n",
            "         372       1.00      0.67      0.80         3\n",
            "         373       1.00      1.00      1.00         1\n",
            "         375       1.00      1.00      1.00         1\n",
            "         376       0.75      1.00      0.86         3\n",
            "         377       1.00      1.00      1.00         1\n",
            "         378       0.50      1.00      0.67         1\n",
            "         379       0.00      0.00      0.00         0\n",
            "         381       1.00      1.00      1.00         1\n",
            "\n",
            "    accuracy                           0.85     87353\n",
            "   macro avg       0.84      0.84      0.83     87353\n",
            "weighted avg       0.85      0.85      0.85     87353\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           6       1.00      1.00      1.00         2\n",
            "           7       1.00      1.00      1.00         2\n",
            "           8       1.00      0.92      0.96        13\n",
            "           9       0.98      0.96      0.97        46\n",
            "          10       0.96      0.94      0.95       108\n",
            "          11       0.96      0.92      0.94       233\n",
            "          12       0.94      0.92      0.93       341\n",
            "          13       0.92      0.92      0.92       465\n",
            "          14       0.93      0.94      0.94       560\n",
            "          15       0.93      0.93      0.93       683\n",
            "          16       0.91      0.93      0.92       774\n",
            "          17       0.93      0.94      0.94       836\n",
            "          18       0.92      0.92      0.92       848\n",
            "          19       0.92      0.93      0.92       873\n",
            "          20       0.91      0.91      0.91       902\n",
            "          21       0.91      0.92      0.91       901\n",
            "          22       0.91      0.91      0.91       934\n",
            "          23       0.91      0.90      0.90       932\n",
            "          24       0.91      0.90      0.90       903\n",
            "          25       0.91      0.90      0.90       911\n",
            "          26       0.90      0.88      0.89       883\n",
            "          27       0.88      0.88      0.88       866\n",
            "          28       0.88      0.88      0.88       919\n",
            "          29       0.89      0.88      0.89       907\n",
            "          30       0.90      0.87      0.88       928\n",
            "          31       0.91      0.89      0.90       859\n",
            "          32       0.90      0.88      0.89       794\n",
            "          33       0.87      0.90      0.88       818\n",
            "          34       0.87      0.88      0.88       828\n",
            "          35       0.87      0.90      0.89       830\n",
            "          36       0.89      0.88      0.88       852\n",
            "          37       0.87      0.85      0.86       826\n",
            "          38       0.90      0.89      0.90       793\n",
            "          39       0.89      0.89      0.89       802\n",
            "          40       0.89      0.89      0.89       785\n",
            "          41       0.90      0.86      0.88       776\n",
            "          42       0.90      0.89      0.89       794\n",
            "          43       0.90      0.86      0.88       795\n",
            "          44       0.89      0.87      0.88       771\n",
            "          45       0.88      0.89      0.88       748\n",
            "          46       0.89      0.87      0.88       754\n",
            "          47       0.87      0.90      0.89       739\n",
            "          48       0.90      0.88      0.89       790\n",
            "          49       0.88      0.90      0.89       759\n",
            "          50       0.88      0.89      0.89       716\n",
            "          51       0.86      0.88      0.87       696\n",
            "          52       0.87      0.88      0.87       705\n",
            "          53       0.87      0.90      0.89       688\n",
            "          54       0.88      0.87      0.87       687\n",
            "          55       0.87      0.90      0.88       725\n",
            "          56       0.90      0.84      0.87       722\n",
            "          57       0.88      0.89      0.89       636\n",
            "          58       0.88      0.89      0.89       665\n",
            "          59       0.87      0.86      0.86       687\n",
            "          60       0.87      0.88      0.87       633\n",
            "          61       0.89      0.87      0.88       706\n",
            "          62       0.89      0.90      0.90       678\n",
            "          63       0.87      0.90      0.88       706\n",
            "          64       0.88      0.89      0.88       601\n",
            "          65       0.88      0.87      0.88       661\n",
            "          66       0.89      0.86      0.87       670\n",
            "          67       0.88      0.88      0.88       622\n",
            "          68       0.86      0.89      0.88       586\n",
            "          69       0.89      0.88      0.89       632\n",
            "          70       0.87      0.86      0.87       591\n",
            "          71       0.86      0.86      0.86       601\n",
            "          72       0.88      0.88      0.88       614\n",
            "          73       0.90      0.85      0.87       610\n",
            "          74       0.87      0.89      0.88       598\n",
            "          75       0.88      0.88      0.88       552\n",
            "          76       0.87      0.85      0.86       568\n",
            "          77       0.88      0.89      0.89       565\n",
            "          78       0.85      0.92      0.88       550\n",
            "          79       0.88      0.87      0.88       558\n",
            "          80       0.87      0.88      0.87       572\n",
            "          81       0.87      0.90      0.88       563\n",
            "          82       0.88      0.86      0.87       487\n",
            "          83       0.86      0.88      0.87       557\n",
            "          84       0.87      0.90      0.88       530\n",
            "          85       0.88      0.83      0.86       506\n",
            "          86       0.86      0.87      0.86       543\n",
            "          87       0.84      0.88      0.86       496\n",
            "          88       0.87      0.85      0.86       507\n",
            "          89       0.84      0.87      0.86       503\n",
            "          90       0.87      0.90      0.89       511\n",
            "          91       0.85      0.90      0.87       474\n",
            "          92       0.89      0.86      0.87       491\n",
            "          93       0.88      0.90      0.89       468\n",
            "          94       0.86      0.88      0.87       512\n",
            "          95       0.86      0.88      0.87       489\n",
            "          96       0.85      0.85      0.85       455\n",
            "          97       0.85      0.87      0.86       461\n",
            "          98       0.89      0.85      0.87       469\n",
            "          99       0.86      0.87      0.86       438\n",
            "         100       0.85      0.87      0.86       482\n",
            "         101       0.87      0.87      0.87       500\n",
            "         102       0.86      0.87      0.87       469\n",
            "         103       0.86      0.89      0.88       466\n",
            "         104       0.85      0.88      0.87       492\n",
            "         105       0.88      0.87      0.87       468\n",
            "         106       0.86      0.91      0.88       444\n",
            "         107       0.85      0.84      0.84       441\n",
            "         108       0.86      0.86      0.86       436\n",
            "         109       0.86      0.85      0.86       432\n",
            "         110       0.88      0.87      0.88       442\n",
            "         111       0.89      0.88      0.88       402\n",
            "         112       0.86      0.87      0.86       414\n",
            "         113       0.85      0.85      0.85       373\n",
            "         114       0.84      0.85      0.85       422\n",
            "         115       0.86      0.84      0.85       432\n",
            "         116       0.87      0.88      0.88       376\n",
            "         117       0.85      0.86      0.86       410\n",
            "         118       0.85      0.85      0.85       383\n",
            "         119       0.85      0.88      0.87       409\n",
            "         120       0.89      0.87      0.88       381\n",
            "         121       0.86      0.86      0.86       394\n",
            "         122       0.84      0.81      0.83       344\n",
            "         123       0.89      0.84      0.87       376\n",
            "         124       0.86      0.85      0.86       357\n",
            "         125       0.82      0.83      0.83       363\n",
            "         126       0.84      0.85      0.85       372\n",
            "         127       0.88      0.86      0.87       346\n",
            "         128       0.86      0.90      0.88       334\n",
            "         129       0.87      0.85      0.86       323\n",
            "         130       0.84      0.85      0.84       314\n",
            "         131       0.87      0.84      0.86       331\n",
            "         132       0.88      0.85      0.86       328\n",
            "         133       0.88      0.91      0.90       325\n",
            "         134       0.86      0.85      0.85       319\n",
            "         135       0.83      0.85      0.84       315\n",
            "         136       0.87      0.83      0.85       306\n",
            "         137       0.86      0.88      0.87       293\n",
            "         138       0.85      0.87      0.86       290\n",
            "         139       0.88      0.84      0.86       284\n",
            "         140       0.89      0.92      0.90       253\n",
            "         141       0.90      0.89      0.90       274\n",
            "         142       0.85      0.84      0.84       240\n",
            "         143       0.89      0.87      0.88       258\n",
            "         144       0.87      0.89      0.88       234\n",
            "         145       0.84      0.84      0.84       229\n",
            "         146       0.86      0.84      0.85       227\n",
            "         147       0.91      0.88      0.90       226\n",
            "         148       0.87      0.90      0.88       210\n",
            "         149       0.82      0.81      0.81       194\n",
            "         150       0.83      0.82      0.82       225\n",
            "         151       0.85      0.89      0.87       203\n",
            "         152       0.84      0.84      0.84       224\n",
            "         153       0.84      0.76      0.80       187\n",
            "         154       0.86      0.86      0.86       191\n",
            "         155       0.87      0.84      0.85       188\n",
            "         156       0.85      0.84      0.84       188\n",
            "         157       0.88      0.88      0.88       188\n",
            "         158       0.87      0.89      0.88       157\n",
            "         159       0.86      0.89      0.88       166\n",
            "         160       0.84      0.85      0.85       171\n",
            "         161       0.87      0.87      0.87       164\n",
            "         162       0.89      0.82      0.85       161\n",
            "         163       0.85      0.87      0.86       137\n",
            "         164       0.83      0.83      0.83       161\n",
            "         165       0.85      0.85      0.85       156\n",
            "         166       0.85      0.82      0.84       137\n",
            "         167       0.85      0.84      0.84       160\n",
            "         168       0.88      0.86      0.87       131\n",
            "         169       0.88      0.85      0.87       140\n",
            "         170       0.85      0.86      0.86       134\n",
            "         171       0.87      0.79      0.83       140\n",
            "         172       0.87      0.89      0.88       131\n",
            "         173       0.89      0.91      0.90       120\n",
            "         174       0.83      0.89      0.86       112\n",
            "         175       0.81      0.81      0.81       115\n",
            "         176       0.81      0.83      0.82       105\n",
            "         177       0.93      0.90      0.91       124\n",
            "         178       0.85      0.87      0.86       104\n",
            "         179       0.85      0.85      0.85       101\n",
            "         180       0.86      0.86      0.86       108\n",
            "         181       0.82      0.89      0.85        99\n",
            "         182       0.85      0.85      0.85       108\n",
            "         183       0.83      0.90      0.86        92\n",
            "         184       0.92      0.83      0.87       110\n",
            "         185       0.85      0.94      0.89       113\n",
            "         186       0.79      0.85      0.82        80\n",
            "         187       0.88      0.81      0.84        79\n",
            "         188       0.89      0.86      0.88       103\n",
            "         189       0.84      0.86      0.85        76\n",
            "         190       0.84      0.86      0.85        80\n",
            "         191       0.84      0.82      0.83        96\n",
            "         192       0.85      0.83      0.84        86\n",
            "         193       0.85      0.90      0.88        70\n",
            "         194       0.83      0.84      0.84        76\n",
            "         195       0.84      0.83      0.84        84\n",
            "         196       0.84      0.84      0.84        85\n",
            "         197       0.91      0.92      0.91        87\n",
            "         198       0.84      0.86      0.85        81\n",
            "         199       0.84      0.87      0.86        75\n",
            "         200       0.78      0.85      0.81        67\n",
            "         201       0.78      0.88      0.83        69\n",
            "         202       0.84      0.83      0.83        81\n",
            "         203       0.80      0.85      0.83        61\n",
            "         204       0.84      0.85      0.84        60\n",
            "         205       0.81      0.92      0.86        65\n",
            "         206       0.82      0.89      0.85        55\n",
            "         207       0.88      0.79      0.83        53\n",
            "         208       0.79      0.76      0.78        50\n",
            "         209       0.88      0.83      0.85        60\n",
            "         210       0.93      0.89      0.91        75\n",
            "         211       0.98      0.93      0.95        54\n",
            "         212       0.83      0.88      0.86        51\n",
            "         213       0.91      0.91      0.91        53\n",
            "         214       0.87      0.87      0.87        46\n",
            "         215       0.75      0.87      0.80        45\n",
            "         216       0.88      0.89      0.88        47\n",
            "         217       0.78      0.78      0.78        40\n",
            "         218       0.94      0.89      0.92        37\n",
            "         219       0.88      0.80      0.84        45\n",
            "         220       0.89      0.90      0.90        52\n",
            "         221       0.87      0.87      0.87        45\n",
            "         222       0.89      0.74      0.80        53\n",
            "         223       0.77      0.84      0.80        43\n",
            "         224       0.85      0.76      0.81        38\n",
            "         225       0.81      0.85      0.83        26\n",
            "         226       0.81      0.83      0.82        36\n",
            "         227       0.89      0.80      0.84        30\n",
            "         228       0.93      0.91      0.92        43\n",
            "         229       0.76      0.79      0.78        24\n",
            "         230       0.94      0.91      0.92        33\n",
            "         231       0.85      0.90      0.88        31\n",
            "         232       0.84      0.90      0.87        41\n",
            "         233       0.77      0.82      0.79        28\n",
            "         234       0.94      0.91      0.93        34\n",
            "         235       0.77      0.73      0.75        33\n",
            "         236       0.89      0.79      0.84        39\n",
            "         237       0.84      0.90      0.87        42\n",
            "         238       0.85      0.79      0.81        28\n",
            "         239       0.86      0.96      0.91        26\n",
            "         240       0.88      0.79      0.83        28\n",
            "         241       0.82      0.92      0.87        25\n",
            "         242       0.81      0.83      0.82        30\n",
            "         243       0.72      0.78      0.75        27\n",
            "         244       0.85      0.89      0.87        38\n",
            "         245       0.90      0.84      0.87        31\n",
            "         246       0.97      0.97      0.97        34\n",
            "         247       0.89      0.83      0.86        30\n",
            "         248       0.98      0.93      0.95        43\n",
            "         249       0.87      0.90      0.89        30\n",
            "         250       0.94      0.73      0.82        22\n",
            "         251       0.91      0.77      0.83        13\n",
            "         252       0.85      0.94      0.89        18\n",
            "         253       0.78      0.88      0.82        16\n",
            "         254       0.86      0.86      0.86        14\n",
            "         255       0.83      0.88      0.86        17\n",
            "         256       0.90      0.64      0.75        14\n",
            "         257       0.92      0.88      0.90        25\n",
            "         258       1.00      0.94      0.97        18\n",
            "         259       0.87      0.93      0.90        14\n",
            "         260       0.87      0.81      0.84        16\n",
            "         261       0.90      1.00      0.95        19\n",
            "         262       0.88      0.93      0.90        15\n",
            "         263       0.87      1.00      0.93        13\n",
            "         264       0.82      1.00      0.90        18\n",
            "         265       0.73      0.85      0.79        13\n",
            "         266       1.00      0.79      0.88        19\n",
            "         267       0.70      0.88      0.78         8\n",
            "         268       0.82      0.82      0.82        11\n",
            "         269       0.89      0.84      0.86        19\n",
            "         270       0.89      1.00      0.94         8\n",
            "         271       0.80      0.80      0.80         5\n",
            "         272       0.88      0.88      0.88        17\n",
            "         273       0.92      0.92      0.92        13\n",
            "         274       0.89      0.80      0.84        10\n",
            "         275       0.94      0.94      0.94        16\n",
            "         276       0.91      1.00      0.95        10\n",
            "         277       1.00      0.94      0.97        16\n",
            "         278       0.67      0.57      0.62         7\n",
            "         279       0.71      0.83      0.77        12\n",
            "         280       0.80      0.57      0.67         7\n",
            "         281       0.50      0.80      0.62         5\n",
            "         282       1.00      0.50      0.67        10\n",
            "         283       0.83      0.91      0.87        11\n",
            "         284       0.88      0.88      0.88        17\n",
            "         285       0.90      0.82      0.86        11\n",
            "         286       0.85      1.00      0.92        11\n",
            "         287       0.88      0.58      0.70        12\n",
            "         288       0.89      0.73      0.80        11\n",
            "         289       0.80      0.89      0.84         9\n",
            "         290       0.86      0.86      0.86         7\n",
            "         291       1.00      0.91      0.95        11\n",
            "         292       0.89      0.89      0.89         9\n",
            "         293       1.00      0.67      0.80         3\n",
            "         294       1.00      1.00      1.00         4\n",
            "         295       1.00      0.83      0.91         6\n",
            "         296       0.70      1.00      0.82         7\n",
            "         297       1.00      0.86      0.92         7\n",
            "         298       1.00      1.00      1.00         7\n",
            "         299       0.78      1.00      0.88         7\n",
            "         300       1.00      0.75      0.86         8\n",
            "         301       0.80      1.00      0.89         4\n",
            "         302       0.75      0.75      0.75         4\n",
            "         303       1.00      0.64      0.78        11\n",
            "         304       1.00      0.83      0.91        12\n",
            "         305       1.00      0.83      0.91         6\n",
            "         306       1.00      0.80      0.89         5\n",
            "         307       0.80      1.00      0.89         4\n",
            "         308       0.71      0.83      0.77         6\n",
            "         309       1.00      0.75      0.86         4\n",
            "         310       0.00      0.00      0.00         1\n",
            "         311       0.67      0.67      0.67         3\n",
            "         312       0.60      1.00      0.75         6\n",
            "         313       0.80      0.80      0.80         5\n",
            "         314       0.88      1.00      0.93         7\n",
            "         315       0.89      0.89      0.89         9\n",
            "         316       0.80      1.00      0.89         4\n",
            "         317       0.89      1.00      0.94         8\n",
            "         318       1.00      1.00      1.00         4\n",
            "         319       0.71      0.71      0.71         7\n",
            "         320       1.00      1.00      1.00         4\n",
            "         321       0.86      1.00      0.92         6\n",
            "         322       1.00      0.67      0.80         3\n",
            "         323       0.75      1.00      0.86         3\n",
            "         324       1.00      0.88      0.93         8\n",
            "         325       0.83      1.00      0.91         5\n",
            "         326       1.00      0.50      0.67         4\n",
            "         328       0.75      1.00      0.86         3\n",
            "         329       0.86      0.86      0.86         7\n",
            "         330       1.00      1.00      1.00         7\n",
            "         331       0.75      1.00      0.86         3\n",
            "         332       0.60      0.60      0.60         5\n",
            "         333       1.00      0.60      0.75         5\n",
            "         334       0.50      1.00      0.67         1\n",
            "         335       1.00      1.00      1.00         3\n",
            "         336       1.00      1.00      1.00         5\n",
            "         337       0.75      1.00      0.86         3\n",
            "         338       1.00      0.50      0.67         2\n",
            "         339       0.67      1.00      0.80         4\n",
            "         340       0.86      0.86      0.86         7\n",
            "         341       1.00      1.00      1.00         2\n",
            "         342       0.75      0.75      0.75         4\n",
            "         343       0.50      1.00      0.67         1\n",
            "         344       1.00      1.00      1.00         3\n",
            "         345       1.00      0.33      0.50         3\n",
            "         346       1.00      1.00      1.00         4\n",
            "         347       1.00      1.00      1.00         2\n",
            "         348       0.50      0.50      0.50         2\n",
            "         349       0.50      0.50      0.50         2\n",
            "         350       1.00      0.67      0.80         3\n",
            "         351       1.00      0.67      0.80         3\n",
            "         352       0.67      0.80      0.73         5\n",
            "         353       0.50      0.67      0.57         3\n",
            "         354       0.50      1.00      0.67         1\n",
            "         355       1.00      0.33      0.50         3\n",
            "         356       1.00      0.67      0.80         3\n",
            "         357       1.00      0.67      0.80         3\n",
            "         358       0.80      1.00      0.89         4\n",
            "         359       0.50      1.00      0.67         1\n",
            "         360       1.00      0.50      0.67         2\n",
            "         362       1.00      1.00      1.00         2\n",
            "         363       0.33      1.00      0.50         1\n",
            "         364       0.00      0.00      0.00         3\n",
            "         365       1.00      1.00      1.00         4\n",
            "         366       1.00      0.80      0.89         5\n",
            "         367       0.75      0.75      0.75         4\n",
            "         368       0.33      1.00      0.50         1\n",
            "         369       0.50      1.00      0.67         2\n",
            "         370       1.00      0.50      0.67         2\n",
            "         371       1.00      0.50      0.67         2\n",
            "         372       1.00      0.33      0.50         3\n",
            "         373       1.00      1.00      1.00         3\n",
            "         374       0.50      1.00      0.67         2\n",
            "         375       1.00      0.50      0.67         2\n",
            "         376       0.00      0.00      0.00         0\n",
            "         377       0.50      1.00      0.67         1\n",
            "         378       1.00      1.00      1.00         2\n",
            "         379       0.67      1.00      0.80         2\n",
            "         380       0.00      0.00      0.00         1\n",
            "         381       1.00      1.00      1.00         1\n",
            "         382       1.00      1.00      1.00         1\n",
            "\n",
            "    accuracy                           0.88     87353\n",
            "   macro avg       0.85      0.85      0.84     87353\n",
            "weighted avg       0.88      0.88      0.88     87353\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xC96iEmbhKZM",
        "colab_type": "text"
      },
      "source": [
        "우리가 훈련한 BERT MODEL을 구글 GDrive에 저장합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E3pc5zhjCet-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 87.7% 79.6%\n",
        "# 모델 저장하기\n",
        "path = \"gdrive/My Drive/Colab Notebooks/squad\"\n",
        "bert_model.save_weights(path+\"/(Uncased)Squad.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6E8ao5rBoADd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 저장한 bert_model 로드하기\n",
        "bert_model = get_bert_finetuning_model(model)\n",
        "path = \"gdrive/My Drive/Colab Notebooks/squad\"\n",
        "bert_model.load_weights(path+\"/(Uncased)Squad.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBRodQAqlXf7",
        "colab_type": "text"
      },
      "source": [
        "우리가 만든 모델이 잘 작동하는지 확인해보도록 하겠습니다.  \n",
        "모델을 훈련할 때 사용하지 않은 test data로 모델이 잘 작동하는지, 우리가 만든 버트 모델의 성능이 어느정도 되는지 알아보는 시간을 가져보겠습니다.  \n",
        "\n",
        "SQAUD 데이터 셋에서 test 용도로 쓰이는 dev 파일을 PANDAS DATAFRAME 형식으로 불러오는 함수를 정의합니다.  \n",
        "train 데이터와 모양이 약간 다르기 때문에, 함수를 새로 정의해야 합니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6FpWAQR1ZqGt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def squad_json_to_dataframe_dev(input_file_path, record_path = ['data','paragraphs','qas','answers'],\n",
        "                           verbose = 1):\n",
        "    \"\"\"\n",
        "    input_file_path: path to the squad json file.\n",
        "    record_path: path to deepest level in json file default value is\n",
        "    ['data','paragraphs','qas','answers']\n",
        "    verbose: 0 to suppress it default is 1\n",
        "    \"\"\"\n",
        "    if verbose:\n",
        "        print(\"Reading the json file\")    \n",
        "    file = json.loads(open(input_file_path).read())\n",
        "    if verbose:\n",
        "        print(\"processing...\")\n",
        "    # parsing different level's in the json file\n",
        "    js = pd.io.json.json_normalize(file , record_path )\n",
        "    m = pd.io.json.json_normalize(file, record_path[:-1] )\n",
        "    r = pd.io.json.json_normalize(file,record_path[:-2])\n",
        "    \n",
        "    #combining it into single dataframe\n",
        "    idx = np.repeat(r['context'].values, r.qas.str.len())\n",
        "    m['context'] = idx\n",
        "    main = m[['id','question','context','answers']].set_index('id').reset_index()\n",
        "    main['c_id'] = main['context'].factorize()[0]\n",
        "    if verbose:\n",
        "        print(\"shape of the dataframe is {}\".format(main.shape))\n",
        "        print(\"Done\")\n",
        "    return main"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMVFr75nZq9G",
        "colab_type": "code",
        "outputId": "2e421662-1c2f-4502-da3f-7a2735954092",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "input_file_path ='dev-v1.1.json'\n",
        "record_path = ['data','paragraphs','qas','answers']\n",
        "verbose = 0\n",
        "dev = squad_json_to_dataframe_dev(input_file_path=input_file_path,record_path=record_path)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading the json file\n",
            "processing...\n",
            "shape of the dataframe is (10570, 5)\n",
            "Done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1QSzFOnPmv4g",
        "colab_type": "text"
      },
      "source": [
        "TEST DATA가 잘 불려왔는지 확인해 보겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tluhycV8ZxqF",
        "colab_type": "code",
        "outputId": "ba149a53-91b0-4802-f91e-0a8a3288210d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 752
        }
      },
      "source": [
        "dev"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answers</th>\n",
              "      <th>c_id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>56be4db0acb8001400a502ec</td>\n",
              "      <td>Which NFL team represented the AFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>56be4db0acb8001400a502ed</td>\n",
              "      <td>Which NFL team represented the NFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 249, 'text': 'Carolina Panth...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>56be4db0acb8001400a502ee</td>\n",
              "      <td>Where did Super Bowl 50 take place?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 403, 'text': 'Santa Clara, C...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>56be4db0acb8001400a502ef</td>\n",
              "      <td>Which NFL team won Super Bowl 50?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>56be4db0acb8001400a502f0</td>\n",
              "      <td>What color was used to emphasize the 50th anni...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 488, 'text': 'gold'}, {'answ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10565</th>\n",
              "      <td>5737aafd1c456719005744fb</td>\n",
              "      <td>What is the metric term less used than the New...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 82, 'text': 'kilogram-force'...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10566</th>\n",
              "      <td>5737aafd1c456719005744fc</td>\n",
              "      <td>What is the kilogram-force sometimes reffered ...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 114, 'text': 'kilopond'}, {'...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10567</th>\n",
              "      <td>5737aafd1c456719005744fd</td>\n",
              "      <td>What is a very seldom used unit of mass in the...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 274, 'text': 'slug'}, {'answ...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10568</th>\n",
              "      <td>5737aafd1c456719005744fe</td>\n",
              "      <td>What seldom used term of a unit of force equal...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 712, 'text': 'kip'}, {'answe...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10569</th>\n",
              "      <td>5737aafd1c456719005744ff</td>\n",
              "      <td>What is the seldom used force unit equal to on...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 665, 'text': 'sthène'}, {'an...</td>\n",
              "      <td>2066</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10570 rows × 5 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                             id  ...  c_id\n",
              "0      56be4db0acb8001400a502ec  ...     0\n",
              "1      56be4db0acb8001400a502ed  ...     0\n",
              "2      56be4db0acb8001400a502ee  ...     0\n",
              "3      56be4db0acb8001400a502ef  ...     0\n",
              "4      56be4db0acb8001400a502f0  ...     0\n",
              "...                         ...  ...   ...\n",
              "10565  5737aafd1c456719005744fb  ...  2066\n",
              "10566  5737aafd1c456719005744fc  ...  2066\n",
              "10567  5737aafd1c456719005744fd  ...  2066\n",
              "10568  5737aafd1c456719005744fe  ...  2066\n",
              "10569  5737aafd1c456719005744ff  ...  2066\n",
              "\n",
              "[10570 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 128
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjU-UmIkhbcW",
        "colab_type": "text"
      },
      "source": [
        "train 데이터와 다르게 SQuAD의 dev 데이터의 정답은 1개 이상입니다.  \n",
        "따라서 여러 정답 중 하나 이상만 버트 모형이 맞춘다면, 정답을 맞춘 것으로 인정하는 것입니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-aFMC6w8Dfjc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 정답의 개수를 정의하는 칼럼 생성\n",
        "dev['answer_len'] = dev['answers'].map(lambda x: len(x))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2iPkqiEWHo9o",
        "colab_type": "code",
        "outputId": "337943d1-7da0-4758-fce7-a88bc5437a0a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 90
        }
      },
      "source": [
        "print(dev['answers'][0])\n",
        "print()\n",
        "print(dev['answers'][0][0]['text'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Denver Broncos\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zTxqFDiWHMUH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 정답들을 다루기 쉽게 리스트로 반환하는 함수 정의\n",
        "def get_text(text_len, answers):\n",
        "  # text_len : 질문(question)과 문장(context)에 해당하는 정답의 개수\n",
        "  # answers : 정답 ex) [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
        "  texts = []\n",
        "  for i in range(text_len):\n",
        "    texts.append(answers[i]['text'])\n",
        "  return texts"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4pxJKiMrovNl",
        "colab_type": "code",
        "outputId": "346c722c-2a66-47b9-c11e-d2d350209c75",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# 정답의 개수(3)와 정답들([{'answer_start' : 177, 'text': 'Denver Broncos'}.....])을 인풋으로 받아서\n",
        "# 정답들을 리스트로 바꿔주는 함수 실행\n",
        "get_text(3, dev['answers'][0])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Denver Broncos', 'Denver Broncos', 'Denver Broncos']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 132
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1ezcP85qD7NA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# texts 칼럼의 모든 데이터에 대해서 수행\n",
        "dev['texts'] = dev.apply(lambda x: get_text(x['answer_len'], x['answers']), axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WY0n5Wt-qHKt",
        "colab_type": "code",
        "outputId": "79cd5553-a9eb-4696-9f57-a1411435c95b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 590
        }
      },
      "source": [
        "dev.head(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answers</th>\n",
              "      <th>c_id</th>\n",
              "      <th>answer_len</th>\n",
              "      <th>texts</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>56be4db0acb8001400a502ec</td>\n",
              "      <td>Which NFL team represented the AFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Denver Broncos, Denver Broncos, Denver Broncos]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>56be4db0acb8001400a502ed</td>\n",
              "      <td>Which NFL team represented the NFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 249, 'text': 'Carolina Panth...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Carolina Panthers, Carolina Panthers, Carolin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>56be4db0acb8001400a502ee</td>\n",
              "      <td>Where did Super Bowl 50 take place?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 403, 'text': 'Santa Clara, C...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Santa Clara, California, Levi's Stadium, Levi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>56be4db0acb8001400a502ef</td>\n",
              "      <td>Which NFL team won Super Bowl 50?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Denver Broncos, Denver Broncos, Denver Broncos]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>56be4db0acb8001400a502f0</td>\n",
              "      <td>What color was used to emphasize the 50th anni...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 488, 'text': 'gold'}, {'answ...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[gold, gold, gold]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         id  ...                                              texts\n",
              "0  56be4db0acb8001400a502ec  ...   [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "1  56be4db0acb8001400a502ed  ...  [Carolina Panthers, Carolina Panthers, Carolin...\n",
              "2  56be4db0acb8001400a502ee  ...  [Santa Clara, California, Levi's Stadium, Levi...\n",
              "3  56be4db0acb8001400a502ef  ...   [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "4  56be4db0acb8001400a502f0  ...                                 [gold, gold, gold]\n",
              "\n",
              "[5 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 134
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gd1UQ9ElIClr",
        "colab_type": "code",
        "outputId": "39caef79-1c3d-48e6-ce82-4248e7b082a3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "dev['texts']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0         [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "1        [Carolina Panthers, Carolina Panthers, Carolin...\n",
              "2        [Santa Clara, California, Levi's Stadium, Levi...\n",
              "3         [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "4                                       [gold, gold, gold]\n",
              "                               ...                        \n",
              "10565    [kilogram-force, pound-force, kilogram-force (...\n",
              "10566    [kilopond, kilopond, kilopond, kilopond, kilop...\n",
              "10567    [slug, metric slug, metric slug, metric slug, ...\n",
              "10568                            [kip, kip, kip, kip, kip]\n",
              "10569             [sthène, sthène, sthène, sthène, sthène]\n",
              "Name: texts, Length: 10570, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9614q2jmcq-",
        "colab_type": "text"
      },
      "source": [
        "테스트 데이터에 대해서 결과를 확인합니다.  \n",
        "훈련에 사용하지 않은 테스트 데이터에 대한 예측을 제법 잘 수행하는 것을 보실 수 있겠습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qdn9bFW3Hhd6",
        "colab_type": "text"
      },
      "source": [
        "딥러닝에서 중요한 것 중 하나는 모델의 성능이 얼마나 되는지의 척도입니다.\n",
        "**F1 Score**와 **Exact Match**를 측정해 보도록 하겠습니다.  \n",
        "SQuAD에서의 **F1 Score**는 예측 단어들 중에서 얼마나 버트 모형이 예측한 것과 일치하는 지를 나타내는 척도입니다.  \n",
        "예를 들면, 버트가 정답으로 [new, england, patriots]를 예측하였고, 실제 정답 또한 [new, england, patriots]라면, 예측 단어와 정답 단어가 모두 다 일치하기 때문에 F1 Score는 1.0 입니다.  \n",
        "만약에, 버트가 정답으로 [england, patriots]만 예측했다면 F1 Score는 세 단어 중 2가지만 예측했기 때문에, 0.66(66%)가 됩니다.  \n",
        "  \n",
        "**EM**(Exact Match)은 정답단어들과 예측단어들이 완전히 일치하면 1, 일치하지 않으면 0입니다.  \n",
        "정답, 예측값 모두 다 [new, england, patriots]가 되어야 합니다.  \n",
        "  \n",
        "**F1**과 **EM**을 계산해보도록 하겠습니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pWotvXjSKbbT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "TEXT_COLUMN = 'texts'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qBNXvvrGqSf_",
        "colab_type": "text"
      },
      "source": [
        "아까 TRAIN 데이터를 버트 인풋으로 변환하기 위한 함수를 살짝 변형하도록 하겠습니다.  \n",
        "우리가 훈련한 버트 모형을 바탕으로 DEV 데이터를 예측하기 위해서 인풋을 만들어보도록 하겠습니다.  \n",
        "TRAIN 데이터를 인풋으로 만들기 위한 함수랑 다른 점은, 정답을 출력하지 않는다는 점입니다. 예측만 하면 되니까요."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GewQ--m3DBYO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def convert_data(data_df):\n",
        "    global tokenizer\n",
        "    indices, segments, target_start, target_end = [], [], [], []\n",
        "\n",
        "    for i in tqdm(range(len(data_df))):\n",
        "        que, _ = tokenizer.encode(data_df[QUESTION_COLUMN][i])\n",
        "        doc, _ = tokenizer.encode(data_df[DATA_COLUMN][i])\n",
        "        doc.pop(0)\n",
        "\n",
        "        que_len = len(que)\n",
        "        doc_len = len(doc)\n",
        "\n",
        "        if que_len > 64:\n",
        "          que = que[:63]\n",
        "          que.append(102)\n",
        "        \n",
        "        if len(que+doc) > SEQ_LEN:\n",
        "          while len(que+doc) != SEQ_LEN:\n",
        "            doc.pop(-1)\n",
        "\n",
        "          doc.pop(-1)\n",
        "          doc.append(102)\n",
        "\n",
        "        segment = [0]*len(que) + [1]*len(doc) + [0]*(SEQ_LEN-len(que)-len(doc))\n",
        "        if len(que + doc) <= SEQ_LEN:\n",
        "          while len(que+doc) != SEQ_LEN:\n",
        "            doc.append(0)\n",
        "\n",
        "        ids = que + doc\n",
        "\n",
        "        texts = data_df[TEXT_COLUMN][i]\n",
        "        for text_element in texts:\n",
        "          text = tokenizer.encode(text_element)[0]\n",
        "\n",
        "          text_slide_len = len(text[1:-1])\n",
        "          for j in range(0,(len(doc))):  \n",
        "              exist_flag = 0\n",
        "              if text[1:-1] == doc[j:j+text_slide_len]:\n",
        "                ans_start = j + len(que)\n",
        "                ans_end = j + text_slide_len - 1 + len(que)\n",
        "                exist_flag = 1\n",
        "                break\n",
        "        \n",
        "          if exist_flag == 0:\n",
        "            ans_start = SEQ_LEN\n",
        "            ans_end = SEQ_LEN\n",
        "\n",
        "        indices.append(ids)\n",
        "        segments.append(segment)\n",
        "        target_start.append(ans_start)\n",
        "        target_end.append(ans_end)\n",
        "        \n",
        "    indices_x = np.array(indices)\n",
        "    segments = np.array(segments)\n",
        "    target_start = np.array(target_start)\n",
        "    target_end = np.array(target_end)\n",
        "\n",
        "    del_list = np.where(target_start!=SEQ_LEN)[0]\n",
        "    not_del_list = np.where(target_start==SEQ_LEN)[0]\n",
        "    indices_x = indices_x[del_list]\n",
        "    segments = segments[del_list]\n",
        "\n",
        "    target_start = target_start[del_list]\n",
        "    target_end = target_end[del_list]\n",
        "    \n",
        "    return [indices_x, segments], del_list\n",
        "\n",
        "def load_data(pandas_dataframe):\n",
        "    data_df = pandas_dataframe\n",
        "    data_df[DATA_COLUMN] = data_df[DATA_COLUMN].astype(str)\n",
        "    data_df[QUESTION_COLUMN] = data_df[QUESTION_COLUMN].astype(str)\n",
        "    data_x, data_y, del_list = convert_data(data_df)\n",
        "\n",
        "    return data_x, data_y, del_list"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bniBY9L60zMW",
        "colab_type": "code",
        "outputId": "d8d90ec5-4aa8-4256-c818-e876659c4b06",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "dev_bert_input = convert_data(dev)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 10570/10570 [00:24<00:00, 428.98it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_luMTVn0o9qY",
        "colab_type": "code",
        "outputId": "22501f75-8f41-4b84-9f13-7f1463236542",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        }
      },
      "source": [
        "dev_bert_input"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "([array([[  101,  2029,  5088, ...,     0,     0,     0],\n",
              "         [  101,  2029,  5088, ...,     0,     0,     0],\n",
              "         [  101,  2073,  2106, ...,     0,     0,     0],\n",
              "         ...,\n",
              "         [  101,  2054,  2003, ...,     0,     0,     0],\n",
              "         [  101,  2054, 15839, ...,     0,     0,     0],\n",
              "         [  101,  2054,  2003, ...,     0,     0,     0]]),\n",
              "  array([[0, 0, 0, ..., 0, 0, 0],\n",
              "         [0, 0, 0, ..., 0, 0, 0],\n",
              "         [0, 0, 0, ..., 0, 0, 0],\n",
              "         ...,\n",
              "         [0, 0, 0, ..., 0, 0, 0],\n",
              "         [0, 0, 0, ..., 0, 0, 0],\n",
              "         [0, 0, 0, ..., 0, 0, 0]])],\n",
              " array([    0,     1,     2, ..., 10567, 10568, 10569]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D5Z1SdHlLrKr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dev_bert_input, del_list = dev_bert_input[0], dev_bert_input[1]\n",
        "dev = dev.iloc[del_list]\n",
        "dev = dev.reset_index(drop=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jtmY0kdbpO-c",
        "colab_type": "code",
        "outputId": "ef0c7246-dfb8-45fa-c1e1-479811f22377",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "dev"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answers</th>\n",
              "      <th>c_id</th>\n",
              "      <th>answer_len</th>\n",
              "      <th>texts</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>56be4db0acb8001400a502ec</td>\n",
              "      <td>Which NFL team represented the AFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Denver Broncos, Denver Broncos, Denver Broncos]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>56be4db0acb8001400a502ed</td>\n",
              "      <td>Which NFL team represented the NFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 249, 'text': 'Carolina Panth...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Carolina Panthers, Carolina Panthers, Carolin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>56be4db0acb8001400a502ee</td>\n",
              "      <td>Where did Super Bowl 50 take place?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 403, 'text': 'Santa Clara, C...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Santa Clara, California, Levi's Stadium, Levi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>56be4db0acb8001400a502ef</td>\n",
              "      <td>Which NFL team won Super Bowl 50?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Denver Broncos, Denver Broncos, Denver Broncos]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>56be4db0acb8001400a502f0</td>\n",
              "      <td>What color was used to emphasize the 50th anni...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 488, 'text': 'gold'}, {'answ...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[gold, gold, gold]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10523</th>\n",
              "      <td>5737aafd1c456719005744fb</td>\n",
              "      <td>What is the metric term less used than the New...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 82, 'text': 'kilogram-force'...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[kilogram-force, pound-force, kilogram-force (...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10524</th>\n",
              "      <td>5737aafd1c456719005744fc</td>\n",
              "      <td>What is the kilogram-force sometimes reffered ...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 114, 'text': 'kilopond'}, {'...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[kilopond, kilopond, kilopond, kilopond, kilop...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10525</th>\n",
              "      <td>5737aafd1c456719005744fd</td>\n",
              "      <td>What is a very seldom used unit of mass in the...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 274, 'text': 'slug'}, {'answ...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[slug, metric slug, metric slug, metric slug, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10526</th>\n",
              "      <td>5737aafd1c456719005744fe</td>\n",
              "      <td>What seldom used term of a unit of force equal...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 712, 'text': 'kip'}, {'answe...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[kip, kip, kip, kip, kip]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10527</th>\n",
              "      <td>5737aafd1c456719005744ff</td>\n",
              "      <td>What is the seldom used force unit equal to on...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 665, 'text': 'sthène'}, {'an...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[sthène, sthène, sthène, sthène, sthène]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10528 rows × 7 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                             id  ...                                              texts\n",
              "0      56be4db0acb8001400a502ec  ...   [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "1      56be4db0acb8001400a502ed  ...  [Carolina Panthers, Carolina Panthers, Carolin...\n",
              "2      56be4db0acb8001400a502ee  ...  [Santa Clara, California, Levi's Stadium, Levi...\n",
              "3      56be4db0acb8001400a502ef  ...   [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "4      56be4db0acb8001400a502f0  ...                                 [gold, gold, gold]\n",
              "...                         ...  ...                                                ...\n",
              "10523  5737aafd1c456719005744fb  ...  [kilogram-force, pound-force, kilogram-force (...\n",
              "10524  5737aafd1c456719005744fc  ...  [kilopond, kilopond, kilopond, kilopond, kilop...\n",
              "10525  5737aafd1c456719005744fd  ...  [slug, metric slug, metric slug, metric slug, ...\n",
              "10526  5737aafd1c456719005744fe  ...                          [kip, kip, kip, kip, kip]\n",
              "10527  5737aafd1c456719005744ff  ...           [sthène, sthène, sthène, sthène, sthène]\n",
              "\n",
              "[10528 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCjnhtTf4qa2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 문장이 토큰화된 것 저장 (1~10527)\n",
        "indexes = dev_bert_input[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1e2QLZUIqxeV",
        "colab_type": "code",
        "outputId": "6a828759-9154-48f5-ce23-07dcc21196cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        }
      },
      "source": [
        "indexes"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  101,  2029,  5088, ...,     0,     0,     0],\n",
              "       [  101,  2029,  5088, ...,     0,     0,     0],\n",
              "       [  101,  2073,  2106, ...,     0,     0,     0],\n",
              "       ...,\n",
              "       [  101,  2054,  2003, ...,     0,     0,     0],\n",
              "       [  101,  2054, 15839, ...,     0,     0,     0],\n",
              "       [  101,  2054,  2003, ...,     0,     0,     0]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 141
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JcFadOpX2XZU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 버트 예측(384차원)\n",
        "bert_predictions = bert_model.predict(dev_bert_input)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N7SSPU7L3HDs",
        "colab_type": "code",
        "outputId": "534d0880-1d9d-4c3a-b08b-5e9822f11d7a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 480
        }
      },
      "source": [
        "bert_predictions"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([[4.5046983e-07, 8.2021506e-07, 4.2243002e-07, ..., 3.9619800e-07,\n",
              "         3.9627736e-07, 3.9691542e-07],\n",
              "        [3.5265381e-07, 6.3832340e-07, 3.5860825e-07, ..., 3.1206858e-07,\n",
              "         3.1224306e-07, 3.1323265e-07],\n",
              "        [1.8186619e-06, 4.2199181e-06, 6.4185650e-07, ..., 4.0183440e-07,\n",
              "         4.0148277e-07, 4.0014453e-07],\n",
              "        ...,\n",
              "        [8.4676009e-07, 1.5694970e-06, 2.4437864e-07, ..., 2.3756277e-07,\n",
              "         2.3934587e-07, 2.3976554e-07],\n",
              "        [7.9948750e-06, 9.1278685e-07, 1.1206664e-06, ..., 4.5795409e-07,\n",
              "         4.6148909e-07, 4.5440413e-07],\n",
              "        [1.7398277e-06, 1.0472634e-06, 1.9529806e-07, ..., 1.8871295e-07,\n",
              "         1.8935978e-07, 1.8979549e-07]], dtype=float32),\n",
              " array([[1.9190898e-06, 1.3929861e-06, 7.9231029e-07, ..., 7.6027987e-07,\n",
              "         7.5993921e-07, 7.6107352e-07],\n",
              "        [7.7802014e-07, 5.3432677e-07, 2.9617212e-07, ..., 2.8263986e-07,\n",
              "         2.8237960e-07, 2.8302773e-07],\n",
              "        [7.0246110e-06, 3.0644499e-06, 7.7331504e-07, ..., 7.4997814e-07,\n",
              "         7.5100877e-07, 7.5404012e-07],\n",
              "        ...,\n",
              "        [1.7342244e-06, 1.3075927e-06, 2.6869893e-07, ..., 2.3657209e-07,\n",
              "         2.3669260e-07, 2.3773300e-07],\n",
              "        [9.2994505e-06, 7.0388427e-07, 7.8431680e-07, ..., 4.7877705e-07,\n",
              "         4.8593330e-07, 4.8797898e-07],\n",
              "        [2.7808217e-06, 6.9109245e-07, 2.2592758e-07, ..., 2.0700429e-07,\n",
              "         2.0640975e-07, 2.0684151e-07]], dtype=float32)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 143
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HMx1btZW3Li6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 예측값 중 가장 큰 값을 가진 부분 추출(가장 큰 확률이 위치해있는 곳)\n",
        "start_indexes = np.argmax(bert_predictions[0], axis=-1)\n",
        "end_indexes = np.argmax(bert_predictions[1], axis=-1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0IiVfSwoftof",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 정답 중 의미없는 부분 삭제\n",
        "# 첫 번째 토큰에 해당하는 start_indexes가 마지막 토큰에 해당하는 end_indexes보다 작다면\n",
        "# 말이 안 되므로, 이러한 경우를 제거\n",
        "not_del_list = np.where(start_indexes <= end_indexes)[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aWo2fNrvgw2l",
        "colab_type": "code",
        "outputId": "6e2a38a0-d752-4a50-e1ae-c704840278bf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "not_del_list.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10283,)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HFPlADlKf3Gy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "start_indexes = start_indexes[not_del_list]\n",
        "end_indexes = end_indexes[not_del_list]\n",
        "indexes = indexes[not_del_list]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P2p2P0Ohne_p",
        "colab_type": "code",
        "outputId": "94afb47b-e012-48b6-c010-5fd4f72e310a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        }
      },
      "source": [
        "start_indexes[0:100]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 46,  57,  78,  43, 118,  16,  72,  42,  16,  22,  73,  41,  80,\n",
              "        91, 156,  35,  83,  91,  80,  58,  77,  31,  42,  53,  41,  35,\n",
              "        42,  77,  11,  44,  27, 133,  66,  40,  87,  44,  43,  83, 127,\n",
              "        26,  28,  33,  87, 127,  95,  25,  43, 132,  42,  29,  44,  46,\n",
              "        24,  44,  65,  58,  81,  14,  59,  72,  25,  36,  57,  43,  77,\n",
              "        64,  60,  75,  84,  59,  70,  40,  41,  49,  61,  56,  38,  47,\n",
              "        61,  74,  16,  34,  55,  65,  76,  15,  31,  62,  62,  12,  36,\n",
              "        67,  97,   9,  31,  56,  66,  10,  55,  62])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 148
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x9Gd_pk4ngbZ",
        "colab_type": "code",
        "outputId": "4d6096a6-cc05-4868-898d-3491cba34538",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159
        }
      },
      "source": [
        "end_indexes[0:100]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 47,  58,  81,  44, 118,  34,  75,  43,  34,  24,  76,  42,  83,\n",
              "        92, 158,  35,  83,  92,  83,  60,  80,  74,  43,  54,  42,  35,\n",
              "        43,  80,  13,  45,  28, 133,  66,  41,  89,  45,  44,  85, 127,\n",
              "        27,  30,  34,  90, 127,  97,  26,  44, 132,  43,  30,  45,  47,\n",
              "        25,  45,  65,  59,  81,  14,  60,  72,  25,  36,  58,  43,  77,\n",
              "        65,  60,  75,  84,  60,  70,  40,  47,  50,  62,  56,  39,  47,\n",
              "        62,  74,  16,  36,  56,  68,  80,  15,  33,  65,  65,  12,  38,\n",
              "        70,  97,   9,  33,  57,  69,  10,  56,  65])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 149
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PoYtGARtggE0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# dev 데이터셋 재조정\n",
        "dev = dev.iloc[not_del_list].reset_index(drop=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ugAVGrskg4TJ",
        "colab_type": "code",
        "outputId": "4f5ce8c9-190e-4bc4-b527-3e7a23da94a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "dev"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>question</th>\n",
              "      <th>context</th>\n",
              "      <th>answers</th>\n",
              "      <th>c_id</th>\n",
              "      <th>answer_len</th>\n",
              "      <th>texts</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>56be4db0acb8001400a502ec</td>\n",
              "      <td>Which NFL team represented the AFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Denver Broncos, Denver Broncos, Denver Broncos]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>56be4db0acb8001400a502ed</td>\n",
              "      <td>Which NFL team represented the NFC at Super Bo...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 249, 'text': 'Carolina Panth...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Carolina Panthers, Carolina Panthers, Carolin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>56be4db0acb8001400a502ee</td>\n",
              "      <td>Where did Super Bowl 50 take place?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 403, 'text': 'Santa Clara, C...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Santa Clara, California, Levi's Stadium, Levi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>56be4db0acb8001400a502ef</td>\n",
              "      <td>Which NFL team won Super Bowl 50?</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 177, 'text': 'Denver Broncos...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[Denver Broncos, Denver Broncos, Denver Broncos]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>56be4db0acb8001400a502f0</td>\n",
              "      <td>What color was used to emphasize the 50th anni...</td>\n",
              "      <td>Super Bowl 50 was an American football game to...</td>\n",
              "      <td>[{'answer_start': 488, 'text': 'gold'}, {'answ...</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>[gold, gold, gold]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10278</th>\n",
              "      <td>5737aafd1c456719005744fb</td>\n",
              "      <td>What is the metric term less used than the New...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 82, 'text': 'kilogram-force'...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[kilogram-force, pound-force, kilogram-force (...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10279</th>\n",
              "      <td>5737aafd1c456719005744fc</td>\n",
              "      <td>What is the kilogram-force sometimes reffered ...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 114, 'text': 'kilopond'}, {'...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[kilopond, kilopond, kilopond, kilopond, kilop...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10280</th>\n",
              "      <td>5737aafd1c456719005744fd</td>\n",
              "      <td>What is a very seldom used unit of mass in the...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 274, 'text': 'slug'}, {'answ...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[slug, metric slug, metric slug, metric slug, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10281</th>\n",
              "      <td>5737aafd1c456719005744fe</td>\n",
              "      <td>What seldom used term of a unit of force equal...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 712, 'text': 'kip'}, {'answe...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[kip, kip, kip, kip, kip]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10282</th>\n",
              "      <td>5737aafd1c456719005744ff</td>\n",
              "      <td>What is the seldom used force unit equal to on...</td>\n",
              "      <td>The pound-force has a metric counterpart, less...</td>\n",
              "      <td>[{'answer_start': 665, 'text': 'sthène'}, {'an...</td>\n",
              "      <td>2066</td>\n",
              "      <td>5</td>\n",
              "      <td>[sthène, sthène, sthène, sthène, sthène]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>10283 rows × 7 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                             id  ...                                              texts\n",
              "0      56be4db0acb8001400a502ec  ...   [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "1      56be4db0acb8001400a502ed  ...  [Carolina Panthers, Carolina Panthers, Carolin...\n",
              "2      56be4db0acb8001400a502ee  ...  [Santa Clara, California, Levi's Stadium, Levi...\n",
              "3      56be4db0acb8001400a502ef  ...   [Denver Broncos, Denver Broncos, Denver Broncos]\n",
              "4      56be4db0acb8001400a502f0  ...                                 [gold, gold, gold]\n",
              "...                         ...  ...                                                ...\n",
              "10278  5737aafd1c456719005744fb  ...  [kilogram-force, pound-force, kilogram-force (...\n",
              "10279  5737aafd1c456719005744fc  ...  [kilopond, kilopond, kilopond, kilopond, kilop...\n",
              "10280  5737aafd1c456719005744fd  ...  [slug, metric slug, metric slug, metric slug, ...\n",
              "10281  5737aafd1c456719005744fe  ...                          [kip, kip, kip, kip, kip]\n",
              "10282  5737aafd1c456719005744ff  ...           [sthène, sthène, sthène, sthène, sthène]\n",
              "\n",
              "[10283 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 151
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hNHYY102nBSH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# length : dev 데이터의 길이\n",
        "length = len(dev)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y2NPGJqQ343L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sentences = []\n",
        "\n",
        "untokenized = []\n",
        "\n",
        "for j in range(len(start_indexes)):\n",
        "  sentence = []\n",
        "  for i in range(start_indexes[j], end_indexes[j]+1):\n",
        "    token_based_word = reverse_token_dict[indexes[j][i]]\n",
        "    sentence.append(token_based_word)\n",
        "    # 문장이 토큰화된 단어 하나 하나를 sentence에 저장\n",
        "  \n",
        "  sentence_string = \"\"\n",
        "  \n",
        "  for w in sentence:\n",
        "    \n",
        "    if w.startswith(\"##\"):\n",
        "      w = w.replace(\"##\", \"\")\n",
        "      # 만약 sentence 안의 토큰이 ##으로 시작한다면, ##을 제거\n",
        "    else:\n",
        "      w = \" \" + w\n",
        "      # 토큰이 ##으로 시작하지 않는다면 글자의 첫 시작이므로, 띄어쓰기 추가\n",
        "    sentence_string += w\n",
        "      # 리스트로 되어 있는 토큰들을 하나로 합쳐줌\n",
        "  if sentence_string.startswith(\" \"):\n",
        "    sentence_string = \"\" + sentence_string[1:]\n",
        "    # sentence_string이 \" \"로 시작하는 경우에는 띄어쓰기를 삭제\n",
        "  untokenized.append(sentence_string)\n",
        "  # 리스트로 되어있는 토큰들을 하나로 합쳐준 것, 이것을 untokenized에 저장\n",
        "  sentences.append(sentence)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-St5R6svvQ_E",
        "colab_type": "code",
        "outputId": "026783bc-e696-4534-f768-a18f6001a5fe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "print(sentences[:30])\n",
        "print(untokenized[:30])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['denver', 'broncos'], ['carolina', 'panthers'], ['levi', \"'\", 's', 'stadium'], ['denver', 'broncos'], ['gold'], ['american', 'football', 'game', 'to', 'determine', 'the', 'champion', 'of', 'the', 'national', 'football', 'league', '(', 'nfl', ')', 'for', 'the', '2015', 'season'], ['february', '7', ',', '2016'], ['denver', 'broncos'], ['american', 'football', 'game', 'to', 'determine', 'the', 'champion', 'of', 'the', 'national', 'football', 'league', '(', 'nfl', ')', 'for', 'the', '2015', 'season'], ['national', 'football', 'league'], ['february', '7', ',', '2016'], ['denver', 'broncos'], ['levi', \"'\", 's', 'stadium'], ['santa', 'clara'], ['super', 'bowl', 'l'], ['2015'], ['2016'], ['santa', 'clara'], ['levi', \"'\", 's', 'stadium'], ['24', '–', '10'], ['february', '7', ',', '2016'], ['2015', 'season', '.', 'the', 'american', 'football', 'conference', '(', 'afc', ')', 'champion', 'denver', 'broncos', 'defeated', 'the', 'national', 'football', 'conference', '(', 'nfc', ')', 'champion', 'carolina', 'panthers', '24', '–', '10', 'to', 'earn', 'their', 'third', 'super', 'bowl', 'title', '.', 'the', 'game', 'was', 'played', 'on', 'february', '7', ',', '2016'], ['denver', 'broncos'], ['carolina', 'panthers'], ['denver', 'broncos'], ['2015'], ['denver', 'broncos'], ['levi', \"'\", 's', 'stadium'], ['super', 'bowl', '50'], ['denver', 'broncos']]\n",
            "['denver broncos', 'carolina panthers', \"levi ' s stadium\", 'denver broncos', 'gold', 'american football game to determine the champion of the national football league ( nfl ) for the 2015 season', 'february 7 , 2016', 'denver broncos', 'american football game to determine the champion of the national football league ( nfl ) for the 2015 season', 'national football league', 'february 7 , 2016', 'denver broncos', \"levi ' s stadium\", 'santa clara', 'super bowl l', '2015', '2016', 'santa clara', \"levi ' s stadium\", '24 – 10', 'february 7 , 2016', '2015 season . the american football conference ( afc ) champion denver broncos defeated the national football conference ( nfc ) champion carolina panthers 24 – 10 to earn their third super bowl title . the game was played on february 7 , 2016', 'denver broncos', 'carolina panthers', 'denver broncos', '2015', 'denver broncos', \"levi ' s stadium\", 'super bowl 50', 'denver broncos']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J80Yp_Tr-Jo2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dev_answers = []\n",
        "for i in range(length):\n",
        "  dev_answer = []\n",
        "  texts_dict = dev['answers'][i]\n",
        "  \n",
        "  for j in range(len(texts_dict)):\n",
        "    dev_answer.append(texts_dict[j]['text'])\n",
        "    # 정답 하나 하나를 리스트로 저장\n",
        "  dev_answers.append(dev_answer)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mKJhc1oOzVwm",
        "colab_type": "code",
        "outputId": "36cd7258-6106-4e66-8bb6-daa36baa1a5c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "dev_answers[0:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['Denver Broncos', 'Denver Broncos', 'Denver Broncos'],\n",
              " ['Carolina Panthers', 'Carolina Panthers', 'Carolina Panthers'],\n",
              " ['Santa Clara, California',\n",
              "  \"Levi's Stadium\",\n",
              "  \"Levi's Stadium in the San Francisco Bay Area at Santa Clara, California.\"],\n",
              " ['Denver Broncos', 'Denver Broncos', 'Denver Broncos'],\n",
              " ['gold', 'gold', 'gold'],\n",
              " ['\"golden anniversary\"', 'gold-themed', '\"golden anniversary'],\n",
              " ['February 7, 2016', 'February 7', 'February 7, 2016'],\n",
              " ['American Football Conference',\n",
              "  'American Football Conference',\n",
              "  'American Football Conference'],\n",
              " ['\"golden anniversary\"', 'gold-themed', 'gold'],\n",
              " ['American Football Conference',\n",
              "  'American Football Conference',\n",
              "  'American Football Conference']]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 156
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J1J_8kRiOf57",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dev_tokens = []\n",
        "for i in dev_answers:\n",
        "  dev_tokened = []\n",
        "  for j in i:\n",
        "    temp_token = tokenizer.tokenize(j)\n",
        "    # 정답을 토큰화\n",
        "    temp_token.pop(0)\n",
        "    # [CLS] 제거\n",
        "    temp_token.pop(-1)\n",
        "    # [SEP] 제거\n",
        "    dev_tokened.append(temp_token)\n",
        "  dev_tokens.append(dev_tokened)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FFvtndDLzppo",
        "colab_type": "code",
        "outputId": "caebc59d-7d67-410e-ecb2-d5f02eb71156",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "print(dev_tokens[:5])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[['denver', 'broncos'], ['denver', 'broncos'], ['denver', 'broncos']], [['carolina', 'panthers'], ['carolina', 'panthers'], ['carolina', 'panthers']], [['santa', 'clara', ',', 'california'], ['levi', \"'\", 's', 'stadium'], ['levi', \"'\", 's', 'stadium', 'in', 'the', 'san', 'francisco', 'bay', 'area', 'at', 'santa', 'clara', ',', 'california', '.']], [['denver', 'broncos'], ['denver', 'broncos'], ['denver', 'broncos']], [['gold'], ['gold'], ['gold']]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJgBa_D4Q74V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 토큰화된 정답을 문장으로 변환시켜주고 합쳐줌\n",
        "dev_answer_lists = []\n",
        "for dev_answers in dev_tokens:\n",
        "  dev_answer_list = []\n",
        "  for dev_answer in dev_answers:\n",
        "    dev_answer_string = \" \".join(dev_answer)\n",
        "    dev_answer_list.append(dev_answer_string)\n",
        "  dev_answer_lists.append(dev_answer_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CJJl1_ilz2UO",
        "colab_type": "code",
        "outputId": "ece3cba9-b259-4548-daf2-23cba5917cfa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "print(dev_answer_lists[:5])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['denver broncos', 'denver broncos', 'denver broncos'], ['carolina panthers', 'carolina panthers', 'carolina panthers'], ['santa clara , california', \"levi ' s stadium\", \"levi ' s stadium in the san francisco bay area at santa clara , california .\"], ['denver broncos', 'denver broncos', 'denver broncos'], ['gold', 'gold', 'gold']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Umg8gLeoTx4-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# untokenizing\n",
        "dev_strings_end = []\n",
        "for dev_strings in dev_answer_lists:\n",
        "  dev_strings_processed = []\n",
        "  for dev_string in dev_strings:\n",
        "    dev_string = dev_string.replace(\" ##\", \"\")\n",
        "    dev_strings_processed.append(dev_string)\n",
        "  dev_strings_end.append(dev_strings_processed)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BztLUJVBUcjl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "dev_answers = dev_strings_end "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fihv0o-q0tF6",
        "colab_type": "code",
        "outputId": "eb1430e0-1d1a-40b7-dabb-a09937eca89f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "print(dev_answers[:5])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['denver broncos', 'denver broncos', 'denver broncos'], ['carolina panthers', 'carolina panthers', 'carolina panthers'], ['santa clara , california', \"levi ' s stadium\", \"levi ' s stadium in the san francisco bay area at santa clara , california .\"], ['denver broncos', 'denver broncos', 'denver broncos'], ['gold', 'gold', 'gold']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9zuuxDyd08yL",
        "colab_type": "text"
      },
      "source": [
        "F1 SCORE와 EXACT MATCH를 계산하는 함수 정의  \n",
        "출처 : https://github.com/allenai/bi-att-flow/blob/master/squad/evaluate-v1.1.py"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4oB5uqOUOagK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from collections import Counter\n",
        "import string\n",
        "\n",
        "def normalize_answer(s):\n",
        "    \"\"\"Lower text and remove punctuation, articles and extra whitespace.\"\"\"\n",
        "    def remove_articles(text):\n",
        "        return re.sub(r'\\b(a|an|the)\\b', ' ', text)\n",
        "\n",
        "    def white_space_fix(text):\n",
        "        return ' '.join(text.split())\n",
        "\n",
        "    def remove_punc(text):\n",
        "        exclude = set(string.punctuation)\n",
        "        return ''.join(ch for ch in text if ch not in exclude)\n",
        "\n",
        "    def lower(text):\n",
        "        return text.lower()\n",
        "\n",
        "    return white_space_fix(remove_articles(remove_punc(lower(s))))\n",
        "\n",
        "def f1_score(prediction, ground_truth):\n",
        "    prediction_tokens = normalize_answer(prediction).split()\n",
        "    ground_truth_tokens = normalize_answer(ground_truth).split()\n",
        "    common = Counter(prediction_tokens) & Counter(ground_truth_tokens)\n",
        "    num_same = sum(common.values())\n",
        "    if num_same == 0:\n",
        "        return 0\n",
        "    precision = 1.0 * num_same / len(prediction_tokens)\n",
        "    recall = 1.0 * num_same / len(ground_truth_tokens)\n",
        "    f1 = (2 * precision * recall) / (precision + recall)\n",
        "    return f1\n",
        "\n",
        "def exact_match_score(prediction, ground_truth):\n",
        "    return (normalize_answer(prediction) == normalize_answer(ground_truth))\n",
        "\n",
        "def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):\n",
        "    scores_for_ground_truths = []\n",
        "    for ground_truth in ground_truths:\n",
        "        score = metric_fn(prediction, ground_truth)\n",
        "        scores_for_ground_truths.append(score)\n",
        "    return max(scores_for_ground_truths)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VzRk3X0Kx9uF",
        "colab_type": "code",
        "outputId": "e6197a9d-7fda-41b2-d88e-3990a4b24229",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "f1_sum = 0\n",
        "\n",
        "for i in range(len(untokenized)):\n",
        "  f1 = metric_max_over_ground_truths(f1_score, untokenized[i], dev_answers[i])\n",
        "  f1_sum += f1\n",
        "print(\"f1 score : \", f1_sum / length)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "f1 score :  0.8743697019675727\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aAlyEE1oBXCc",
        "colab_type": "code",
        "outputId": "befa7595-a596-4497-c1ee-f7b4ce501209",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "EM_sum = 0\n",
        "\n",
        "for i in range(len(untokenized)):\n",
        "  \n",
        "  EM = metric_max_over_ground_truths(exact_match_score, untokenized[i], dev_answers[i])\n",
        "  EM_sum += EM\n",
        "print(\"EM Score : \", EM_sum / length)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "EM Score :  0.7902363123602062\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p-zIeFE289xr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def convert_data(data_df):\n",
        "    global tokenizer\n",
        "    indices, segments = [], []\n",
        "\n",
        "  \n",
        "    que, _ = tokenizer.encode(data_df[QUESTION_COLUMN])\n",
        "    doc, _ = tokenizer.encode(data_df[DATA_COLUMN])\n",
        "    doc.pop(0)\n",
        "\n",
        "    que_len = len(que)\n",
        "    doc_len = len(doc)\n",
        "\n",
        "    if que_len > 64:\n",
        "      que = que[:63]\n",
        "      que.append(102)\n",
        "        \n",
        "    if len(que+doc) > SEQ_LEN:\n",
        "      while len(que+doc) != SEQ_LEN:\n",
        "        doc.pop(-1)\n",
        "\n",
        "      doc.pop(-1)\n",
        "      doc.append(102)\n",
        "\n",
        "    segment = [0]*len(que) + [1]*len(doc) + [0]*(SEQ_LEN-len(que)-len(doc))\n",
        "    if len(que + doc) <= SEQ_LEN:\n",
        "      while len(que+doc) != SEQ_LEN:\n",
        "        doc.append(0)\n",
        "\n",
        "    ids = que + doc\n",
        "\n",
        "\n",
        "    indices.append(ids)\n",
        "    segments.append(segment)\n",
        "    indices = np.array(indices)\n",
        "    segments = np.array(segments)\n",
        "    return [indices, segments]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Engs2SIlW0SP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def predict_letter(df):\n",
        "  \n",
        "  test_input = convert_data(df)\n",
        "  test_start, test_end = bert_model.predict(test_input)\n",
        "  \n",
        "  indexes = test_input[0].tolist()[0]\n",
        "  start = np.argmax(test_start, axis=1).item()\n",
        "  end = np.argmax(test_end, axis=1).item()\n",
        "  start_tok = indexes[start]\n",
        "  end_tok = indexes[end]\n",
        "\n",
        "  print(\"Question : \", df['question'])\n",
        "  print(\"-\"*50)\n",
        "  print(\"Context : \", end = \" \")\n",
        "  \n",
        "  def split_text(text, n):\n",
        "    for line in text.splitlines():\n",
        "        while len(line) > n:\n",
        "           x, line = line[:n], line[n:]\n",
        "           yield x\n",
        "        yield line\n",
        "  \n",
        "  for line in split_text(df['context'], 150):\n",
        "    print(line)\n",
        "\n",
        "  print(\"-\"*50)\n",
        "  print(\"ANSWER : \", end = \" \")\n",
        "  print(\"\\n\")\n",
        "  sentences = []\n",
        "  \n",
        "  for i in range(start, end+1):\n",
        "    token_based_word = reverse_token_dict[indexes[i]]\n",
        "    sentences.append(token_based_word)\n",
        "    print(token_based_word, end= \" \")\n",
        "  \n",
        "  print(\"\\n\")\n",
        "  print(\"Untokenized Answer : \", end = \"\")\n",
        "  for w in sentences:\n",
        "    if w.startswith(\"##\"):\n",
        "      w = w.replace(\"##\", \"\")\n",
        "    else:\n",
        "      w = \" \" + w\n",
        "    \n",
        "    print(w, end=\"\")\n",
        "  print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lv0mXh6Z8Gnt",
        "colab_type": "text"
      },
      "source": [
        "임의의 Question과 Context를 받아서, 정답을 맞춰보는 것을 실험해보도록 하겠습니다.  \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IBmiDNInax0z",
        "colab_type": "code",
        "outputId": "f5b541b8-98b7-4050-f05c-ef7e0639ef7f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "import random\n",
        "for i in random.sample(range(300),300):\n",
        "  answers = dev['answers'][i]\n",
        "  predict_letter(dev.iloc[i])\n",
        "  print(\"\")\n",
        "  print(\"real answer : \", answers)\n",
        "  print(\"\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Question :  How many fumbles did Von Miller force?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}]\n",
            "\n",
            "Question :  How many fumbles did Von Miller force in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 314, 'text': '2'}, {'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}]\n",
            "\n",
            "Question :  What is the name of San Francisco's stadium when looked at as a possibility for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 153, 'text': \"Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium.\"}]\n",
            "\n",
            "Question :  What day was the game played on?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "february 7 , 2016 \n",
            "\n",
            "Untokenized Answer :  february 7 , 2016\n",
            "\n",
            "real answer :  [{'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7'}, {'answer_start': 334, 'text': 'February 7, 2016'}]\n",
            "\n",
            "Question :  How many times have the Panthers been in the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "eight \n",
            "\n",
            "Untokenized Answer :  eight\n",
            "\n",
            "real answer :  [{'answer_start': 345, 'text': '2'}, {'answer_start': 227, 'text': 'second'}, {'answer_start': 227, 'text': 'second'}]\n",
            "\n",
            "Question :  What year did Roger Goodell announce that Super Bowl 50 would be \"important\"?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2012 \n",
            "\n",
            "Untokenized Answer :  2012\n",
            "\n",
            "real answer :  [{'answer_start': 9, 'text': '2012'}, {'answer_start': 9, 'text': '2012'}, {'answer_start': 9, 'text': '2012'}]\n",
            "\n",
            "Question :  How many yards did Newton throw for in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "3 , 83 ##7 \n",
            "\n",
            "Untokenized Answer :  3 , 837\n",
            "\n",
            "real answer :  [{'answer_start': 197, 'text': '3,837'}, {'answer_start': 197, 'text': '3,837'}, {'answer_start': 197, 'text': '3,837'}]\n",
            "\n",
            "Question :  What year was Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 season . the american football conference ( afc ) champion denver broncos defeated the national football conference ( nfc ) champion carolina panthers 24 – 10 to earn their third super bowl title . the game was played on february 7 , 2016 \n",
            "\n",
            "Untokenized Answer :  2015 season . the american football conference ( afc ) champion denver broncos defeated the national football conference ( nfc ) champion carolina panthers 24 – 10 to earn their third super bowl title . the game was played on february 7 , 2016\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 346, 'text': '2016'}, {'answer_start': 346, 'text': '2016'}]\n",
            "\n",
            "Question :  What position does Jerricho Cotchery play?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "quarterback cam newton had one of his best seasons , throwing for 3 , 83 ##7 yards and rushing for 63 ##6 , while recording a career - high and league - leading 45 total touchdowns ( 35 passing , 10 rushing ) , a career - low 10 interceptions , and a career - best quarterback rating of 99 . 4 . newton ' s leading receivers were tight end greg olsen , who caught a career - high 77 passes for 1 , 104 yards and seven touchdowns , and wide receiver ted gin ##n , jr . , who caught 44 passes for 73 ##9 yards and 10 touchdowns ; gin ##n also rushed for 60 yards and returned 27 punt ##s for 277 yards . other key receivers \n",
            "\n",
            "Untokenized Answer :  quarterback cam newton had one of his best seasons , throwing for 3 , 837 yards and rushing for 636 , while recording a career - high and league - leading 45 total touchdowns ( 35 passing , 10 rushing ) , a career - low 10 interceptions , and a career - best quarterback rating of 99 . 4 . newton ' s leading receivers were tight end greg olsen , who caught a career - high 77 passes for 1 , 104 yards and seven touchdowns , and wide receiver ted ginn , jr . , who caught 44 passes for 739 yards and 10 touchdowns ; ginn also rushed for 60 yards and returned 27 punts for 277 yards . other key receivers\n",
            "\n",
            "real answer :  [{'answer_start': 419, 'text': 'receivers'}, {'answer_start': 693, 'text': 'receivers'}, {'answer_start': 693, 'text': 'receivers'}]\n",
            "\n",
            "Question :  What site is located in the San Francisco Bay Area?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 153, 'text': \"Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium\"}]\n",
            "\n",
            "Question :  Which team did not get a chance to defend their Super Bowl XLIX win in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new england patriots \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  Which NFL team represented the AFC at Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  What station aired the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cbs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  Who did Carolina beat in the NFC championship game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  Who was the quarterback for the Broncos 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "peyton manning \n",
            "\n",
            "Untokenized Answer :  peyton manning\n",
            "\n",
            "real answer :  [{'answer_start': 442, 'text': 'Peyton Manning'}, {'answer_start': 442, 'text': 'Peyton Manning'}, {'answer_start': 566, 'text': 'Manning'}]\n",
            "\n",
            "Question :  What team had the best start ever in the NFL?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "carolina panthers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 137, 'text': 'Carolina Panthers'}, {'answer_start': 695, 'text': 'the Panthers'}, {'answer_start': 330, 'text': 'Carolina'}]\n",
            "\n",
            "Question :  Who is Denver's defensive coordinator?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "wade phillips \n",
            "\n",
            "Untokenized Answer :  wade phillips\n",
            "\n",
            "real answer :  [{'answer_start': 1640, 'text': 'Wade Phillips'}, {'answer_start': 1640, 'text': 'Wade Phillips'}, {'answer_start': 1645, 'text': 'Phillips'}]\n",
            "\n",
            "Question :  Who was the NFL Commissioner in early 2012?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "roger good ##ell \n",
            "\n",
            "Untokenized Answer :  roger goodell\n",
            "\n",
            "real answer :  [{'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 38, 'text': 'Goodell'}]\n",
            "\n",
            "Question :  In early 2012, Goodell said that Super Bowl 50 would be what?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "spectacular \n",
            "\n",
            "Untokenized Answer :  spectacular\n",
            "\n",
            "real answer :  [{'answer_start': 106, 'text': 'spectacular'}, {'answer_start': 141, 'text': 'an important game for us as a league'}, {'answer_start': 106, 'text': 'spectacular'}]\n",
            "\n",
            "Question :  How much did it cost to build Levi's Stadium?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 1 . 2 billion \n",
            "\n",
            "Untokenized Answer :  $ 1 . 2 billion\n",
            "\n",
            "real answer :  [{'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}]\n",
            "\n",
            "Question :  How many tackles did Luke Kuechly register?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "118 \n",
            "\n",
            "Untokenized Answer :  118\n",
            "\n",
            "real answer :  [{'answer_start': 789, 'text': '118'}, {'answer_start': 789, 'text': '118'}, {'answer_start': 789, 'text': '118'}]\n",
            "\n",
            "Question :  The name of the NFL championship game is?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl 50 \n",
            "\n",
            "Untokenized Answer :  super bowl 50\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'Super Bowl'}, {'answer_start': 0, 'text': 'Super Bowl'}, {'answer_start': 0, 'text': 'Super Bowl'}]\n",
            "\n",
            "Question :  Who was limited by Denver's defense?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "newton \n",
            "\n",
            "Untokenized Answer :  newton\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': \"Newton was limited by Denver's defense\"}, {'answer_start': 67, 'text': 'Newton'}, {'answer_start': 67, 'text': 'Newton'}]\n",
            "\n",
            "Question :  Who registered the most sacks on the team this season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ka ##wan ##n short \n",
            "\n",
            "Untokenized Answer :  kawann short\n",
            "\n",
            "real answer :  [{'answer_start': 192, 'text': 'Kawann Short'}, {'answer_start': 192, 'text': 'Kawann Short'}, {'answer_start': 199, 'text': 'Short'}]\n",
            "\n",
            "Question :  Who was the Super Bowl 50 MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  Who was first on the team in total tackles?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brandon marshall \n",
            "\n",
            "Untokenized Answer :  brandon marshall\n",
            "\n",
            "real answer :  [{'answer_start': 458, 'text': 'Brandon Marshall'}, {'answer_start': 458, 'text': 'Brandon Marshall'}, {'answer_start': 466, 'text': 'Marshall'}]\n",
            "\n",
            "Question :  How many Panthers went to the Pro Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ten \n",
            "\n",
            "Untokenized Answer :  ten\n",
            "\n",
            "real answer :  [{'answer_start': 807, 'text': 'Ten'}, {'answer_start': 807, 'text': 'Ten'}, {'answer_start': 807, 'text': 'Ten'}]\n",
            "\n",
            "Question :  What New Orleans stadium was considered for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "mercedes - benz super ##dome \n",
            "\n",
            "Untokenized Answer :  mercedes - benz superdome\n",
            "\n",
            "real answer :  [{'answer_start': 69, 'text': 'Mercedes-Benz Superdome'}, {'answer_start': 69, 'text': 'Mercedes-Benz Superdome'}, {'answer_start': 69, 'text': 'Mercedes-Benz Superdome,'}]\n",
            "\n",
            "Question :  Who is the quarterback for the Carolina Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "mike to ##lbert \n",
            "\n",
            "Untokenized Answer :  mike tolbert\n",
            "\n",
            "real answer :  [{'answer_start': 144, 'text': 'Cam Newton'}, {'answer_start': 144, 'text': 'Cam Newton'}, {'answer_start': 144, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  Which performers joined the headliner during the Super Bowl 50 halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}]\n",
            "\n",
            "Question :  In what year did the NFL switch to a 16-game regular season?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1978 \n",
            "\n",
            "Untokenized Answer :  1978\n",
            "\n",
            "real answer :  [{'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}]\n",
            "\n",
            "Question :  How old was Manning at the beginning of the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "39 \n",
            "\n",
            "Untokenized Answer :  39\n",
            "\n",
            "real answer :  [{'answer_start': 817, 'text': '39'}, {'answer_start': 817, 'text': '39'}, {'answer_start': 817, 'text': '39'}]\n",
            "\n",
            "Question :  How many touchdowns did Newton have in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "45 \n",
            "\n",
            "Untokenized Answer :  45\n",
            "\n",
            "real answer :  [{'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}]\n",
            "\n",
            "Question :  Which player did the Panthers lose to an ACL injury in a preseason game?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "kelvin benjamin \n",
            "\n",
            "Untokenized Answer :  kelvin benjamin\n",
            "\n",
            "real answer :  [{'answer_start': 85, 'text': 'Kelvin Benjamin'}, {'answer_start': 85, 'text': 'Kelvin Benjamin'}, {'answer_start': 92, 'text': 'Benjamin'}]\n",
            "\n",
            "Question :  How many times did the Broncos cause turnovers in the game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}]\n",
            "\n",
            "Question :  What was the average cost of a 30-second commercial?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  When was the last time California hosted a Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2003 \n",
            "\n",
            "Untokenized Answer :  2003\n",
            "\n",
            "real answer :  [{'answer_start': 321, 'text': '2003.'}, {'answer_start': 321, 'text': '2003'}, {'answer_start': 321, 'text': '2003'}]\n",
            "\n",
            "Question :  Who led the Panthers in sacks?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ka ##wan ##n short \n",
            "\n",
            "Untokenized Answer :  kawann short\n",
            "\n",
            "real answer :  [{'answer_start': 192, 'text': 'Kawann Short'}, {'answer_start': 192, 'text': 'Kawann Short'}, {'answer_start': 199, 'text': 'Short'}]\n",
            "\n",
            "Question :  Prior to Super Bowl 50, what was the last Super Bowl in California?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xx ##x ##vi ##i \n",
            "\n",
            "Untokenized Answer :  super bowl xxxvii\n",
            "\n",
            "real answer :  [{'answer_start': 276, 'text': 'Super Bowl XXXVII'}, {'answer_start': 276, 'text': 'Super Bowl XXXVII'}, {'answer_start': 287, 'text': 'XXXVII'}]\n",
            "\n",
            "Question :  Where was the meeting held when the NFL owners voted on the location for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "boston \n",
            "\n",
            "Untokenized Answer :  boston\n",
            "\n",
            "real answer :  [{'answer_start': 56, 'text': 'Boston'}, {'answer_start': 53, 'text': 'in Boston'}, {'answer_start': 56, 'text': 'Boston'}]\n",
            "\n",
            "Question :  How many points did the Panthers defense surrender?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "308 \n",
            "\n",
            "Untokenized Answer :  308\n",
            "\n",
            "real answer :  [{'answer_start': 34, 'text': '308'}, {'answer_start': 34, 'text': '308'}, {'answer_start': 34, 'text': '308'}]\n",
            "\n",
            "Question :  Who lost to the Panthers in the divisional round of the playoffs?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers beat the Seattle Seahawks in the divisional round, running up a 31–0 halftime lead and then holding off a furious second half comeback at\n",
            "tempt to win 31–24, avenging their elimination from a year earlier. The Panthers then blew out the Arizona Cardinals in the NFC Championship Game, 49–\n",
            "15, racking up 487 yards and forcing seven turnovers.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seattle seahawks \n",
            "\n",
            "Untokenized Answer :  seattle seahawks\n",
            "\n",
            "real answer :  [{'answer_start': 22, 'text': 'Seattle Seahawks'}, {'answer_start': 18, 'text': 'the Seattle Seahawks'}, {'answer_start': 30, 'text': 'Seahawks'}]\n",
            "\n",
            "Question :  What band headlined half-time during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  How many total touchdowns did Cam Newton score?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "45 \n",
            "\n",
            "Untokenized Answer :  45\n",
            "\n",
            "real answer :  [{'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}]\n",
            "\n",
            "Question :  Who did Denver beat in the 2015 AFC Championship game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  Who had five sacks in nine games as a Carolina Panthers starter?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ko ##ny ea ##ly \n",
            "\n",
            "Untokenized Answer :  kony ealy\n",
            "\n",
            "real answer :  [{'answer_start': 500, 'text': 'Kony Ealy'}, {'answer_start': 500, 'text': 'Kony Ealy'}]\n",
            "\n",
            "Question :  Prior to this consideration, when did San Francisco last host a Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1985 \n",
            "\n",
            "Untokenized Answer :  1985\n",
            "\n",
            "real answer :  [{'answer_start': 309, 'text': '1985'}, {'answer_start': 309, 'text': '1985'}, {'answer_start': 309, 'text': '1985'}]\n",
            "\n",
            "Question :  How many yards did the Broncos' defense give up?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "4 , 530 \n",
            "\n",
            "Untokenized Answer :  4 , 530\n",
            "\n",
            "real answer :  [{'answer_start': 60, 'text': '4,530'}, {'answer_start': 60, 'text': '4,530'}, {'answer_start': 60, 'text': '4,530'}]\n",
            "\n",
            "Question :  Which Louisiana venue was one of three considered for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new orleans ' mercedes - benz super ##dome , miami ' s sun life stadium , and the san francisco bay area ' s levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  new orleans ' mercedes - benz superdome , miami ' s sun life stadium , and the san francisco bay area ' s levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 56, 'text': \"New Orleans' Mercedes-Benz Superdome\"}, {'answer_start': 56, 'text': \"New Orleans' Mercedes-Benz Superdome\"}, {'answer_start': 69, 'text': 'Mercedes-Benz Superdome'}]\n",
            "\n",
            "Question :  What was the passer rating for Peyton Manning that he finished out with for the season? \n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "67 . 9 \n",
            "\n",
            "Untokenized Answer :  67 . 9\n",
            "\n",
            "real answer :  [{'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}]\n",
            "\n",
            "Question :  What Denver player caught the ball 76 times in the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "emmanuel sanders \n",
            "\n",
            "Untokenized Answer :  emmanuel sanders\n",
            "\n",
            "real answer :  [{'answer_start': 345, 'text': 'Emmanuel Sanders'}, {'answer_start': 345, 'text': 'Emmanuel Sanders'}, {'answer_start': 354, 'text': 'Sanders'}]\n",
            "\n",
            "Question :  What were the win/loss game stats for the Denver Bronco's regular season in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "12 – 4 record \n",
            "\n",
            "Untokenized Answer :  12 – 4 record\n",
            "\n",
            "real answer :  [{'answer_start': 344, 'text': '12–4'}, {'answer_start': 344, 'text': '12–4'}, {'answer_start': 344, 'text': '12–4'}]\n",
            "\n",
            "Question :  Who is the head coach of the Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gary ku ##bia ##k \n",
            "\n",
            "Untokenized Answer :  gary kubiak\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 332, 'text': 'Kubiak'}]\n",
            "\n",
            "Question :  What year was the Carolina Panthers franchise founded?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1995 \n",
            "\n",
            "Untokenized Answer :  1995\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': '1995'}, {'answer_start': 291, 'text': '1995'}, {'answer_start': 291, 'text': '1995'}]\n",
            "\n",
            "Question :  Who is the Commissioner of the National Football League?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "roger good ##ell \n",
            "\n",
            "Untokenized Answer :  roger goodell\n",
            "\n",
            "real answer :  [{'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 38, 'text': 'Goodell'}]\n",
            "\n",
            "Question :  What day was the Super Bowl played on?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "february 7 , 2016 \n",
            "\n",
            "Untokenized Answer :  february 7 , 2016\n",
            "\n",
            "real answer :  [{'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7'}, {'answer_start': 334, 'text': 'February 7, 2016'}]\n",
            "\n",
            "Question :  Who is the quarterback for the Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam newton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  What year did the Denver Broncos secure a Super Bowl title for the third time?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2016 \n",
            "\n",
            "Untokenized Answer :  2016\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 346, 'text': '2016'}, {'answer_start': 116, 'text': '2015'}]\n",
            "\n",
            "Question :  How many times did the Denver defense force Newton into turnovers?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven times and forced him into three \n",
            "\n",
            "Untokenized Answer :  seven times and forced him into three\n",
            "\n",
            "real answer :  [{'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}]\n",
            "\n",
            "Question :  What was the given name of Miami's stadium at the time of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "sun life stadium \n",
            "\n",
            "Untokenized Answer :  sun life stadium\n",
            "\n",
            "real answer :  [{'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}]\n",
            "\n",
            "Question :  How many touchdowns did Brock Osweiler throw in the season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "10 \n",
            "\n",
            "Untokenized Answer :  10\n",
            "\n",
            "real answer :  [{'answer_start': 178, 'text': '10'}, {'answer_start': 178, 'text': '10'}, {'answer_start': 178, 'text': '10'}]\n",
            "\n",
            "Question :  Who was the Broncos' leading rusher for the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c . j . anderson \n",
            "\n",
            "Untokenized Answer :  c . j . anderson\n",
            "\n",
            "real answer :  [{'answer_start': 572, 'text': 'C. J. Anderson'}, {'answer_start': 572, 'text': 'C. J. Anderson'}, {'answer_start': 578, 'text': 'Anderson'}]\n",
            "\n",
            "Question :  What Denver player had 109 tackels for the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brandon marshall \n",
            "\n",
            "Untokenized Answer :  brandon marshall\n",
            "\n",
            "real answer :  [{'answer_start': 447, 'text': 'Linebacker Brandon Marshall'}, {'answer_start': 458, 'text': 'Brandon Marshall'}, {'answer_start': 466, 'text': 'Marshall'}]\n",
            "\n",
            "Question :  What is the position Derek Wolfe plays currently? \n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "defensive ends \n",
            "\n",
            "Untokenized Answer :  defensive ends\n",
            "\n",
            "real answer :  [{'answer_start': 144, 'text': 'Defensive ends'}, {'answer_start': 144, 'text': 'Defensive ends'}, {'answer_start': 144, 'text': 'Defensive ends'}]\n",
            "\n",
            "Question :  How many times was Cam Newton sacked in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven \n",
            "\n",
            "Untokenized Answer :  seven\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}]\n",
            "\n",
            "Question :  How many balls did Josh Norman intercept?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "88 \n",
            "\n",
            "Untokenized Answer :  88\n",
            "\n",
            "real answer :  [{'answer_start': 140, 'text': 'four'}, {'answer_start': 1104, 'text': 'four'}, {'answer_start': 1104, 'text': 'four'}]\n",
            "\n",
            "Question :  How many receptions did Cotchery  get for the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "39 \n",
            "\n",
            "Untokenized Answer :  39\n",
            "\n",
            "real answer :  [{'answer_start': 588, 'text': '39'}, {'answer_start': 739, 'text': '39'}, {'answer_start': 739, 'text': '39'}]\n",
            "\n",
            "Question :  How many teams have played in the Super Bowl eight times?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 170, 'text': '4'}, {'answer_start': 575, 'text': 'four'}, {'answer_start': 575, 'text': 'four'}]\n",
            "\n",
            "Question :  What performer lead the Super Bowl XLVIII halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  In what year was the Super Bowl last held in the Miami/South Florida area?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2010 \n",
            "\n",
            "Untokenized Answer :  2010\n",
            "\n",
            "real answer :  [{'answer_start': 261, 'text': '2010'}, {'answer_start': 261, 'text': '2010'}, {'answer_start': 261, 'text': '2010'}]\n",
            "\n",
            "Question :  How many Panthers defense players were selected for the Pro Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 140, 'text': 'four'}, {'answer_start': 140, 'text': 'four'}, {'answer_start': 140, 'text': 'four'}]\n",
            "\n",
            "Question :  How many forced fumbles did Thomas Davis have?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 140, 'text': 'four'}, {'answer_start': 695, 'text': '5½'}, {'answer_start': 705, 'text': 'four'}]\n",
            "\n",
            "Question :  Which team won Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  Who was the male singer who performed as a special guest during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 245, 'text': 'Bruno Mars,'}]\n",
            "\n",
            "Question :  How many interceptions did Aqib Talib have for the regular season?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 302, 'text': 'three.'}, {'answer_start': 585, 'text': 'three'}, {'answer_start': 585, 'text': 'three'}]\n",
            "\n",
            "Question :  How many touchdowns did Ronnie Hillman make?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 290, 'text': '5'}, {'answer_start': 761, 'text': 'five'}, {'answer_start': 761, 'text': 'five'}]\n",
            "\n",
            "Question :  In what venue did Super Bowl XIX take place?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "stanford stadium \n",
            "\n",
            "Untokenized Answer :  stanford stadium\n",
            "\n",
            "real answer :  [{'answer_start': 340, 'text': 'Stanford Stadium'}, {'answer_start': 340, 'text': 'Stanford Stadium'}, {'answer_start': 340, 'text': 'Stanford Stadium'}]\n",
            "\n",
            "Question :  How many teams up to Super Bowl 50 have been to the championship game eight times?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 300, 'text': 'four'}, {'answer_start': 300, 'text': 'four'}, {'answer_start': 300, 'text': 'four'}]\n",
            "\n",
            "Question :  Who did the Panthers beat to become the NFC champs?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  What team did the Panthers defeat?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  Which Carolina Panthers wide receiver suffered a torn ACL before the season began?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "kelvin benjamin \n",
            "\n",
            "Untokenized Answer :  kelvin benjamin\n",
            "\n",
            "real answer :  [{'answer_start': 85, 'text': 'Kelvin Benjamin'}, {'answer_start': 85, 'text': 'Kelvin Benjamin'}, {'answer_start': 92, 'text': 'Benjamin'}]\n",
            "\n",
            "Question :  When were the finalists announced?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "october 16 , 2012 \n",
            "\n",
            "Untokenized Answer :  october 16 , 2012\n",
            "\n",
            "real answer :  [{'answer_start': 24, 'text': 'October 16, 2012'}, {'answer_start': 24, 'text': 'October 16, 2012'}, {'answer_start': 24, 'text': 'October 16, 2012,'}]\n",
            "\n",
            "Question :  What position does Brandon Marshall currently play? \n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "linebacker \n",
            "\n",
            "Untokenized Answer :  linebacker\n",
            "\n",
            "real answer :  [{'answer_start': 309, 'text': 'Linebacker'}, {'answer_start': 447, 'text': 'Linebacker'}, {'answer_start': 447, 'text': 'Linebacker'}]\n",
            "\n",
            "Question :  How many picks did Peyton Manning throw in the season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2 , 249 yards and nine touchdowns \n",
            "\n",
            "Untokenized Answer :  2 , 249 yards and nine touchdowns\n",
            "\n",
            "real answer :  [{'answer_start': 115, 'text': '17'}, {'answer_start': 115, 'text': '17'}, {'answer_start': 115, 'text': '17'}]\n",
            "\n",
            "Question :  How many Offensive players from the Panthers were selected to play in the Pro Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "six \n",
            "\n",
            "Untokenized Answer :  six\n",
            "\n",
            "real answer :  [{'answer_start': 98, 'text': 'six'}, {'answer_start': 98, 'text': 'six'}, {'answer_start': 98, 'text': 'six'}]\n",
            "\n",
            "Question :  How many Pro Bowlers were on the Panthers offense?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "six \n",
            "\n",
            "Untokenized Answer :  six\n",
            "\n",
            "real answer :  [{'answer_start': 98, 'text': 'six'}, {'answer_start': 98, 'text': 'six'}, {'answer_start': 98, 'text': 'six'}]\n",
            "\n",
            "Question :  How much did Levi's Stadium cost?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 1 . 2 billion \n",
            "\n",
            "Untokenized Answer :  $ 1 . 2 billion\n",
            "\n",
            "real answer :  [{'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}]\n",
            "\n",
            "Question :  Which linebacker had the most sacks on the team?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brandon marshall \n",
            "\n",
            "Untokenized Answer :  brandon marshall\n",
            "\n",
            "real answer :  [{'answer_start': 228, 'text': 'Von Miller'}, {'answer_start': 228, 'text': 'Von Miller'}, {'answer_start': 232, 'text': 'Miller'}]\n",
            "\n",
            "Question :  Who was the broadcaster for Super Bowl 50 in the United States?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cbs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  how many yards did Newton get for passes in the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "3 , 83 ##7 \n",
            "\n",
            "Untokenized Answer :  3 , 837\n",
            "\n",
            "real answer :  [{'answer_start': 197, 'text': '3,837'}, {'answer_start': 197, 'text': '3,837'}, {'answer_start': 197, 'text': '3,837'}]\n",
            "\n",
            "Question :  Who was the head coach of the Broncos in Super Bowl XLVIII?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "john fox \n",
            "\n",
            "Untokenized Answer :  john fox\n",
            "\n",
            "real answer :  [{'answer_start': 636, 'text': 'John Fox'}, {'answer_start': 636, 'text': 'John Fox'}, {'answer_start': 641, 'text': 'Fox'}]\n",
            "\n",
            "Question :  How many teams can boast a 15–1 regular season record?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "six \n",
            "\n",
            "Untokenized Answer :  six\n",
            "\n",
            "real answer :  [{'answer_start': 219, 'text': 'six'}, {'answer_start': 138, 'text': 'ten'}, {'answer_start': 138, 'text': 'ten'}]\n",
            "\n",
            "Question :  What year did the league begin having schedules with 16 games in them?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1978 \n",
            "\n",
            "Untokenized Answer :  1978\n",
            "\n",
            "real answer :  [{'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}]\n",
            "\n",
            "Question :  Which NFL team won Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  Who did Denver beat in the AFC championship?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  Who is the commissioner of the NFL?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "roger good ##ell \n",
            "\n",
            "Untokenized Answer :  roger goodell\n",
            "\n",
            "real answer :  [{'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 38, 'text': 'Goodell'}]\n",
            "\n",
            "Question :  Who was the 2015 NFL MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam newton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  What team was the NFC champion?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "carolina panthers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}]\n",
            "\n",
            "Question :  How many appearances have the Denver Broncos made in the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "eight \n",
            "\n",
            "Untokenized Answer :  eight\n",
            "\n",
            "real answer :  [{'answer_start': 467, 'text': '8'}, {'answer_start': 601, 'text': 'eight'}, {'answer_start': 601, 'text': 'eight'}]\n",
            "\n",
            "Question :  Who broadcast the Super Bowl on TV?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cbs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What Florida stadium was considered for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}]\n",
            "\n",
            "Question :  Which linebacker led the Broncos in tackles?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brandon marshall \n",
            "\n",
            "Untokenized Answer :  brandon marshall\n",
            "\n",
            "real answer :  [{'answer_start': 458, 'text': 'Brandon Marshall'}, {'answer_start': 458, 'text': 'Brandon Marshall'}, {'answer_start': 466, 'text': 'Marshall'}]\n",
            "\n",
            "Question :  If Roman numerals were used, what would Super Bowl 50 have been called?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl l \n",
            "\n",
            "Untokenized Answer :  super bowl l\n",
            "\n",
            "real answer :  [{'answer_start': 693, 'text': 'Super Bowl L'}, {'answer_start': 704, 'text': 'L'}, {'answer_start': 693, 'text': 'Super Bowl L'}]\n",
            "\n",
            "Question :  What is the AFC short for?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}]\n",
            "\n",
            "Question :  Who had the best record in the NFC?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "carolina panthers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 137, 'text': 'Carolina Panthers'}, {'answer_start': 695, 'text': 'the Panthers'}, {'answer_start': 330, 'text': 'Carolina'}]\n",
            "\n",
            "Question :  When did the San Francisco Bay area last host the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1985 \n",
            "\n",
            "Untokenized Answer :  1985\n",
            "\n",
            "real answer :  [{'answer_start': 309, 'text': '1985'}, {'answer_start': 309, 'text': '1985'}, {'answer_start': 309, 'text': '1985'}]\n",
            "\n",
            "Question :  When did the Packers arrive at a record of 13-0?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2011 \n",
            "\n",
            "Untokenized Answer :  2011\n",
            "\n",
            "real answer :  [{'answer_start': 623, 'text': '2011'}, {'answer_start': 623, 'text': '2011'}, {'answer_start': 623, 'text': '2011'}]\n",
            "\n",
            "Question :  In 2012, how many stadiums were named as finalists for hosting Super Bowl 50 before the final stadium was chosen?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 51, 'text': 'two'}, {'answer_start': 51, 'text': 'two'}, {'answer_start': 51, 'text': 'two'}]\n",
            "\n",
            "Question :  Which network broadcasted Super Bowl 50 in the U.S.?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cbs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What injury did the Carolina Panthers lose Kelvin Benjamin to during their preseason?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "torn ac ##l \n",
            "\n",
            "Untokenized Answer :  torn acl\n",
            "\n",
            "real answer :  [{'answer_start': 106, 'text': 'torn ACL'}, {'answer_start': 104, 'text': 'a torn ACL'}, {'answer_start': 106, 'text': 'torn ACL'}]\n",
            "\n",
            "Question :  How many NFL teams have had only one loss by the end of a regular season?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ten \n",
            "\n",
            "Untokenized Answer :  ten\n",
            "\n",
            "real answer :  [{'answer_start': 138, 'text': 'ten'}, {'answer_start': 138, 'text': 'ten'}, {'answer_start': 138, 'text': 'ten'}]\n",
            "\n",
            "Question :  What foot was injured on Manning that sidelined him in week 10?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "left \n",
            "\n",
            "Untokenized Answer :  left\n",
            "\n",
            "real answer :  [{'answer_start': 993, 'text': 'left foot.'}, {'answer_start': 993, 'text': 'left'}, {'answer_start': 993, 'text': 'left'}]\n",
            "\n",
            "Question :  How many years was John Fox the head coach of the Denver Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 201, 'text': 'four'}, {'answer_start': 238, 'text': 'four'}, {'answer_start': 238, 'text': 'four'}]\n",
            "\n",
            "Question :  How many teams have had a 15-1 record for the regular season?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "six \n",
            "\n",
            "Untokenized Answer :  six\n",
            "\n",
            "real answer :  [{'answer_start': 219, 'text': 'six'}, {'answer_start': 138, 'text': 'ten'}, {'answer_start': 138, 'text': 'ten'}]\n",
            "\n",
            "Question :  Who were the defending Super Bowl champions?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new england patriots \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  Who were special guests for the Super Bowl halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}]\n",
            "\n",
            "Question :  Which Panthers RB scored 6 TDs in the 13 games leading up to Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "jonathan stewart \n",
            "\n",
            "Untokenized Answer :  jonathan stewart\n",
            "\n",
            "real answer :  [{'answer_start': 964, 'text': 'Jonathan Stewart'}, {'answer_start': 964, 'text': 'Jonathan Stewart'}, {'answer_start': 964, 'text': 'Jonathan Stewart'}]\n",
            "\n",
            "Question :  In the 10th week of the 2015 season, what injury was Peyton Manning dealing with?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "partial tear of the plant ##ar fa ##sc ##ii ##tis in his left foot \n",
            "\n",
            "Untokenized Answer :  partial tear of the plantar fasciitis in his left foot\n",
            "\n",
            "real answer :  [{'answer_start': 678, 'text': 'plantar fasciitis'}, {'answer_start': 946, 'text': 'a partial tear of the plantar fasciitis'}]\n",
            "\n",
            "Question :  What was the last Super Bowl that took place at Sun Life Stadium in Miami? \n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xi ##x \n",
            "\n",
            "Untokenized Answer :  super bowl xix\n",
            "\n",
            "real answer :  [{'answer_start': 242, 'text': 'Super Bowl XLIV'}, {'answer_start': 242, 'text': 'Super Bowl XLIV'}, {'answer_start': 261, 'text': '2010'}]\n",
            "\n",
            "Question :  Which Denver linebacker was named Super Bowl MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 237, 'text': 'linebacker Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  Which Super Bowl did Roger Goodell speak about?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "the 50th \n",
            "\n",
            "Untokenized Answer :  the 50th\n",
            "\n",
            "real answer :  [{'answer_start': 85, 'text': 'the 50th Super Bowl'}, {'answer_start': 85, 'text': 'the 50th'}, {'answer_start': 89, 'text': '50th'}]\n",
            "\n",
            "Question :  When was Levi's Stadium awarded the right to host Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "may 21 , 2013 \n",
            "\n",
            "Untokenized Answer :  may 21 , 2013\n",
            "\n",
            "real answer :  [{'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013,'}]\n",
            "\n",
            "Question :  What color was used to emphasize the 50th anniversary of the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gold \n",
            "\n",
            "Untokenized Answer :  gold\n",
            "\n",
            "real answer :  [{'answer_start': 488, 'text': 'gold'}, {'answer_start': 488, 'text': 'gold'}, {'answer_start': 521, 'text': 'gold'}]\n",
            "\n",
            "Question :  What was the number of solo tackles that Von Miller had in Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 45, 'text': '5'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  Before Super Bowl 50, what was the coach's name that coached both teams for their last Super Bowl appearances?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "john fox \n",
            "\n",
            "Untokenized Answer :  john fox\n",
            "\n",
            "real answer :  [{'answer_start': 636, 'text': 'John Fox'}, {'answer_start': 636, 'text': 'John Fox'}, {'answer_start': 636, 'text': 'John Fox'}]\n",
            "\n",
            "Question :  What was the entity that stepped in and caused Miami's Sun Life Stadium to no longer be in the running to host Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "the florida legislature \n",
            "\n",
            "Untokenized Answer :  the florida legislature\n",
            "\n",
            "real answer :  [{'answer_start': 508, 'text': 'Florida legislature'}, {'answer_start': 504, 'text': 'the Florida legislature'}, {'answer_start': 508, 'text': 'Florida legislature'}]\n",
            "\n",
            "Question :  When was the last time San Francisco hosted a Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1985 \n",
            "\n",
            "Untokenized Answer :  1985\n",
            "\n",
            "real answer :  [{'answer_start': 309, 'text': '1985'}, {'answer_start': 309, 'text': '1985'}, {'answer_start': 309, 'text': '1985'}]\n",
            "\n",
            "Question :  What was the theme of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "american football game to determine the champion of the national football league ( nfl ) for the 2015 season \n",
            "\n",
            "Untokenized Answer :  american football game to determine the champion of the national football league ( nfl ) for the 2015 season\n",
            "\n",
            "real answer :  [{'answer_start': 487, 'text': '\"golden anniversary\"'}, {'answer_start': 521, 'text': 'gold-themed'}, {'answer_start': 487, 'text': '\"golden anniversary'}]\n",
            "\n",
            "Question :  What other two famous performers were part of the Super Bowl 50 halftime?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}]\n",
            "\n",
            "Question :  What was Ronnie Hillman's average yards per carry in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "4 . 7 \n",
            "\n",
            "Untokenized Answer :  4 . 7\n",
            "\n",
            "real answer :  [{'answer_start': 799, 'text': '4.7'}, {'answer_start': 799, 'text': '4.7'}, {'answer_start': 799, 'text': '4.7'}]\n",
            "\n",
            "Question :  How many points did Carolina lead the NFL in scoring for offensive plays?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "500 \n",
            "\n",
            "Untokenized Answer :  500\n",
            "\n",
            "real answer :  [{'answer_start': 52, 'text': '500'}, {'answer_start': 52, 'text': '500'}, {'answer_start': 52, 'text': '500'}]\n",
            "\n",
            "Question :  When did Levi's Stadium open?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2014 \n",
            "\n",
            "Untokenized Answer :  2014\n",
            "\n",
            "real answer :  [{'answer_start': 144, 'text': '2014.'}, {'answer_start': 141, 'text': 'in 2014'}, {'answer_start': 144, 'text': '2014'}]\n",
            "\n",
            "Question :  How many turnovers did Cam Newton have?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}, {'answer_start': 156, 'text': 'three'}]\n",
            "\n",
            "Question :  When was the most recent Super Bowl hosted in the South Florida/Miami area?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2010 \n",
            "\n",
            "Untokenized Answer :  2010\n",
            "\n",
            "real answer :  [{'answer_start': 261, 'text': '2010'}, {'answer_start': 261, 'text': '2010'}, {'answer_start': 261, 'text': '2010'}]\n",
            "\n",
            "Question :  How many touchdowns did Newton get in the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "45 \n",
            "\n",
            "Untokenized Answer :  45\n",
            "\n",
            "real answer :  [{'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}]\n",
            "\n",
            "Question :  How may yards did Peyton Manning throw?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2 , 249 \n",
            "\n",
            "Untokenized Answer :  2 , 249\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': '2,249'}, {'answer_start': 77, 'text': '2,249'}, {'answer_start': 77, 'text': '2,249'}]\n",
            "\n",
            "Question :  Who headlined the halftime show for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  Who voted on the venue for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "nfl owners \n",
            "\n",
            "Untokenized Answer :  nfl owners\n",
            "\n",
            "real answer :  [{'answer_start': 17, 'text': 'NFL owners'}, {'answer_start': 17, 'text': 'NFL owners'}, {'answer_start': 17, 'text': 'NFL owners'}]\n",
            "\n",
            "Question :  How many NFL teams have finished the regular season with one loss?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ten \n",
            "\n",
            "Untokenized Answer :  ten\n",
            "\n",
            "real answer :  [{'answer_start': 138, 'text': 'ten'}, {'answer_start': 138, 'text': 'ten'}, {'answer_start': 219, 'text': 'six'}]\n",
            "\n",
            "Question :  Whic Carolina Panthers running back was waived?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "dean ##gel ##o williams \n",
            "\n",
            "Untokenized Answer :  deangelo williams\n",
            "\n",
            "real answer :  [{'answer_start': 38, 'text': 'DeAngelo Williams'}, {'answer_start': 38, 'text': 'DeAngelo Williams'}, {'answer_start': 47, 'text': 'Williams'}]\n",
            "\n",
            "Question :  What is the name of the stadium in San Francisco Bay Area?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 153, 'text': \"Levi's Stadium.\"}, {'answer_start': 153, 'text': \"Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium.\"}]\n",
            "\n",
            "Question :  Which network broadcasted the 50th Super Bowl game? \n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cbs \n",
            "\n",
            "Untokenized Answer :  cbs\n",
            "\n",
            "real answer :  [{'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}, {'answer_start': 0, 'text': 'CBS'}]\n",
            "\n",
            "Question :  What seed was the Denver Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "number one \n",
            "\n",
            "Untokenized Answer :  number one\n",
            "\n",
            "real answer :  [{'answer_start': 35, 'text': 'number one'}, {'answer_start': 35, 'text': 'number one'}, {'answer_start': 42, 'text': 'one'}]\n",
            "\n",
            "Question :  What date were the top two stadium choices for Super Bowl 50 announced?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "october 16 , 2012 \n",
            "\n",
            "Untokenized Answer :  october 16 , 2012\n",
            "\n",
            "real answer :  [{'answer_start': 24, 'text': 'October 16, 2012'}, {'answer_start': 24, 'text': 'October 16, 2012'}, {'answer_start': 24, 'text': 'October 16, 2012,'}]\n",
            "\n",
            "Question :  Who coached each Super Bowl 50 participant in their most recent Super Bowl appearance prior to Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "john fox \n",
            "\n",
            "Untokenized Answer :  john fox\n",
            "\n",
            "real answer :  [{'answer_start': 636, 'text': 'John Fox'}, {'answer_start': 636, 'text': 'John Fox'}, {'answer_start': 641, 'text': 'Fox'}]\n",
            "\n",
            "Question :  What was the first Super Bowl that the Carolina Panthers played in? \n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xx ##x ##vi ##ii \n",
            "\n",
            "Untokenized Answer :  super bowl xxxviii\n",
            "\n",
            "real answer :  [{'answer_start': 573, 'text': 'Super Bowl XXXVIII'}, {'answer_start': 573, 'text': 'Super Bowl XXXVIII'}, {'answer_start': 584, 'text': 'XXXVIII'}]\n",
            "\n",
            "Question :  Who did the Carolina Panthers beat in the 2015 NFC Championship game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  How many points did the Broncos defense allow their opponents to get?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "296 \n",
            "\n",
            "Untokenized Answer :  296\n",
            "\n",
            "real answer :  [{'answer_start': 138, 'text': '296'}, {'answer_start': 138, 'text': '296'}, {'answer_start': 138, 'text': '296'}]\n",
            "\n",
            "Question :  What was the name of New Orleans' superdome at the time that Super Bowl 50 took place?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "mercedes - benz super ##dome \n",
            "\n",
            "Untokenized Answer :  mercedes - benz superdome\n",
            "\n",
            "real answer :  [{'answer_start': 69, 'text': 'Mercedes-Benz Superdome'}, {'answer_start': 69, 'text': 'Mercedes-Benz Superdome'}, {'answer_start': 69, 'text': 'Mercedes-Benz Superdome'}]\n",
            "\n",
            "Question :  How many Panthers players were selected to the Pro Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ten \n",
            "\n",
            "Untokenized Answer :  ten\n",
            "\n",
            "real answer :  [{'answer_start': 807, 'text': 'Ten'}, {'answer_start': 807, 'text': 'Ten'}, {'answer_start': 807, 'text': 'Ten'}]\n",
            "\n",
            "Question :  On what date was Super Bowl 50 given to Levi's Stadium?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "may 21 , 2013 \n",
            "\n",
            "Untokenized Answer :  may 21 , 2013\n",
            "\n",
            "real answer :  [{'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013,'}]\n",
            "\n",
            "Question :  How many times was Manning intercepted during the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "17 \n",
            "\n",
            "Untokenized Answer :  17\n",
            "\n",
            "real answer :  [{'answer_start': 115, 'text': '17'}, {'answer_start': 115, 'text': '17'}, {'answer_start': 115, 'text': '17'}]\n",
            "\n",
            "Question :  What ranking does the Super Bowl 50 halftime show have on the list of most watched TV broadcasts?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "third \n",
            "\n",
            "Untokenized Answer :  third\n",
            "\n",
            "real answer :  [{'answer_start': 355, 'text': 'third'}, {'answer_start': 355, 'text': 'third'}, {'answer_start': 355, 'text': 'third'}]\n",
            "\n",
            "Question :  Which Super Bowl halftime show did Beyoncé headline?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xl ##vi ##ii \n",
            "\n",
            "Untokenized Answer :  super bowl xlviii\n",
            "\n",
            "real answer :  [{'answer_start': 275, 'text': 'Super Bowl XLVII'}, {'answer_start': 275, 'text': 'Super Bowl XLVII'}, {'answer_start': 275, 'text': 'Super Bowl XLVII'}]\n",
            "\n",
            "Question :  Who did the Broncos prevent from going to the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new england patriots \n",
            "\n",
            "Untokenized Answer :  new england patriots\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  What year did Levi's Stadium become fully approved to host Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2013 \n",
            "\n",
            "Untokenized Answer :  2013\n",
            "\n",
            "real answer :  [{'answer_start': 11, 'text': '2013'}, {'answer_start': 11, 'text': '2013'}, {'answer_start': 11, 'text': '2013'}]\n",
            "\n",
            "Question :  When did Levi's Stadium open?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2014 \n",
            "\n",
            "Untokenized Answer :  2014\n",
            "\n",
            "real answer :  [{'answer_start': 144, 'text': '2014'}, {'answer_start': 141, 'text': 'in 2014'}, {'answer_start': 144, 'text': '2014'}]\n",
            "\n",
            "Question :  What was Newton's quarterback rating for 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "99 . 4 \n",
            "\n",
            "Untokenized Answer :  99 . 4\n",
            "\n",
            "real answer :  [{'answer_start': 396, 'text': '99.4.'}, {'answer_start': 396, 'text': '99.4'}, {'answer_start': 396, 'text': '99.4.'}]\n",
            "\n",
            "Question :  How many games did the Panthers lose in the regular season before Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ten \n",
            "\n",
            "Untokenized Answer :  ten\n",
            "\n",
            "real answer :  [{'answer_start': 42, 'text': 'one'}, {'answer_start': 251, 'text': '1'}, {'answer_start': 251, 'text': '1'}]\n",
            "\n",
            "Question :  Prior to Super Bowl 50, when were the Carolina Panthers last there?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "prior to super bowl 50 \n",
            "\n",
            "Untokenized Answer :  prior to super bowl 50\n",
            "\n",
            "real answer :  [{'answer_start': 573, 'text': 'Super Bowl XXXVIII.'}, {'answer_start': 573, 'text': 'Super Bowl XXXVIII'}, {'answer_start': 573, 'text': 'Super Bowl XXXVIII'}]\n",
            "\n",
            "Question :  What California city last hosted the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "san diego \n",
            "\n",
            "Untokenized Answer :  san diego\n",
            "\n",
            "real answer :  [{'answer_start': 308, 'text': 'San Diego'}, {'answer_start': 308, 'text': 'San Diego'}, {'answer_start': 308, 'text': 'San Diego'}]\n",
            "\n",
            "Question :  What was the final score of Super Bowl 50? \n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "24 – 10 \n",
            "\n",
            "Untokenized Answer :  24 – 10\n",
            "\n",
            "real answer :  [{'answer_start': 267, 'text': '24–10'}, {'answer_start': 267, 'text': '24–10'}, {'answer_start': 267, 'text': '24–10'}]\n",
            "\n",
            "Question :  How many sacks did Derek Wolfe register?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "5 ##½ \n",
            "\n",
            "Untokenized Answer :  5½\n",
            "\n",
            "real answer :  [{'answer_start': 198, 'text': '5½'}, {'answer_start': 198, 'text': '5½'}, {'answer_start': 198, 'text': '5½'}]\n",
            "\n",
            "Question :  What city did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "santa clara \n",
            "\n",
            "Untokenized Answer :  santa clara\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}]\n",
            "\n",
            "Question :  Who was the Panthers' tackle leader for 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ka ##wan ##n short \n",
            "\n",
            "Untokenized Answer :  kawann short\n",
            "\n",
            "real answer :  [{'answer_start': 666, 'text': 'Luke Kuechly.'}, {'answer_start': 666, 'text': 'Luke Kuechly'}, {'answer_start': 199, 'text': 'Short'}]\n",
            "\n",
            "Question :  How many sites did the NFL narrow down Super Bowl 50's location to?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three \n",
            "\n",
            "Untokenized Answer :  three\n",
            "\n",
            "real answer :  [{'answer_start': 43, 'text': 'three'}, {'answer_start': 43, 'text': 'three'}, {'answer_start': 43, 'text': 'three'}]\n",
            "\n",
            "Question :  How many tackles did Von Miller accomlish by himself in the game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  Who won the MVP for the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}]\n",
            "\n",
            "Question :  What was the most recent Super Bowl hosted in the South Florida/Miami area?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xl ##iv \n",
            "\n",
            "Untokenized Answer :  super bowl xliv\n",
            "\n",
            "real answer :  [{'answer_start': 242, 'text': 'Super Bowl XLIV'}, {'answer_start': 242, 'text': 'Super Bowl XLIV'}, {'answer_start': 261, 'text': '2010'}]\n",
            "\n",
            "Question :  Who led the Broncos with 105 receptions?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "dem ##ary ##ius thomas \n",
            "\n",
            "Untokenized Answer :  demaryius thomas\n",
            "\n",
            "real answer :  [{'answer_start': 253, 'text': 'Demaryius Thomas'}, {'answer_start': 253, 'text': 'Demaryius Thomas'}, {'answer_start': 263, 'text': 'Thomas'}]\n",
            "\n",
            "Question :  What is the name of the Bronco's head coach, who was hired after John Fox? \n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gary ku ##bia ##k \n",
            "\n",
            "Untokenized Answer :  gary kubiak\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 332, 'text': 'Kubiak'}]\n",
            "\n",
            "Question :  How many interceptions are the Panthers defense credited with in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "24 \n",
            "\n",
            "Untokenized Answer :  24\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': '24'}, {'answer_start': 124, 'text': '24'}, {'answer_start': 124, 'text': '24'}]\n",
            "\n",
            "Question :  How many tackles did Von Miller get during the game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 45, 'text': '5'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  How old was Peyton Manning in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "39 \n",
            "\n",
            "Untokenized Answer :  39\n",
            "\n",
            "real answer :  [{'answer_start': 817, 'text': '39'}, {'answer_start': 817, 'text': '39'}, {'answer_start': 817, 'text': '39'}]\n",
            "\n",
            "Question :  What does AFC stand for?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "national football league \n",
            "\n",
            "Untokenized Answer :  national football league\n",
            "\n",
            "real answer :  [{'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}, {'answer_start': 133, 'text': 'American Football Conference'}]\n",
            "\n",
            "Question :  What two Denver players ranked at 5 percent for sacks?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "derek wolfe and malik jackson \n",
            "\n",
            "Untokenized Answer :  derek wolfe and malik jackson\n",
            "\n",
            "real answer :  [{'answer_start': 159, 'text': 'Derek Wolfe and Malik Jackson'}, {'answer_start': 159, 'text': 'Derek Wolfe and Malik Jackson'}]\n",
            "\n",
            "Question :  Which NFL team represented the NFC at Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "carolina panthers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}, {'answer_start': 249, 'text': 'Carolina Panthers'}]\n",
            "\n",
            "Question :  Who won Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  What Panther tore his ACL in the preseason?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "kelvin benjamin \n",
            "\n",
            "Untokenized Answer :  kelvin benjamin\n",
            "\n",
            "real answer :  [{'answer_start': 85, 'text': 'Kelvin Benjamin'}, {'answer_start': 85, 'text': 'Kelvin Benjamin'}, {'answer_start': 92, 'text': 'Benjamin'}]\n",
            "\n",
            "Question :  Who lost to the Broncos in the AFC Championship?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 372, 'text': 'New England Patriots'}, {'answer_start': 368, 'text': 'the New England Patriots'}, {'answer_start': 372, 'text': 'New England Patriots'}]\n",
            "\n",
            "Question :  Who was the Most Valuable Player for the 2015 NFL season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam newton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  In what year did Roger Goodell call Super Bowl 50 'an important game for us as a league'?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2012 \n",
            "\n",
            "Untokenized Answer :  2012\n",
            "\n",
            "real answer :  [{'answer_start': 9, 'text': '2012'}, {'answer_start': 9, 'text': '2012'}, {'answer_start': 9, 'text': '2012'}]\n",
            "\n",
            "Question :  How many career sacks did Jared Allen have?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "136 \n",
            "\n",
            "Untokenized Answer :  136\n",
            "\n",
            "real answer :  [{'answer_start': 470, 'text': '136'}, {'answer_start': 470, 'text': '136'}, {'answer_start': 470, 'text': '136'}]\n",
            "\n",
            "Question :  How much money did a 1/2 minute commercial cost?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  What is the name of the stadium where Super Bowl 50 was played?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 153, 'text': \"Levi's Stadium.\"}, {'answer_start': 153, 'text': \"Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium.\"}]\n",
            "\n",
            "Question :  What was Manning's passer rating at the end of the season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "67 . 9 \n",
            "\n",
            "Untokenized Answer :  67 . 9\n",
            "\n",
            "real answer :  [{'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}]\n",
            "\n",
            "Question :  How many times was Cam Newton sacked?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven \n",
            "\n",
            "Untokenized Answer :  seven\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}]\n",
            "\n",
            "Question :  Super Bowl 50 determined the NFL champion for what season?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 \n",
            "\n",
            "Untokenized Answer :  2015\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 112, 'text': 'the 2015 season'}, {'answer_start': 116, 'text': '2015'}]\n",
            "\n",
            "Question :  What was the first team Peyton Manning began playing for?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "indianapolis colts \n",
            "\n",
            "Untokenized Answer :  indianapolis colts\n",
            "\n",
            "real answer :  [{'answer_start': 641, 'text': 'Indianapolis Colts'}, {'answer_start': 637, 'text': 'the Indianapolis Colts'}, {'answer_start': 663, 'text': '1998'}]\n",
            "\n",
            "Question :  Who was this season's NFL MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam newton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  Who decided not to approve paying for renovations at Sun Life Stadium that the league wanted for them to do to host Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "florida legislature \n",
            "\n",
            "Untokenized Answer :  florida legislature\n",
            "\n",
            "real answer :  [{'answer_start': 508, 'text': 'Florida legislature'}, {'answer_start': 504, 'text': 'the Florida legislature'}, {'answer_start': 508, 'text': 'Florida legislature'}]\n",
            "\n",
            "Question :  How much did it cost to build the stadium where Super Bowl 50 was played?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 1 . 2 billion \n",
            "\n",
            "Untokenized Answer :  $ 1 . 2 billion\n",
            "\n",
            "real answer :  [{'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}]\n",
            "\n",
            "Question :  Which team had the best regular season in their history?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "carolina panthers \n",
            "\n",
            "Untokenized Answer :  carolina panthers\n",
            "\n",
            "real answer :  [{'answer_start': 137, 'text': 'Carolina Panthers'}, {'answer_start': 695, 'text': 'the Panthers'}, {'answer_start': 330, 'text': 'Carolina'}]\n",
            "\n",
            "Question :  What was Peyton Manning's passer rating for the season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "67 . 9 \n",
            "\n",
            "Untokenized Answer :  67 . 9\n",
            "\n",
            "real answer :  [{'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}]\n",
            "\n",
            "Question :  Who did the Panthers face in the NFC Championship Game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers beat the Seattle Seahawks in the divisional round, running up a 31–0 halftime lead and then holding off a furious second half comeback at\n",
            "tempt to win 31–24, avenging their elimination from a year earlier. The Panthers then blew out the Arizona Cardinals in the NFC Championship Game, 49–\n",
            "15, racking up 487 yards and forcing seven turnovers.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 249, 'text': 'Arizona Cardinals'}, {'answer_start': 249, 'text': 'Arizona Cardinals'}, {'answer_start': 257, 'text': 'Cardinals'}]\n",
            "\n",
            "Question :  Which Carolina Panthers player was named Most Valuable Player?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam newton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  Who was given the esteemed status of MVP for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  Who if the commissioner of the NFL?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "roger good ##ell \n",
            "\n",
            "Untokenized Answer :  roger goodell\n",
            "\n",
            "real answer :  [{'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 38, 'text': 'Goodell'}]\n",
            "\n",
            "Question :  Who was the commissioner of the NFL in 2012? \n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "roger good ##ell \n",
            "\n",
            "Untokenized Answer :  roger goodell\n",
            "\n",
            "real answer :  [{'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 32, 'text': 'Roger Goodell'}, {'answer_start': 38, 'text': 'Goodell'}]\n",
            "\n",
            "Question :  What stadium did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}]\n",
            "\n",
            "Question :  Prior to Super Bowl 50, when were the Broncos last there?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "prior to super bowl 50 . \n",
            "\n",
            "Untokenized Answer :  prior to super bowl 50 .\n",
            "\n",
            "real answer :  [{'answer_start': 444, 'text': 'Super Bowl XLVIII'}, {'answer_start': 444, 'text': 'Super Bowl XLVIII'}, {'answer_start': 444, 'text': 'Super Bowl XLVIII'}]\n",
            "\n",
            "Question :  What was the average cost for a TV ad lasting 30 seconds during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  Which running back did the Panthers waive?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "dean ##gel ##o williams \n",
            "\n",
            "Untokenized Answer :  deangelo williams\n",
            "\n",
            "real answer :  [{'answer_start': 38, 'text': 'DeAngelo Williams'}, {'answer_start': 38, 'text': 'DeAngelo Williams'}, {'answer_start': 47, 'text': 'Williams'}]\n",
            "\n",
            "Question :  What was Manning's passer rating for the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "67 . 9 \n",
            "\n",
            "Untokenized Answer :  67 . 9\n",
            "\n",
            "real answer :  [{'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}, {'answer_start': 44, 'text': '67.9'}]\n",
            "\n",
            "Question :  On what date did the Florida legislature decide against the plan to renovate the Miami stadium?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "may 3 , 2013 \n",
            "\n",
            "Untokenized Answer :  may 3 , 2013\n",
            "\n",
            "real answer :  [{'answer_start': 491, 'text': 'May 3, 2013'}, {'answer_start': 491, 'text': 'May 3, 2013'}, {'answer_start': 491, 'text': 'May 3, 2013'}]\n",
            "\n",
            "Question :  Who registered the most receptions on the Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "dem ##ary ##ius thomas \n",
            "\n",
            "Untokenized Answer :  demaryius thomas\n",
            "\n",
            "real answer :  [{'answer_start': 253, 'text': 'Demaryius Thomas'}, {'answer_start': 253, 'text': 'Demaryius Thomas'}, {'answer_start': 263, 'text': 'Thomas'}]\n",
            "\n",
            "Question :  What was the theme of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "american football game to determine the champion of the national football league ( nfl ) for the 2015 season \n",
            "\n",
            "Untokenized Answer :  american football game to determine the champion of the national football league ( nfl ) for the 2015 season\n",
            "\n",
            "real answer :  [{'answer_start': 487, 'text': '\"golden anniversary\"'}, {'answer_start': 521, 'text': 'gold-themed'}, {'answer_start': 521, 'text': 'gold'}]\n",
            "\n",
            "Question :  When did Levi's stadium open to the public? \n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2014 \n",
            "\n",
            "Untokenized Answer :  2014\n",
            "\n",
            "real answer :  [{'answer_start': 144, 'text': '2014'}, {'answer_start': 144, 'text': '2014'}, {'answer_start': 144, 'text': '2014'}]\n",
            "\n",
            "Question :  What month, day and year did Super Bowl 50 take place? \n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "february 7 , 2016 \n",
            "\n",
            "Untokenized Answer :  february 7 , 2016\n",
            "\n",
            "real answer :  [{'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7, 2016'}, {'answer_start': 334, 'text': 'February 7, 2016'}]\n",
            "\n",
            "Question :  How much did it cost to build Levi's Stadium?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 1 . 2 billion \n",
            "\n",
            "Untokenized Answer :  $ 1 . 2 billion\n",
            "\n",
            "real answer :  [{'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}, {'answer_start': 113, 'text': '$1.2 billion'}]\n",
            "\n",
            "Question :  How many games did the Broncos lose during their regular 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "13 – 7 \n",
            "\n",
            "Untokenized Answer :  13 – 7\n",
            "\n",
            "real answer :  [{'answer_start': 201, 'text': 'four'}, {'answer_start': 1991, 'text': '4'}, {'answer_start': 1991, 'text': '4'}]\n",
            "\n",
            "Question :  How many times has a Super Bowl taken place at Miami's Sun Life Stadium?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "10 \n",
            "\n",
            "Untokenized Answer :  10\n",
            "\n",
            "real answer :  [{'answer_start': 51, 'text': 'two'}, {'answer_start': 168, 'text': '10'}, {'answer_start': 168, 'text': '10'}]\n",
            "\n",
            "Question :  What was the cost for a half minute ad?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million for a 30-second'}]\n",
            "\n",
            "Question :  What seed was the Carolina Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "number one \n",
            "\n",
            "Untokenized Answer :  number one\n",
            "\n",
            "real answer :  [{'answer_start': 35, 'text': 'number one'}, {'answer_start': 35, 'text': 'number one'}, {'answer_start': 42, 'text': 'one'}]\n",
            "\n",
            "Question :  When did the league go from 15 to 16 games in the regular season?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1978 \n",
            "\n",
            "Untokenized Answer :  1978\n",
            "\n",
            "real answer :  [{'answer_start': 324, 'text': '1978.'}, {'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}]\n",
            "\n",
            "Question :  Which group headlined the Super Bowl 50 halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  What city is tied with Miami for hosting the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new orleans \n",
            "\n",
            "Untokenized Answer :  new orleans\n",
            "\n",
            "real answer :  [{'answer_start': 197, 'text': 'New Orleans'}, {'answer_start': 197, 'text': 'New Orleans'}, {'answer_start': 197, 'text': 'New Orleans'}]\n",
            "\n",
            "Question :  What was the other finalist besides Levi's Stadium?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "sun life stadium \n",
            "\n",
            "Untokenized Answer :  sun life stadium\n",
            "\n",
            "real answer :  [{'answer_start': 70, 'text': 'Sun Life Stadium'}, {'answer_start': 70, 'text': 'Sun Life Stadium'}, {'answer_start': 70, 'text': 'Sun Life Stadium'}]\n",
            "\n",
            "Question :  How many Panthers were designated All-Pro?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "eight \n",
            "\n",
            "Untokenized Answer :  eight\n",
            "\n",
            "real answer :  [{'answer_start': 892, 'text': 'eight'}, {'answer_start': 892, 'text': 'eight'}, {'answer_start': 892, 'text': 'eight'}]\n",
            "\n",
            "Question :  What are the most passes that Greg Olsen has had?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "77 \n",
            "\n",
            "Untokenized Answer :  77\n",
            "\n",
            "real answer :  [{'answer_start': 481, 'text': '77 passes'}, {'answer_start': 481, 'text': '77'}, {'answer_start': 481, 'text': '77'}]\n",
            "\n",
            "Question :  Super Bowl 50 decided the NFL champion for what season?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2015 \n",
            "\n",
            "Untokenized Answer :  2015\n",
            "\n",
            "real answer :  [{'answer_start': 116, 'text': '2015'}, {'answer_start': 112, 'text': 'the 2015 season'}, {'answer_start': 116, 'text': '2015'}]\n",
            "\n",
            "Question :  Which Florida venue was one of three considered for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "san francisco bay area ' s levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  san francisco bay area ' s levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 94, 'text': \"Miami's Sun Life Stadium\"}, {'answer_start': 94, 'text': \"Miami's Sun Life Stadium\"}, {'answer_start': 102, 'text': 'Sun Life Stadium'}]\n",
            "\n",
            "Question :  Who did the Panthers beat in the NFC Championship Game?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "arizona cardinals \n",
            "\n",
            "Untokenized Answer :  arizona cardinals\n",
            "\n",
            "real answer :  [{'answer_start': 152, 'text': 'Arizona Cardinals'}, {'answer_start': 148, 'text': 'the Arizona Cardinals'}, {'answer_start': 152, 'text': 'Arizona Cardinals'}]\n",
            "\n",
            "Question :  What position does Von Miller play for the Denver Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "linebacker \n",
            "\n",
            "Untokenized Answer :  linebacker\n",
            "\n",
            "real answer :  [{'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}]\n",
            "\n",
            "Question :  What halftime performer previously headlined Super Bowl XLVII?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé'}, {'answer_start': 233, 'text': 'Beyoncé'}, {'answer_start': 233, 'text': 'Beyoncé'}]\n",
            "\n",
            "Question :  Who was the Denver head coach for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gary ku ##bia ##k \n",
            "\n",
            "Untokenized Answer :  gary kubiak\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 296, 'text': 'Kubiak'}]\n",
            "\n",
            "Question :  What year did the Carolina Panthers form?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1995 \n",
            "\n",
            "Untokenized Answer :  1995\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': '1995.'}, {'answer_start': 291, 'text': '1995'}, {'answer_start': 291, 'text': '1995'}]\n",
            "\n",
            "Question :  How many times has the South Florida/Miami area hosted the Super Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "10 \n",
            "\n",
            "Untokenized Answer :  10\n",
            "\n",
            "real answer :  [{'answer_start': 168, 'text': '10'}, {'answer_start': 168, 'text': '10'}, {'answer_start': 168, 'text': '10'}]\n",
            "\n",
            "Question :  What team was the AFC champion?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  How many times was the Panthers' quarterback sacked?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seven times \n",
            "\n",
            "Untokenized Answer :  seven times\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}, {'answer_start': 124, 'text': 'seven'}]\n",
            "\n",
            "Question :  What was the last Super Bowl the Broncos participated in?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl 50 \n",
            "\n",
            "Untokenized Answer :  super bowl 50\n",
            "\n",
            "real answer :  [{'answer_start': 444, 'text': 'Super Bowl XLVIII'}, {'answer_start': 444, 'text': 'Super Bowl XLVIII'}, {'answer_start': 455, 'text': 'XLVIII'}]\n",
            "\n",
            "Question :  What 2015 NFL team one the AFC playoff?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  What city did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "santa clara \n",
            "\n",
            "Untokenized Answer :  santa clara\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}, {'answer_start': 403, 'text': 'Santa Clara'}]\n",
            "\n",
            "Question :  How many solo tackles did Von Miller make at Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "five \n",
            "\n",
            "Untokenized Answer :  five\n",
            "\n",
            "real answer :  [{'answer_start': 295, 'text': 'five solo tackles'}, {'answer_start': 295, 'text': 'five'}, {'answer_start': 295, 'text': 'five'}]\n",
            "\n",
            "Question :  Which team in Super Bowl 50 had a 15-1 record?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 101, 'text': 'Carolina Panthers'}, {'answer_start': 97, 'text': 'The Carolina Panthers'}, {'answer_start': 110, 'text': 'Panthers'}]\n",
            "\n",
            "Question :  Where did the spring meetings of the NFL owners take place?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "boston \n",
            "\n",
            "Untokenized Answer :  boston\n",
            "\n",
            "real answer :  [{'answer_start': 56, 'text': 'Boston'}, {'answer_start': 53, 'text': 'in Boston'}, {'answer_start': 3, 'text': 'May 21, 2013'}]\n",
            "\n",
            "Question :  What halftime performer previously headlined Super Bowl XLVIII?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 245, 'text': 'Bruno Mars'}, {'answer_start': 251, 'text': 'Mars'}]\n",
            "\n",
            "Question :  Who did the Broncos play in the last week of the regular season?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "peyton manning ' s shotgun passing skills , but struggled with numerous changes and injuries to the offensive line , as well as manning having his worst statistical season since his rookie year with the indianapolis colts in 1998 , due to a plant ##ar fa ##sc ##ii ##tis injury in his heel that he had suffered since the summer , and the simple fact that manning was getting old , as he turned 39 in the 2015 off - season . although the team had a 7 – 0 start , manning led the nfl in interceptions . in week 10 , manning suffered a partial tear of the plant ##ar fa ##sc ##ii ##tis in his left foot . he set the nfl ' s all - time record for career passing yards in this game , but was bench ##ed after throwing four interceptions in favor of backup quarterback brock os ##weiler \n",
            "\n",
            "Untokenized Answer :  peyton manning ' s shotgun passing skills , but struggled with numerous changes and injuries to the offensive line , as well as manning having his worst statistical season since his rookie year with the indianapolis colts in 1998 , due to a plantar fasciitis injury in his heel that he had suffered since the summer , and the simple fact that manning was getting old , as he turned 39 in the 2015 off - season . although the team had a 7 – 0 start , manning led the nfl in interceptions . in week 10 , manning suffered a partial tear of the plantar fasciitis in his left foot . he set the nfl ' s all - time record for career passing yards in this game , but was benched after throwing four interceptions in favor of backup quarterback brock osweiler\n",
            "\n",
            "real answer :  [{'answer_start': 1404, 'text': 'San Diego Chargers'}, {'answer_start': 1404, 'text': 'San Diego Chargers'}, {'answer_start': 1414, 'text': 'Chargers'}]\n",
            "\n",
            "Question :  One of the sites, Merceds-Benz Superdome, is located where?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new orleans \n",
            "\n",
            "Untokenized Answer :  new orleans\n",
            "\n",
            "real answer :  [{'answer_start': 56, 'text': 'New Orleans'}, {'answer_start': 56, 'text': 'New Orleans'}, {'answer_start': 56, 'text': \"New Orleans'\"}]\n",
            "\n",
            "Question :  Where was Super Bowl 50 held?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara, California.'}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}]\n",
            "\n",
            "Question :  What position does Demaryius Thomas play?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "receiver \n",
            "\n",
            "Untokenized Answer :  receiver\n",
            "\n",
            "real answer :  [{'answer_start': 244, 'text': 'receiver'}, {'answer_start': 244, 'text': 'receiver'}, {'answer_start': 263, 'text': 'Thomas'}]\n",
            "\n",
            "Question :  Who did Peyton Manning play for as a rookie?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "indianapolis colts \n",
            "\n",
            "Untokenized Answer :  indianapolis colts\n",
            "\n",
            "real answer :  [{'answer_start': 641, 'text': 'Indianapolis Colts'}, {'answer_start': 637, 'text': 'the Indianapolis Colts'}, {'answer_start': 654, 'text': 'Colts'}]\n",
            "\n",
            "Question :  Who played quarterback for the Broncos after Peyton Manning was benched?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "brock os ##weiler \n",
            "\n",
            "Untokenized Answer :  brock osweiler\n",
            "\n",
            "real answer :  [{'answer_start': 1157, 'text': 'Brock Osweiler'}, {'answer_start': 1157, 'text': 'Brock Osweiler'}, {'answer_start': 1163, 'text': 'Osweiler'}]\n",
            "\n",
            "Question :  In what city did the last Super Bowl in California occur?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "san diego \n",
            "\n",
            "Untokenized Answer :  san diego\n",
            "\n",
            "real answer :  [{'answer_start': 308, 'text': 'San Diego'}, {'answer_start': 308, 'text': 'San Diego'}, {'answer_start': 308, 'text': 'San Diego'}]\n",
            "\n",
            "Question :  How many times prios has the Sun Life Stadium had Super Bowls?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "10 \n",
            "\n",
            "Untokenized Answer :  10\n",
            "\n",
            "real answer :  [{'answer_start': 263, 'text': '10.'}, {'answer_start': 168, 'text': '10'}, {'answer_start': 168, 'text': '10 times'}]\n",
            "\n",
            "Question :  Who had the most rushing yards on the Broncos?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "c . j . anderson \n",
            "\n",
            "Untokenized Answer :  c . j . anderson\n",
            "\n",
            "real answer :  [{'answer_start': 572, 'text': 'C. J. Anderson'}, {'answer_start': 572, 'text': 'C. J. Anderson'}, {'answer_start': 578, 'text': 'Anderson'}]\n",
            "\n",
            "Question :  How many teams have been in the Super Bowl eight times?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "four \n",
            "\n",
            "Untokenized Answer :  four\n",
            "\n",
            "real answer :  [{'answer_start': 575, 'text': 'four'}, {'answer_start': 575, 'text': 'four'}, {'answer_start': 575, 'text': 'four'}]\n",
            "\n",
            "Question :  What Denver player caused two fumbles for the Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}]\n",
            "\n",
            "Question :  When were the two finalists for hosting Super Bowl 50 announced?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "october 16 , 2012 \n",
            "\n",
            "Untokenized Answer :  october 16 , 2012\n",
            "\n",
            "real answer :  [{'answer_start': 24, 'text': 'October 16, 2012'}, {'answer_start': 24, 'text': 'October 16, 2012,'}, {'answer_start': 24, 'text': 'October 16, 2012'}]\n",
            "\n",
            "Question :  At which Super Bowl did Beyonce headline the halftime show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xl ##vi ##ii \n",
            "\n",
            "Untokenized Answer :  super bowl xlviii\n",
            "\n",
            "real answer :  [{'answer_start': 275, 'text': 'Super Bowl XLVII'}, {'answer_start': 275, 'text': 'Super Bowl XLVII'}, {'answer_start': 286, 'text': 'XLVII'}]\n",
            "\n",
            "Question :  Who started at tight end for the Panthers?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "greg olsen \n",
            "\n",
            "Untokenized Answer :  greg olsen\n",
            "\n",
            "real answer :  [{'answer_start': 444, 'text': 'Greg Olsen'}, {'answer_start': 444, 'text': 'Greg Olsen'}, {'answer_start': 449, 'text': 'Olsen'}]\n",
            "\n",
            "Question :  How many 2015 season interceptions did the Panthers' defense get?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "24 \n",
            "\n",
            "Untokenized Answer :  24\n",
            "\n",
            "real answer :  [{'answer_start': 124, 'text': '24'}, {'answer_start': 124, 'text': '24'}, {'answer_start': 124, 'text': '24'}]\n",
            "\n",
            "Question :  What position does DeMarcus currently play? \n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "linebacker \n",
            "\n",
            "Untokenized Answer :  linebacker\n",
            "\n",
            "real answer :  [{'answer_start': 309, 'text': 'Linebacker'}, {'answer_start': 309, 'text': 'Linebacker'}, {'answer_start': 309, 'text': 'Linebacker'}]\n",
            "\n",
            "Question :  Which Newton turnover resulted in seven points for Denver?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "a fumble \n",
            "\n",
            "Untokenized Answer :  a fumble\n",
            "\n",
            "real answer :  [{'answer_start': 183, 'text': 'a fumble'}, {'answer_start': 183, 'text': 'a fumble'}, {'answer_start': 185, 'text': 'fumble'}]\n",
            "\n",
            "Question :  What is the record number of touchdowns Cam Newton has had? \n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "45 \n",
            "\n",
            "Untokenized Answer :  45\n",
            "\n",
            "real answer :  [{'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}, {'answer_start': 279, 'text': '45'}]\n",
            "\n",
            "Question :  What two artists came out with Coldplay during the half-time show?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "beyonce and bruno mars \n",
            "\n",
            "Untokenized Answer :  beyonce and bruno mars\n",
            "\n",
            "real answer :  [{'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}, {'answer_start': 233, 'text': 'Beyoncé and Bruno Mars'}]\n",
            "\n",
            "Question :  Who was the defensive coordinator for the Broncos in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "wade phillips \n",
            "\n",
            "Untokenized Answer :  wade phillips\n",
            "\n",
            "real answer :  [{'answer_start': 1640, 'text': 'Wade Phillips'}, {'answer_start': 1640, 'text': 'Wade Phillips'}, {'answer_start': 1645, 'text': 'Phillips'}]\n",
            "\n",
            "Question :  According to game stats, which Super Bowl 50 quarterback had his worst year since his first season as a player in the NFL?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "peyton manning \n",
            "\n",
            "Untokenized Answer :  peyton manning\n",
            "\n",
            "real answer :  [{'answer_start': 442, 'text': 'Peyton Manning'}, {'answer_start': 442, 'text': 'Peyton Manning'}, {'answer_start': 566, 'text': 'Manning'}]\n",
            "\n",
            "Question :  How many interceptions did Cam Newton throw?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "10 \n",
            "\n",
            "Untokenized Answer :  10\n",
            "\n",
            "real answer :  [{'answer_start': 312, 'text': '10'}, {'answer_start': 338, 'text': '10'}, {'answer_start': 338, 'text': '10'}]\n",
            "\n",
            "Question :  Who won Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "denver broncos \n",
            "\n",
            "Untokenized Answer :  denver broncos\n",
            "\n",
            "real answer :  [{'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}, {'answer_start': 177, 'text': 'Denver Broncos'}]\n",
            "\n",
            "Question :  Who was the Most Valuable Player of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 248, 'text': 'Von Miller'}]\n",
            "\n",
            "Question :  How many interceptions did Josh Norman score touchdowns with in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 284, 'text': 'two.'}, {'answer_start': 1124, 'text': 'two'}, {'answer_start': 124, 'text': '24'}]\n",
            "\n",
            "Question :  How many forced fumbles did Von Miller have during the Super Bowl 50 game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "two \n",
            "\n",
            "Untokenized Answer :  two\n",
            "\n",
            "real answer :  [{'answer_start': 314, 'text': '2'}, {'answer_start': 328, 'text': 'two'}, {'answer_start': 328, 'text': 'two'}]\n",
            "\n",
            "Question :  What one word did the NFL commissioner use to describe what Super Bowl 50 was intended to be?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "spectacular \n",
            "\n",
            "Untokenized Answer :  spectacular\n",
            "\n",
            "real answer :  [{'answer_start': 106, 'text': 'spectacular'}, {'answer_start': 106, 'text': 'spectacular'}, {'answer_start': 106, 'text': 'spectacular'}]\n",
            "\n",
            "Question :  Who lead the Super Bowl 50 halftime performance?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cold ##play \n",
            "\n",
            "Untokenized Answer :  coldplay\n",
            "\n",
            "real answer :  [{'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}, {'answer_start': 194, 'text': 'Coldplay'}]\n",
            "\n",
            "Question :  Which player had the most interceptions for the season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers defense gave up just 308 points, ranking sixth in the league, while also leading the NFL in interceptions with 24 and boasting four Pro B\n",
            "owl selections. Pro Bowl defensive tackle Kawann Short led the team in sacks with 11, while also forcing three fumbles and recovering two. Fellow line\n",
            "man Mario Addison added 6½ sacks. The Panthers line also featured veteran defensive end Jared Allen, a 5-time pro bowler who was the NFL's active care\n",
            "er sack leader with 136, along with defensive end Kony Ealy, who had 5 sacks in just 9 starts. Behind them, two of the Panthers three starting linebac\n",
            "kers were also selected to play in the Pro Bowl: Thomas Davis and Luke Kuechly. Davis compiled 5½ sacks, four forced fumbles, and four interceptions, \n",
            "while Kuechly led the team in tackles (118) forced two fumbles, and intercepted four passes of his own. Carolina's secondary featured Pro Bowl safety \n",
            "Kurt Coleman, who led the team with a career high seven interceptions, while also racking up 88 tackles and Pro Bowl cornerback Josh Norman, who devel\n",
            "oped into a shutdown corner during the season and had four interceptions, two of which were returned for touchdowns.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "josh norman \n",
            "\n",
            "Untokenized Answer :  josh norman\n",
            "\n",
            "real answer :  [{'answer_start': 900, 'text': 'Kurt Coleman'}, {'answer_start': 900, 'text': 'Kurt Coleman'}, {'answer_start': 905, 'text': 'Coleman'}]\n",
            "\n",
            "Question :  When did Lev's Stadium open?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2014 \n",
            "\n",
            "Untokenized Answer :  2014\n",
            "\n",
            "real answer :  [{'answer_start': 144, 'text': '2014'}, {'answer_start': 141, 'text': 'in 2014'}, {'answer_start': 144, 'text': '2014'}]\n",
            "\n",
            "Question :  How many touchdowns did Jonathan Stewart have in 13 games?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "six \n",
            "\n",
            "Untokenized Answer :  six\n",
            "\n",
            "real answer :  [{'answer_start': 98, 'text': 'six'}, {'answer_start': 1026, 'text': 'six'}, {'answer_start': 1026, 'text': 'six'}]\n",
            "\n",
            "Question :  What injury did Manning suffer the summer before the season started?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "plant ##ar fa ##sc ##ii ##tis \n",
            "\n",
            "Untokenized Answer :  plantar fasciitis\n",
            "\n",
            "real answer :  [{'answer_start': 676, 'text': 'a plantar fasciitis injury'}, {'answer_start': 676, 'text': 'a plantar fasciitis injury'}, {'answer_start': 678, 'text': 'plantar fasciitis'}]\n",
            "\n",
            "Question :  How many picks did Aqib Talib have?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "three interceptions ) and chris harris , jr . ( two interceptions ) were the other two \n",
            "\n",
            "Untokenized Answer :  three interceptions ) and chris harris , jr . ( two interceptions ) were the other two\n",
            "\n",
            "real answer :  [{'answer_start': 302, 'text': 'three'}, {'answer_start': 585, 'text': 'three'}, {'answer_start': 585, 'text': 'three'}]\n",
            "\n",
            "Question :  Which team held the scoring lead throughout the entire game?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "broncos \n",
            "\n",
            "Untokenized Answer :  broncos\n",
            "\n",
            "real answer :  [{'answer_start': 4, 'text': 'Broncos'}, {'answer_start': 0, 'text': 'The Broncos'}, {'answer_start': 4, 'text': 'Broncos'}]\n",
            "\n",
            "Question :  What was the average cost for a 30 second commercial during Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime s\n",
            "how was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Supe\n",
            "r Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "$ 5 million \n",
            "\n",
            "Untokenized Answer :  $ 5 million\n",
            "\n",
            "real answer :  [{'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}, {'answer_start': 67, 'text': '$5 million'}]\n",
            "\n",
            "Question :  What was the win/loss ratio in 2015 for the Carolina Panthers during their regular season?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "15 – 1 \n",
            "\n",
            "Untokenized Answer :  15 – 1\n",
            "\n",
            "real answer :  [{'answer_start': 48, 'text': '15–1'}, {'answer_start': 48, 'text': '15–1'}, {'answer_start': 48, 'text': '15–1'}]\n",
            "\n",
            "Question :  What venue in Miami was a candidate for the site of Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "sun life stadium \n",
            "\n",
            "Untokenized Answer :  sun life stadium\n",
            "\n",
            "real answer :  [{'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}]\n",
            "\n",
            "Question :  When did the NFL start their 16 game seasons?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1978 \n",
            "\n",
            "Untokenized Answer :  1978\n",
            "\n",
            "real answer :  [{'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}, {'answer_start': 324, 'text': '1978'}]\n",
            "\n",
            "Question :  When was Levi's Stadium picked for Super bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "may 21 , 2013 \n",
            "\n",
            "Untokenized Answer :  may 21 , 2013\n",
            "\n",
            "real answer :  [{'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013'}]\n",
            "\n",
            "Question :  What position does Von Miller play?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "linebacker \n",
            "\n",
            "Untokenized Answer :  linebacker\n",
            "\n",
            "real answer :  [{'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}, {'answer_start': 237, 'text': 'linebacker'}]\n",
            "\n",
            "Question :  How many Panthers players were chosen for the 2015 season's Pro Bowl?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "ten \n",
            "\n",
            "Untokenized Answer :  ten\n",
            "\n",
            "real answer :  [{'answer_start': 807, 'text': 'Ten'}, {'answer_start': 807, 'text': 'Ten'}, {'answer_start': 807, 'text': 'Ten'}]\n",
            "\n",
            "Question :  How many NFL teams have gone 15-1 in one season?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "six \n",
            "\n",
            "Untokenized Answer :  six\n",
            "\n",
            "real answer :  [{'answer_start': 219, 'text': 'six'}, {'answer_start': 219, 'text': 'six'}, {'answer_start': 219, 'text': 'six'}]\n",
            "\n",
            "Question :  Who forced four fumbles for the Broncos in the 2015 season?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos' defense ranked first in the NFL yards allowed (4,530) for the first time in franchise history, and fourth in points allowed (296). Defens\n",
            "ive ends Derek Wolfe and Malik Jackson each had 5½ sacks. Pro Bowl linebacker Von Miller led the team with 11 sacks, forced four fumbles, and recovere\n",
            "d three. Linebacker DeMarcus Ware was selected to play in the Pro Bowl for the ninth time in his career, ranking second on the team with 7½ sacks. Lin\n",
            "ebacker Brandon Marshall led the team in total tackles with 109, while Danny Trevathan ranked second with 102. Cornerbacks Aqib Talib (three intercept\n",
            "ions) and Chris Harris, Jr. (two interceptions) were the other two Pro Bowl selections from the defense.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 228, 'text': 'Von Miller'}, {'answer_start': 228, 'text': 'Von Miller'}, {'answer_start': 232, 'text': 'Miller'}]\n",
            "\n",
            "Question :  Which California venue was one of three considered for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "san francisco bay area ' s levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  san francisco bay area ' s levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 128, 'text': \"San Francisco Bay Area's Levi's Stadium\"}, {'answer_start': 128, 'text': \"San Francisco Bay Area's Levi's Stadium\"}, {'answer_start': 153, 'text': \"Levi's Stadium\"}]\n",
            "\n",
            "Question :  What venue did Super Bowl 50 take place in?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium in the San Francisco Bay Area at Santa Clara\"}]\n",
            "\n",
            "Question :  When did he make the quoted remarks about Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  In early 2012, NFL Commissioner Roger Goodell stated that the league planned to make the 50th Super Bowl \"spectacular\" and that it would be \"an import\n",
            "ant game for us as a league\".\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "early 2012 \n",
            "\n",
            "Untokenized Answer :  early 2012\n",
            "\n",
            "real answer :  [{'answer_start': 3, 'text': 'early 2012'}, {'answer_start': 0, 'text': 'In early 2012'}, {'answer_start': 9, 'text': '2012'}]\n",
            "\n",
            "Question :  What was the number of times the Denver Broncos played in a Super Bowl by the time they reached Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  For the third straight season, the number one seeds from both conferences met in the Super Bowl. The Carolina Panthers became one of only ten teams to\n",
            " have completed a regular season with only one loss, and one of only six teams to have acquired a 15–1 record, while the Denver Broncos became one of \n",
            "four teams to have made eight appearances in the Super Bowl. The Broncos made their second Super Bowl appearance in three years, having reached Super \n",
            "Bowl XLVIII, while the Panthers made their second Super Bowl appearance in franchise history, their other appearance being Super Bowl XXXVIII. Coincid\n",
            "entally, both teams were coached by John Fox in their last Super Bowl appearance prior to Super Bowl 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "eight \n",
            "\n",
            "Untokenized Answer :  eight\n",
            "\n",
            "real answer :  [{'answer_start': 324, 'text': 'eight'}, {'answer_start': 324, 'text': 'eight'}, {'answer_start': 324, 'text': 'eight'}]\n",
            "\n",
            "Question :  What is the name of the stadium in Miami that was considered?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "sun life stadium \n",
            "\n",
            "Untokenized Answer :  sun life stadium\n",
            "\n",
            "real answer :  [{'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}, {'answer_start': 102, 'text': 'Sun Life Stadium'}]\n",
            "\n",
            "Question :  What year did a Super Bowl play in the bay area around San Francisco, prior to Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "1985 \n",
            "\n",
            "Untokenized Answer :  1985\n",
            "\n",
            "real answer :  [{'answer_start': 236, 'text': '1985'}, {'answer_start': 236, 'text': '1985'}, {'answer_start': 236, 'text': '1985'}]\n",
            "\n",
            "Question :  How many interceptions did manning have in 2015?\n",
            "--------------------------------------------------\n",
            "Context :  Manning finished the year with a career-low 67.9 passer rating, throwing for 2,249 yards and nine touchdowns, with 17 interceptions. In contrast, Oswe\n",
            "iler threw for 1,967 yards, 10 touchdowns and six interceptions for a rating of 86.4. Veteran receiver Demaryius Thomas led the team with 105 receptio\n",
            "ns for 1,304 yards and six touchdowns, while Emmanuel Sanders caught 76 passes for 1,135 yards and six scores, while adding another 106 yards returnin\n",
            "g punts. Tight end Owen Daniels was also a big element of the passing game with 46 receptions for 517 yards. Running back C. J. Anderson was the team'\n",
            "s leading rusher 863 yards and seven touchdowns, while also catching 25 passes for 183 yards. Running back Ronnie Hillman also made a big impact with \n",
            "720 yards, five touchdowns, 24 receptions, and a 4.7 yards per carry average. Overall, the offense ranked 19th in scoring with 355 points and did not \n",
            "have any Pro Bowl selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "17 \n",
            "\n",
            "Untokenized Answer :  17\n",
            "\n",
            "real answer :  [{'answer_start': 115, 'text': '17'}, {'answer_start': 115, 'text': '17'}, {'answer_start': 115, 'text': '17'}]\n",
            "\n",
            "Question :  When was San Francisco voted to be the location for Super Bowl 50?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2003 \n",
            "\n",
            "Untokenized Answer :  2003\n",
            "\n",
            "real answer :  [{'answer_start': 3, 'text': 'May 21, 2013'}, {'answer_start': 3, 'text': 'May 21, 2013,'}, {'answer_start': 3, 'text': 'May 21, 2013'}]\n",
            "\n",
            "Question :  Which Super Bowl was hosted in San Diego in 2003? \n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "super bowl xx ##x ##vi ##i \n",
            "\n",
            "Untokenized Answer :  super bowl xxxvii\n",
            "\n",
            "real answer :  [{'answer_start': 276, 'text': 'Super Bowl XXXVII'}, {'answer_start': 276, 'text': 'Super Bowl XXXVII'}, {'answer_start': 287, 'text': 'XXXVII'}]\n",
            "\n",
            "Question :  What other city has hosted the Super Bowl ten times?\n",
            "--------------------------------------------------\n",
            "Context :  The league announced on October 16, 2012, that the two finalists were Sun Life Stadium and Levi's Stadium. The South Florida/Miami area has previously\n",
            " hosted the event 10 times (tied for most with New Orleans), with the most recent one being Super Bowl XLIV in 2010. The San Francisco Bay Area last h\n",
            "osted in 1985 (Super Bowl XIX), held at Stanford Stadium in Stanford, California, won by the home team 49ers. The Miami bid depended on whether the st\n",
            "adium underwent renovations. However, on May 3, 2013, the Florida legislature refused to approve the funding plan to pay for the renovations, dealing \n",
            "a significant blow to Miami's chances.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new orleans \n",
            "\n",
            "Untokenized Answer :  new orleans\n",
            "\n",
            "real answer :  [{'answer_start': 197, 'text': 'New Orleans'}, {'answer_start': 197, 'text': 'New Orleans'}, {'answer_start': 197, 'text': 'New Orleans'}]\n",
            "\n",
            "Question :  When John Fox left as head coach for the Broncos, who replaced him?\n",
            "--------------------------------------------------\n",
            "Context :  Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mu\n",
            "tual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubi\n",
            "ak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton M\n",
            "anning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistic\n",
            "al season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summ\n",
            "er, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL i\n",
            "n interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career pa\n",
            "ssing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starte\n",
            "r for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale\n",
            ", where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the p\n",
            "layoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phill\n",
            "ips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has\n",
            " continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Bronc\n",
            "os finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "gary ku ##bia ##k \n",
            "\n",
            "Untokenized Answer :  gary kubiak\n",
            "\n",
            "real answer :  [{'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 291, 'text': 'Gary Kubiak'}, {'answer_start': 332, 'text': 'Kubiak'}]\n",
            "\n",
            "Question :  How many punts did Ted Ginn Jr. return?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "27 \n",
            "\n",
            "Untokenized Answer :  27\n",
            "\n",
            "real answer :  [{'answer_start': 659, 'text': '27'}, {'answer_start': 659, 'text': '27'}, {'answer_start': 659, 'text': '27'}]\n",
            "\n",
            "Question :  Which Carolina Panthers team member was picked as the team's MVP in 2015? \n",
            "--------------------------------------------------\n",
            "Context :  The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated th\n",
            "e Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The\n",
            " Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by de\n",
            "feating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made\n",
            " eight appearances in the Super Bowl.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "cam newton \n",
            "\n",
            "Untokenized Answer :  cam newton\n",
            "\n",
            "real answer :  [{'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}, {'answer_start': 77, 'text': 'Cam Newton'}]\n",
            "\n",
            "Question :  What year did the the Saints hit a 13-0 record?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2009 \n",
            "\n",
            "Untokenized Answer :  2009\n",
            "\n",
            "real answer :  [{'answer_start': 591, 'text': '2009'}, {'answer_start': 591, 'text': '2009'}, {'answer_start': 591, 'text': '2009'}]\n",
            "\n",
            "Question :  Who won the Super Bowl MVP?\n",
            "--------------------------------------------------\n",
            "Context :  The Broncos took an early lead in Super Bowl 50 and never trailed. Newton was limited by Denver's defense, which sacked him seven times and forced him\n",
            " into three turnovers, including a fumble which they recovered for a touchdown. Denver linebacker Von Miller was named Super Bowl MVP, recording five \n",
            "solo tackles, 2½ sacks, and two forced fumbles.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "von miller \n",
            "\n",
            "Untokenized Answer :  von miller\n",
            "\n",
            "real answer :  [{'answer_start': 248, 'text': 'Von Miller'}, {'answer_start': 0, 'text': 'The Broncos'}, {'answer_start': 252, 'text': 'Miller'}]\n",
            "\n",
            "Question :  How many teams have won 15 regular season games since the 16-game schedule was adopted?\n",
            "--------------------------------------------------\n",
            "Context :  Despite waiving longtime running back DeAngelo Williams and losing top wide receiver Kelvin Benjamin to a torn ACL in the preseason, the Carolina Pant\n",
            "hers had their best regular season in franchise history, becoming the seventh team to win at least 15 regular season games since the league expanded t\n",
            "o a 16-game schedule in 1978. Carolina started the season 14–0, not only setting franchise records for the best start and the longest single-season wi\n",
            "nning streak, but also posting the best start to a season by an NFC team in NFL history, breaking the 13–0 record previously shared with the 2009 New \n",
            "Orleans Saints and the 2011 Green Bay Packers. With their NFC-best 15–1 regular season record, the Panthers clinched home-field advantage throughout t\n",
            "he NFC playoffs for the first time in franchise history. Ten players were selected to the Pro Bowl (the most in franchise history) along with eight Al\n",
            "l-Pro selections.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "seventh \n",
            "\n",
            "Untokenized Answer :  seventh\n",
            "\n",
            "real answer :  [{'answer_start': 326, 'text': '7'}, {'answer_start': 220, 'text': 'seventh'}, {'answer_start': 220, 'text': 'seventh'}]\n",
            "\n",
            "Question :  Where did Super Bowl 50 take place?\n",
            "--------------------------------------------------\n",
            "Context :  Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football\n",
            " Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super B\n",
            "owl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th S\n",
            "uper Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of nam\n",
            "ing each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently featur\n",
            "e the Arabic numerals 50.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 403, 'text': 'Santa Clara, California'}, {'answer_start': 355, 'text': \"Levi's Stadium\"}, {'answer_start': 355, 'text': \"Levi's Stadium in the San Francisco Bay Area at Santa Clara, California.\"}]\n",
            "\n",
            "Question :  When was the last Super Bowl in California?\n",
            "--------------------------------------------------\n",
            "Context :  On May 21, 2013, NFL owners at their spring meetings in Boston voted and awarded the game to Levi's Stadium. The $1.2 billion stadium opened in 2014. \n",
            "It is the first Super Bowl held in the San Francisco Bay Area since Super Bowl XIX in 1985, and the first in California since Super Bowl XXXVII took p\n",
            "lace in San Diego in 2003.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "2003 \n",
            "\n",
            "Untokenized Answer :  2003\n",
            "\n",
            "real answer :  [{'answer_start': 321, 'text': '2003'}, {'answer_start': 318, 'text': 'in 2003'}, {'answer_start': 321, 'text': '2003'}]\n",
            "\n",
            "Question :  What three stadiums did the NFL decide between for the game?\n",
            "--------------------------------------------------\n",
            "Context :  The league eventually narrowed the bids to three sites: New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area\n",
            "'s Levi's Stadium.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "new orleans ' mercedes - benz super ##dome , miami ' s sun life stadium , and the san francisco bay area ' s levi ' s stadium \n",
            "\n",
            "Untokenized Answer :  new orleans ' mercedes - benz superdome , miami ' s sun life stadium , and the san francisco bay area ' s levi ' s stadium\n",
            "\n",
            "real answer :  [{'answer_start': 56, 'text': \"New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area's Levi's Stadium\"}, {'answer_start': 56, 'text': \"New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area's Levi's Stadium.\"}, {'answer_start': 56, 'text': \"New Orleans' Mercedes-Benz Superdome, Miami's Sun Life Stadium, and the San Francisco Bay Area's Levi's Stadium.\"}]\n",
            "\n",
            "Question :  What is the best QB ranking that Cam Newton holds?\n",
            "--------------------------------------------------\n",
            "Context :  The Panthers offense, which led the NFL in scoring (500 points), was loaded with talent, boasting six Pro Bowl selections. Pro Bowl quarterback Cam Ne\n",
            "wton had one of his best seasons, throwing for 3,837 yards and rushing for 636, while recording a career-high and league-leading 45 total touchdowns (\n",
            "35 passing, 10 rushing), a career-low 10 interceptions, and a career-best quarterback rating of 99.4. Newton's leading receivers were tight end Greg O\n",
            "lsen, who caught a career-high 77 passes for 1,104 yards and seven touchdowns, and wide receiver Ted Ginn, Jr., who caught 44 passes for 739 yards and\n",
            " 10 touchdowns; Ginn also rushed for 60 yards and returned 27 punts for 277 yards. Other key receivers included veteran Jerricho Cotchery (39 receptio\n",
            "ns for 485 yards), rookie Devin Funchess (31 receptions for 473 yards and five touchdowns), and second-year receiver Corey Brown (31 receptions for 44\n",
            "7 yards). The Panthers backfield featured Pro Bowl running back Jonathan Stewart, who led the team with 989 rushing yards and six touchdowns in 13 gam\n",
            "es, along with Pro Bowl fullback Mike Tolbert, who rushed for 256 yards and caught 18 passes for another 154 yards. Carolina's offensive line also fea\n",
            "tured two Pro Bowl selections: center Ryan Kalil and guard Trai Turner.\n",
            "--------------------------------------------------\n",
            "ANSWER :  \n",
            "\n",
            "99 . 4 \n",
            "\n",
            "Untokenized Answer :  99 . 4\n",
            "\n",
            "real answer :  [{'answer_start': 396, 'text': '99.4'}, {'answer_start': 396, 'text': '99.4'}, {'answer_start': 396, 'text': '99.4.'}]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tf3ppjH9sTof",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}