colab_question_answering_BERT_base_cased_squad_v11_pt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "__3XJxZRzRq0"
   },
   "source": [
    "# NLP | Modelo de Question Answering em qualquer idioma baseado no BERT base (estudo de caso em português)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zwxTJ52nzRq8"
   },
   "source": [
    "- **Credit**: this notebook is a modified version of the notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb) of Hugging Face. Each new cell is signalized by **new** and each modificated one by **modified**.\n",
    "- **Acknowledgement**: Hugging Face, Neuralmind.ai, Deep Learning Brasil group, Google Colab and AI Lab ([see the whole list](https://medium.com/@pierre_guillou/nlp-modelo-de-question-answering-em-qualquer-idioma-baseado-no-bert-base-estudo-de-caso-em-12093d385e78#c572)).\n",
    "- **Author**: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou/)\n",
    "- **Date**: 12/02/2021\n",
    "- **Blog post**: [NLP | Modelo de Question Answering em qualquer idioma baseado no BERT base (estudo de caso em português)](https://medium.com/@pierre_guillou/nlp-modelo-de-question-answering-em-qualquer-idioma-baseado-no-bert-base-estudo-de-caso-em-12093d385e78)\n",
    "- **Model in the Model Hub of Hugging Face**: [Portuguese BERT base cased QA (Question Answering), finetuned on SQUAD v1.1](https://huggingface.co/pierreguillou/bert-base-cased-squad-v1.1-portuguese)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X4cRE8IbIrIV"
   },
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MOsHUjgdIrIW",
    "outputId": "9ff142f3-f815-4f80-f218-fe7ae3e0bdf4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting datasets\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/06/9b/d097f2238fc3c028495cf5f8c65378972b9f1b2cbb27f3c57c7219195aa9/datasets-1.2.1-py3-none-any.whl (159kB)\n",
      "\u001b[K     |████████████████████████████████| 163kB 8.4MB/s \n",
      "\u001b[?25hCollecting transformers\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/98/87/ef312eef26f5cecd8b17ae9654cdd8d1fae1eb6dbd87257d6d73c128a4d0/transformers-4.3.2-py3-none-any.whl (1.8MB)\n",
      "\u001b[K     |████████████████████████████████| 1.8MB 11.1MB/s \n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.6/dist-packages (from datasets) (1.19.5)\n",
      "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from datasets) (1.1.5)\n",
      "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.6/dist-packages (from datasets) (2.23.0)\n",
      "Requirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from datasets) (0.8)\n",
      "Collecting xxhash\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f7/73/826b19f3594756cb1c6c23d2fbd8ca6a77a9cd3b650c9dec5acc85004c38/xxhash-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl (242kB)\n",
      "\u001b[K     |████████████████████████████████| 245kB 34.3MB/s \n",
      "\u001b[?25hRequirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from datasets) (3.4.0)\n",
      "Requirement already satisfied: dill in /usr/local/lib/python3.6/dist-packages (from datasets) (0.3.3)\n",
      "Requirement already satisfied: multiprocess in /usr/local/lib/python3.6/dist-packages (from datasets) (0.70.11.1)\n",
      "Requirement already satisfied: tqdm<4.50.0,>=4.27 in /usr/local/lib/python3.6/dist-packages (from datasets) (4.41.1)\n",
      "Collecting pyarrow>=0.17.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/33/67/2f4fcce1b41bcc7e88a6bfdb42046597ae72e5bc95c2789b7c5ac893c433/pyarrow-3.0.0-cp36-cp36m-manylinux2014_x86_64.whl (20.7MB)\n",
      "\u001b[K     |████████████████████████████████| 20.7MB 1.5MB/s \n",
      "\u001b[?25hCollecting sacremoses\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
      "\u001b[K     |████████████████████████████████| 890kB 57.5MB/s \n",
      "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.9)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
      "Collecting tokenizers<0.11,>=0.10.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fd/5b/44baae602e0a30bcc53fbdbc60bd940c15e143d252d658dfdefce736ece5/tokenizers-0.10.1-cp36-cp36m-manylinux2010_x86_64.whl (3.2MB)\n",
      "\u001b[K     |████████████████████████████████| 3.2MB 56.4MB/s \n",
      "\u001b[?25hRequirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas->datasets) (2.8.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->datasets) (2018.9)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->datasets) (2020.12.5)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->datasets) (1.24.3)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->datasets) (3.0.4)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19.0->datasets) (2.10)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets) (3.4.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->datasets) (3.7.4.3)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.0.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
      "Building wheels for collected packages: sacremoses\n",
      "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893261 sha256=a41144d840cb76d097d124040e03efdcc5066b7c4430a5044ac1e5eb6cb356cc\n",
      "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
      "Successfully built sacremoses\n",
      "Installing collected packages: xxhash, pyarrow, datasets, sacremoses, tokenizers, transformers\n",
      "  Found existing installation: pyarrow 0.14.1\n",
      "    Uninstalling pyarrow-0.14.1:\n",
      "      Successfully uninstalled pyarrow-0.14.1\n",
      "Successfully installed datasets-1.2.1 pyarrow-3.0.0 sacremoses-0.0.43 tokenizers-0.10.1 transformers-4.3.2 xxhash-2.0.0\n"
     ]
    }
   ],
   "source": [
    "! pip install datasets transformers"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UMLWrks8-2LM",
    "outputId": "019f3963-aec1-40c2-b6b1-011def95c78f"
   },
   "source": [
    "import sys; print('python:',sys.version)\r\n",
    "\r\n",
    "print('transformers:',transformers.__version__)\r\n",
    "\r\n",
    "import torch; print('Pytorch:',torch.__version__)"
    ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "python: 3.6.9 (default, Oct  8 2020, 12:12:24) \n",
      "[GCC 8.4.0]\n",
      "transformers: 4.3.2\n",
      "Pytorch: 1.7.0+cu101\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rfV_Mk7MEH_M",
    "outputId": "44dd1579-c9f0-48de-bfe1-67cbf187a56e"
   },
   "source": [
    "!nvcc --version"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "nvcc: NVIDIA (R) Cuda compiler driver\n",
      "Copyright (c) 2005-2019 NVIDIA Corporation\n",
      "Built on Sun_Jul_28_19:07:16_PDT_2019\n",
      "Cuda compilation tools, release 10.1, V10.1.243\n"
     ],
     "name": "stdout"
    }
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5_q2wU0g-uMR",
    "outputId": "26c6868d-8bf5-46ad-e587-d35194bd6c7a"
   },
   "source": [
    "!nvidia-smi"
   ],
   "execution_count": null,
   "outputs": [
    {
     "output_type": "stream",
     "text": [
      "Tue Feb 16 17:54:49 2021       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 460.39       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   38C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "|  No running processes found                                                 |\n",
      "+-----------------------------------------------------------------------------+\n"
     ],
     "name": "stdout"
    }
   ]
  },  
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HFASsisvIrIb"
   },
   "source": [
    "If you're opening this notebook locally, make sure your environment has an install from the last version of those libraries.\n",
    "\n",
    "You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs [here](https://github.com/huggingface/transformers/tree/master/examples/question-answering)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rEJBSTyZIrIb"
   },
   "source": [
    "# Fine-tuning a model on a question-answering task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "frXPnqo8JzFG"
   },
   "source": [
    "In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) model to a question answering task, which is the task of extracting the answer to a question from a given context. We will see how to easily load a dataset for these kinds of tasks and use the `Trainer` API to fine-tune a model on it.\n",
    "\n",
    "![Widget inference representing the QA task](https://github.com/huggingface/notebooks/blob/master/examples/images/question_answering.png?raw=1)\n",
    "\n",
    "**Note:** This notebook finetunes models that answer question by taking a substring of a context, not by generating new text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4RRkXuteIrIh"
   },
   "source": [
    "This notebook is built to run on any question answering task with the same format as SQUAD (version 1 or 2), with any model checkpoint from the [Model Hub](https://huggingface.co/models) as long as that model has a version with a token classification head and a fast tokenizer (check on [this table](https://huggingface.co/transformers/index.html#bigtable) if this is the case). It might just need some small adjustments if you decide to use a different dataset than the one used here. Depending on you model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set those three parameters, then the rest of the notebook should run smoothly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FNudMyDMzRrB"
   },
   "outputs": [],
   "source": [
    "# modified \n",
    "\n",
    "# # This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n",
    "# # answers are allowed or not).\n",
    "# squad_v2 = False\n",
    "# model_checkpoint = \"distilbert-base-uncased\"\n",
    "# batch_size = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zVvslsfMIrIh"
   },
   "outputs": [],
   "source": [
    "# new\n",
    "\n",
    "# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n",
    "# answers are allowed or not).\n",
    "squad_v2 = False\n",
    "model_checkpoint = \"neuralmind/bert-base-portuguese-cased\"\n",
    "batch_size = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "whPRbBNbIrIl"
   },
   "source": [
    "## Loading the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W7QYTpxXIrIl"
   },
   "source": [
    "We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download the data and get the metric we need to use for evaluation (to compare our model to the benchmark). This can be easily done with the functions `load_dataset` and `load_metric`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IreSlFmlIrIm"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset, load_metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CKx2zKs5IrIq"
   },
   "source": [
    "For our example here, we'll use the [SQUAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook should work with any question answering dataset provided by the 🤗 Datasets library. If you're using your own dataset defined from a JSON or csv file (see the [Datasets documentation](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) on how to load them), it might need some adjustments in the names of the columns used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "s_AY1ATSIrIq"
   },
   "outputs": [],
   "source": [
    "# modified \n",
    "\n",
    "# datasets = load_dataset(\"squad_v2\" if squad_v2 else \"squad\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VIIUXab9Mxyr",
    "outputId": "13a3ba28-cc76-4f27-a2c1-f59762625b4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-02-12 10:34:45--  https://docs.google.com/uc?export=download&confirm=QDR6&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn\n",
      "Resolving docs.google.com (docs.google.com)... 74.125.142.100, 74.125.142.102, 74.125.142.138, ...\n",
      "Connecting to docs.google.com (docs.google.com)|74.125.142.100|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
      "Location: https://doc-0o-30-docs.googleusercontent.com/docs/securesc/6gcgjpf8a72cdguiblrqjvovod4pbhtu/eh1cp7e1hab1ojg87nqvqvqd3la45onu/1613126025000/03445611175480770093/01992478746163201708Z/1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn?e=download [following]\n",
      "--2021-02-12 10:34:45--  https://doc-0o-30-docs.googleusercontent.com/docs/securesc/6gcgjpf8a72cdguiblrqjvovod4pbhtu/eh1cp7e1hab1ojg87nqvqvqd3la45onu/1613126025000/03445611175480770093/01992478746163201708Z/1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn?e=download\n",
      "Resolving doc-0o-30-docs.googleusercontent.com (doc-0o-30-docs.googleusercontent.com)... 172.253.117.132, 2607:f8b0:400e:c0a::84\n",
      "Connecting to doc-0o-30-docs.googleusercontent.com (doc-0o-30-docs.googleusercontent.com)|172.253.117.132|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://docs.google.com/nonceSigner?nonce=1evp740c9jrfk&continue=https://doc-0o-30-docs.googleusercontent.com/docs/securesc/6gcgjpf8a72cdguiblrqjvovod4pbhtu/eh1cp7e1hab1ojg87nqvqvqd3la45onu/1613126025000/03445611175480770093/01992478746163201708Z/1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn?e%3Ddownload&hash=o8clcke3tp6tphp3jh96osumelhdjum9 [following]\n",
      "--2021-02-12 10:34:45--  https://docs.google.com/nonceSigner?nonce=1evp740c9jrfk&continue=https://doc-0o-30-docs.googleusercontent.com/docs/securesc/6gcgjpf8a72cdguiblrqjvovod4pbhtu/eh1cp7e1hab1ojg87nqvqvqd3la45onu/1613126025000/03445611175480770093/01992478746163201708Z/1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn?e%3Ddownload&hash=o8clcke3tp6tphp3jh96osumelhdjum9\n",
      "Connecting to docs.google.com (docs.google.com)|74.125.142.100|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://doc-0o-30-docs.googleusercontent.com/docs/securesc/6gcgjpf8a72cdguiblrqjvovod4pbhtu/eh1cp7e1hab1ojg87nqvqvqd3la45onu/1613126025000/03445611175480770093/01992478746163201708Z/1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn?e=download&nonce=1evp740c9jrfk&user=01992478746163201708Z&hash=jn6s224u8pbkbjbk1sanqvh1667bm2og [following]\n",
      "--2021-02-12 10:34:45--  https://doc-0o-30-docs.googleusercontent.com/docs/securesc/6gcgjpf8a72cdguiblrqjvovod4pbhtu/eh1cp7e1hab1ojg87nqvqvqd3la45onu/1613126025000/03445611175480770093/01992478746163201708Z/1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn?e=download&nonce=1evp740c9jrfk&user=01992478746163201708Z&hash=jn6s224u8pbkbjbk1sanqvh1667bm2og\n",
      "Connecting to doc-0o-30-docs.googleusercontent.com (doc-0o-30-docs.googleusercontent.com)|172.253.117.132|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: unspecified [application/gzip]\n",
      "Saving to: ‘squad-pt.tar.gz’\n",
      "\n",
      "squad-pt.tar.gz         [ <=>                ]  26.34M  --.-KB/s    in 0.1s    \n",
      "\n",
      "2021-02-12 10:34:46 (212 MB/s) - ‘squad-pt.tar.gz’ saved [27623048]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# new\n",
    "\n",
    "# Get dataset SQUAD in Portuguese\n",
    "!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn\" -O squad-pt.tar.gz && rm -rf /tmp/cookies.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "S4cOaDzLWViW",
    "outputId": "e7cb1047-8292-427b-cce4-901df58bbea0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "squad-train-v1.1.json\n",
      "squad-dev-v1.1.json\n"
     ]
    }
   ],
   "source": [
    "# new\n",
    "\n",
    "!tar -xvf squad-pt.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NkqM9gzvzRrE",
    "outputId": "9812a37c-ae5b-4bda-eaad-645bfff9d792"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 1s, sys: 339 ms, total: 1min 1s\n",
      "Wall time: 1min 1s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# new\n",
    "\n",
    "# Get the train and validation json file in the HF script format \n",
    "# inspiration: file squad.py at https://github.com/huggingface/datasets/tree/master/datasets/squad\n",
    "\n",
    "import json \n",
    "\n",
    "files = ['squad-train-v1.1.json','squad-dev-v1.1.json']\n",
    "\n",
    "for file in files:\n",
    "    \n",
    "    # Opening JSON file & returns JSON object as a dictionary \n",
    "    f = open(file, encoding=\"utf-8\") \n",
    "    data = json.load(f) \n",
    "    \n",
    "    # Iterating through the json list \n",
    "    entry_list = list()\n",
    "    id_list = list()\n",
    "\n",
    "    for row in data['data']: \n",
    "        title = row['title']\n",
    "        \n",
    "        for paragraph in row['paragraphs']:\n",
    "            context = paragraph['context']\n",
    "\n",
    "            for qa in paragraph['qas']:\n",
    "                entry = {}\n",
    "\n",
    "                qa_id = qa['id']\n",
    "                question = qa['question']\n",
    "                answers = qa['answers']\n",
    "                \n",
    "                entry['id'] = qa_id\n",
    "                entry['title'] = title.strip()\n",
    "                entry['context'] = context.strip()\n",
    "                entry['question'] = question.strip()\n",
    "                \n",
    "                answer_starts = [answer[\"answer_start\"] for answer in answers]\n",
    "                answer_texts = [answer[\"text\"].strip() for answer in answers]\n",
    "                entry['answers'] = {}\n",
    "                entry['answers']['answer_start'] = answer_starts\n",
    "                entry['answers']['text'] = answer_texts\n",
    "\n",
    "                entry_list.append(entry)\n",
    "                \n",
    "    reverse_entry_list = entry_list[::-1]\n",
    "    \n",
    "    # for entries with same id, keep only last one (corrected texts by he group Deep Learning Brasil)\n",
    "    unique_ids_list = list()\n",
    "    unique_entry_list = list()\n",
    "    for entry in reverse_entry_list:\n",
    "        qa_id = entry['id']\n",
    "        if qa_id not in unique_ids_list:\n",
    "            unique_ids_list.append(qa_id)\n",
    "            unique_entry_list.append(entry)\n",
    "        \n",
    "    # Closing file \n",
    "    f.close() \n",
    "\n",
    "    new_dict = {}\n",
    "    new_dict['data'] = unique_entry_list\n",
    "\n",
    "    file_name = 'pt_' + str(file)\n",
    "    with open(file_name, 'w') as json_file:\n",
    "        json.dump(new_dict, json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 139,
     "referenced_widgets": [
      "f9c22d5144824c518fad4ec9e0acc2b1",
      "b4962780811d4ecd9a80b8ad2703cc0b",
      "8a51db15df8f48e197cc8cb717325044",
      "bc63c8a02c1040f68c28825b5acfd60c",
      "94ca5160b56145ea91b95e69ca47dce7",
      "21e75a90d0cb4d778e7bee589c1e375b",
      "d58718e51dbd4dc4bf7dd9ef6bde7951",
      "2246016d810d4f25b5cf9e696252752a",
      "8031f77075de48c6bdcf2a879f404fec",
      "e3bc9a84092049009c39758400895ff8",
      "c1e3222db9bb4aa0a7a4e895e1e69eac",
      "bb2b7b66bbf64ee290774aa96c0f4c00",
      "0a38ad48054b4793af78af528164f84d",
      "78cc7c33fc664fb8b45019a2d8d1a81c",
      "f04b87a736a5439998716ec2dc6ba670",
      "20aa40592cb7449d93cd22c37274e621",
      "36b27f842246446d951e90d22ceece63",
      "5404a9f0aab54897b2c6e2eeaaa5c636",
      "6ded78516c3d4d5ba0aca74f89ac930e",
      "ad3303422ce84d8597a9291dba86c0fc",
      "4fa075949df84e0390bfdfc4cb87fc19",
      "fcf5c3c66b9b49258d0f1769d3081dd1",
      "06d7277c8e574430a065a9a85f88eea6",
      "3f8faf6e856c460b8bcde6fe430d1c74"
     ]
    },
    "id": "NrvZrQYFYDSE",
    "outputId": "5c248f44-d88e-4467-e0aa-db0430309e73"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f9c22d5144824c518fad4ec9e0acc2b1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1202.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading and preparing dataset json/default-95645863fdba248c (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/json/default-95645863fdba248c/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8031f77075de48c6bdcf2a879f404fec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36b27f842246446d951e90d22ceece63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/default-95645863fdba248c/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "# new\n",
    "\n",
    "from datasets import load_dataset, load_metric\n",
    "datasets = load_dataset('json', \n",
    "                        data_files={'train': 'pt_squad-train-v1.1.json', 'validation': 'pt_squad-dev-v1.1.json'}, \n",
    "                        field='data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RzfPtOMoIrIu"
   },
   "source": [
    "The `datasets` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GWiVUF0jIrIv",
    "outputId": "2aff36e3-bc2d-4eed-deb2-008d2c2903e2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'title', 'context', 'question', 'answers'],\n",
       "        num_rows: 87510\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['id', 'title', 'context', 'question', 'answers'],\n",
       "        num_rows: 10570\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5vVFhEVNJzFJ"
   },
   "source": [
    "We can see the training, validation and test sets all have a column for the context, the question and the answers to those questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u3EtYfeHIrIz"
   },
   "source": [
    "To access an actual element, you need to select a split first, then give an index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X6HrpprwIrIz",
    "outputId": "d7670bc0-42e4-4c09-8a6a-5c018ded7d95"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answers': {'answer_start': [515], 'text': ['Saint Bernadette Soubirous']},\n",
       " 'context': 'Architecturally, the school has a Catholic character. Atop the Main Building\\'s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.',\n",
       " 'id': '5733be284776f41900661182',\n",
       " 'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',\n",
       " 'title': 'University_of_Notre_Dame'}"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# modified\n",
    "\n",
    "# datasets[\"train\"][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "A2dqv_QgzRrH",
    "outputId": "7b1be770-3d94-4207-c7a3-575517dce285"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answers': {'answer_start': [2],\n",
       "  'text': ['Cidade Metropolitana de Catmandu']},\n",
       " 'context': 'A Cidade Metropolitana de Catmandu (KMC), a fim de promover as relações internacionais, criou uma Secretaria de Relações Internacionais (IRC). O primeiro relacionamento internacional da KMC foi estabelecido em 1975 com a cidade de Eugene, Oregon, Estados Unidos. Essa atividade foi aprimorada ainda mais com o estabelecimento de relações formais com outras 8 cidades: Cidade de Motsumoto, Japão, Rochester, EUA, Yangon (antiga Rangum) de Mianmar, Xian da República Popular da China, Minsk da Bielorrússia e Pyongyang de República Democrática da Coréia. O esforço constante da KMC é aprimorar sua interação com os países da SAARC, outras agências internacionais e muitas outras grandes cidades do mundo para alcançar melhores programas de gestão urbana e desenvolvimento para Katmandu.',\n",
       " 'id': '5735d259012e2f140011a0a1',\n",
       " 'question': 'De que KMC é um inicialismo?',\n",
       " 'title': 'Kathmandu'}"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new\n",
    "\n",
    "datasets[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3FELx4kPJzFN"
   },
   "source": [
    "We can see the answers are indicated by their start position in the text (here at character 515) and their full text, which is a substring of the context as we mentioned above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WHUmphG3IrI3"
   },
   "source": [
    "To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset (automatically decoding the labels in passing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "i3j8APAoIrI3"
   },
   "outputs": [],
   "source": [
    "from datasets import ClassLabel, Sequence\n",
    "import random\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "def show_random_elements(dataset, num_examples=10):\n",
    "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
    "    picks = []\n",
    "    for _ in range(num_examples):\n",
    "        pick = random.randint(0, len(dataset)-1)\n",
    "        while pick in picks:\n",
    "            pick = random.randint(0, len(dataset)-1)\n",
    "        picks.append(pick)\n",
    "    \n",
    "    df = pd.DataFrame(dataset[picks])\n",
    "    for column, typ in dataset.features.items():\n",
    "        if isinstance(typ, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
    "        elif isinstance(typ, Sequence) and isinstance(typ.feature, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda x: [typ.feature.names[i] for i in x])\n",
    "    display(HTML(df.to_html()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "SZy5tRB_IrI7",
    "outputId": "c12b432b-3d3e-4f36-c664-d8d8d448fb39",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>56de3e40cffd8e1900b4b6dc</td>\n",
       "      <td>Separation_of_powers_under_the_United_States_Constitution</td>\n",
       "      <td>O presidente exerce um controle sobre o Congresso através de seu poder de vetar projetos de lei, mas o Congresso pode substituir qualquer veto (excluindo o chamado \"veto de bolso\") por maioria de dois terços em cada casa. Quando as duas casas do Congresso não chegarem a um prazo de adiamento, o presidente poderá resolver a disputa. Qualquer casa ou ambas as casas podem ser convocadas para uma sessão de emergência pelo presidente. O vice-presidente atua como presidente do Senado, mas ele só pode votar para romper o empate.</td>\n",
       "      <td>Quem pode determinar uma data de adiamento se o congresso não concordar?</td>\n",
       "      <td>{'answer_start': [422], 'text': ['presidente']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>572f871db2c2fd14005681c0</td>\n",
       "      <td>Armenia</td>\n",
       "      <td>Durante o final do século VI aC, a primeira entidade geográfica chamada Armênia pelas populações vizinhas foi estabelecida sob a dinastia Orontid, dentro do Império Aquemênida, como parte dos territórios dos últimos. O reino tornou-se totalmente soberano da esfera de influência do Império Selêucida em 190 aC, sob o rei Artaxias I, e iniciou o governo da dinastia Artaxiad. A Armênia atingiu seu auge entre 95 e 66 aC sob Tigranes, o Grande, tornando-se o reino mais poderoso de seu tempo a leste da República Romana.</td>\n",
       "      <td>Qual governante é responsável pela prosperidade de Armênia?</td>\n",
       "      <td>{'answer_start': [423], 'text': ['Tigranes, o Grande']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5707feb09e06ca38007e94d9</td>\n",
       "      <td>Imamah_(Shia_doctrine)</td>\n",
       "      <td>Os xiitas acreditam que Imamah é dos Princípios da Fé (Usul al-Din). Como o versículo 4: 165 do Alcorão expressa a necessidade da nomeação dos profetas; assim, após a morte do profeta, que desempenhará o papel do profeta; até que o povo não tenha nenhum argumento contra Allah. Portanto, a mesma lógica que exigia a designação de profetas também é aplicada a Imama. Isso é Allah Deve designar alguém semelhante ao profeta em seus atributos e Ismah como seu sucessor para guiar o povo sem nenhum desvio. na religião. Eles se referem ao versículo (... Neste dia eu aperfeiçoei para você sua religião e completei Meu favor sobre você e aprovei para você o Islã como religião ...) 5: 3 do Alcorão que foi revelado ao profeta quando ele designou Ali como seu sucessor no dia de Ghadir Khumm.</td>\n",
       "      <td>Quem acredita que Imamah é dos Princípios da Fé?</td>\n",
       "      <td>{'answer_start': [3], 'text': ['xiitas']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56f8fba19b226e1400dd1231</td>\n",
       "      <td>Near_East</td>\n",
       "      <td>Posteriormente, com a desgraça do \"Próximo Oriente\" nos círculos diplomáticos e militares, o \"Oriente Médio\" prevaleceu. No entanto, o \"Oriente Próximo\" continua em alguns círculos, a critério da agência ou departamento acadêmico de definição. Eles geralmente não são considerados regiões distintas, como estavam em sua definição original.</td>\n",
       "      <td>O que continua em alguns círculos, a critério da agência ou departamento acadêmico definidor?</td>\n",
       "      <td>{'answer_start': [136], 'text': ['Oriente Próximo']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>56f72a10711bf01900a44a16</td>\n",
       "      <td>Classical_music</td>\n",
       "      <td>Embora o temperamento igual tenha sido gradualmente aceito como o temperamento musical dominante durante o século 18, diferentes temperamentos históricos são frequentemente usados para a música de períodos anteriores. Por exemplo, a música do Renascimento inglês é frequentemente tocada em temperamento intencional.</td>\n",
       "      <td>Que tipo de temperamento foi aceito no século XVIII?</td>\n",
       "      <td>{'answer_start': [9], 'text': ['temperamento igual']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>56fb7c108ddada1400cd6458</td>\n",
       "      <td>Middle_Ages</td>\n",
       "      <td>No centro e no norte da Itália e na Flandres, a ascensão de cidades autônomas estimulou o crescimento econômico e criou um ambiente para novos tipos de associações comerciais. As cidades comerciais às margens do Báltico entraram em acordos conhecidos como Liga Hanseática, e as repúblicas marítimas italianas como Veneza, Génova e Pisa expandiram seu comércio por todo o Mediterrâneo. [V] Grandes feiras comerciais foram estabelecidas e floresceram no norte da França. durante o período, permitindo que comerciantes italianos e alemães negociassem entre si, bem como comerciantes locais. No final do século 13, novas rotas terrestres e marítimas para o Extremo Oriente foram pioneiras, famosamente descritas em As Viagens de Marco Polo, escrita por um dos comerciantes, Marco Polo (m. 1324). Além de novas oportunidades comerciais, as melhorias agrícolas e tecnológicas permitiram um aumento no rendimento das culturas, o que, por sua vez, permitiu a expansão das redes comerciais. O aumento do comércio trouxe novos métodos de lidar com dinheiro, e a cunhagem de ouro foi novamente cunhada na Europa, primeiro na Itália e depois na França e em outros países. Novas formas de contratos comerciais surgiram, permitindo que os riscos fossem compartilhados entre os comerciantes. Os métodos contábeis foram aprimorados, em parte através do uso da contabilidade de dupla entrada; também apareceram cartas de crédito, permitindo fácil transmissão de dinheiro.</td>\n",
       "      <td>Juntamente com Veneza e Pisa, o que era uma notável república marítima italiana?</td>\n",
       "      <td>{'answer_start': [322], 'text': ['Génova']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>56d43f7e2ccc5a1400d830c7</td>\n",
       "      <td>Beyoncé</td>\n",
       "      <td>Uma autodeclarada \"feminista moderna\", Beyoncé cria músicas que geralmente são caracterizadas por temas de amor, relacionamentos e monogamia, além de sexualidade e empoderamento feminino. No palco, suas performances dinâmicas e altamente coreografadas levaram os críticos a saudá-la como uma das melhores artistas da música popular contemporânea. Ao longo de uma carreira de 19 anos, ela vendeu mais de 118 milhões de discos como artista solo e mais 60 milhões com Destiny's Child, tornando-a uma das artistas musicais mais vendidas de todos os tempos. Ela ganhou 20 prêmios Grammy e é a mulher mais indicada na história do prêmio. A Associação da Indústria de Gravação da América a reconheceu como a Melhor Artista Certificada da América durante a década de 2000. Em 2009, a Billboard a nomeou a Melhor Artista de Canções de Rádio da Década, a Melhor Artista Feminina dos anos 2000 e sua Artista do Milênio em 2011. A Time a listou entre as 100 pessoas mais influentes do mundo em 2013 e 2014. Revista Forbes também a listou como a musica feminina mais poderosa de 2015.</td>\n",
       "      <td>Quantos discos Beyoncé vendeu como parte de Destiny's Child?</td>\n",
       "      <td>{'answer_start': [450], 'text': ['60 milhões']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>57301d42947a6a140053d12b</td>\n",
       "      <td>Georgian_architecture</td>\n",
       "      <td>Até a Lei de Construção de Igrejas de 1818, o período viu relativamente poucas igrejas construídas na Grã-Bretanha, o que já era bem suprido, embora nos últimos anos a demanda por locais de culto não conformista e católico romano tenha aumentado bastante. As igrejas anglicanas que foram construídas foram projetadas internamente para permitir a máxima audibilidade e visibilidade para a pregação, de modo que a nave principal era geralmente mais larga e mais curta do que nos planos medievais, e muitas vezes não havia corredores laterais. Galerias eram comuns em novas igrejas. Especialmente nas paróquias do interior, a aparência externa geralmente mantinha os significantes familiares de uma igreja gótica, com uma torre ou torre, uma grande frente oeste com uma ou mais portas e janelas muito grandes ao longo da nave, mas todas com qualquer ornamento extraído da igreja. vocabulário clássico. Onde os fundos permitirem, um pórtico clássico do templo com colunas e frontão pode ser usado na frente oeste. A decoração interna era muito limitada, mas as igrejas se enchiam de monumentos para os prósperos.</td>\n",
       "      <td>O que era comum em novas igrejas?</td>\n",
       "      <td>{'answer_start': [541], 'text': ['Galerias']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>57274c69f1498d1400e8f5ef</td>\n",
       "      <td>Affirmative_action_in_the_United_States</td>\n",
       "      <td>Em 1976, um grupo de professores ítalo-americanos da City University de Nova York pediu para ser adicionado como uma categoria de ação afirmativa para promoção e contratação. Os ítalo-americanos são geralmente considerados brancos nos EUA e não seriam cobertos por políticas de ação afirmativa, mas os professores acreditavam que estavam sub-representados. O economista libertário Thomas Sowell escreveu em seu livro Ação afirmativa ao redor do mundo: um estudo empírico, que as políticas de ação afirmativa incentivam grupos não preferenciais a se designarem como membros de grupos preferenciais [isto é, os principais beneficiários da ação afirmativa] para tirar vantagem do grupo políticas de preferência.</td>\n",
       "      <td>Qual foi o título do livro escrito por Thomas Sowell?</td>\n",
       "      <td>{'answer_start': [417], 'text': ['Ação afirmativa ao redor do mundo: um estudo empírico']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>57062dc675f01819005e7a2c</td>\n",
       "      <td>Atlantic_City,_New_Jersey</td>\n",
       "      <td>Com o redesenvolvimento de Las Vegas e a abertura de dois cassinos em Connecticut no início dos anos 90, junto com os cassinos recém-construídos na área metropolitana da Filadélfia, nos anos 2000, o turismo de Atlantic City começou a declinar devido à sua falta de diversificação dos jogos. Determinada a expandir, em 1999, a Atlantic City Redevelopment Authority estabeleceu uma parceria com o magnata do cassino de Las Vegas, Steve Wynn, para desenvolver uma nova estrada para uma seção árida da cidade perto da Marina. Apelidado de \"The Tunnel Project\", Steve Wynn planejou a proposta de 'Mirage Atlantic City' com base na idéia de que ele conectaria o túnel de US $ 330 milhões que se estende por 4,0 km da via expressa de Atlantic City a seu novo resort. A estrada foi mais tarde nomeada oficialmente como Atlantic City-Brigantine Connector e canaliza o tráfego de entrada da via expressa para o distrito marina da cidade e Brigantine, Nova Jersey.</td>\n",
       "      <td>\"The Tunnel Project\" mais tarde foi oficialmente chamado de quê?</td>\n",
       "      <td>{'answer_start': [811], 'text': ['Atlantic City-Brigantine Connector']}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_random_elements(datasets[\"train\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n9qywopnIrJH"
   },
   "source": [
    "## Preprocessing the training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YVx71GdAIrJH"
   },
   "source": [
    "Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.\n",
    "\n",
    "To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
    "\n",
    "- we get a tokenizer that corresponds to the model architecture we want to use,\n",
    "- we download the vocabulary used when pretraining this specific checkpoint.\n",
    "\n",
    "That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 264,
     "referenced_widgets": [
      "e32ea0f0df12457fb401e89457a4df48",
      "a7b82c77d9d94135bd5f2977585670af",
      "84edc8ef4721471c823c4502c503d923",
      "12742631aec9417881e3e7c471b1dd77",
      "1baccf373ec24ac2ae16d761479e9944",
      "f44bf3eef0e840f99f982fbf0910d108",
      "f224c0c64f994806a13847d32c34f714",
      "06dae62919924bc0a560b0dc8185cafe",
      "d55529a706484c28bf664f3757575729",
      "551640d85a3b42f5b8b75c184cbfed45",
      "ad995c2518444bc2b8d64289d41edc0d",
      "4dc1e1d8ff9c434d81fc0cab52fefbcf",
      "11fffa69385e429f82a472a7dc2f8dba",
      "4c48350b493f4edea02d8b2d3f2608e8",
      "7d2eceabf1784b219db412785212c4b5",
      "0d84ebc014b54a1b8fb1b35ea917e36c",
      "972d29583fba4f8a9fa95181728c0aba",
      "c742e996d874416184c0699b242cfd05",
      "0722ba5db9a940cba9fc3ec42b42efc8",
      "6fc6cf356dbb47e2b69e42cf205ab824",
      "684135d8e3a041559a98fac8af25d72d",
      "feaef1aea58e446e847a45d38d8366c9",
      "b34d85775ba14711a038d153c97d4802",
      "35c8fdad45674aec8fca9007ac63242f",
      "173fe205f1a141af9466938cba0ec9b3",
      "abc26b3c8a044f18896650f1c4f36d8b",
      "cdc77948b42d40f698b1b150b2eca9f4",
      "06d0cdf758404a99b3bfa0df48c9e348",
      "cf1b41ef505646738f705629b40b1c5d",
      "22091b5d454f44e693ee9f0a0396a447",
      "56abcb61373e4d9e976a6dc6cd1560da",
      "204e637901534b95b6ec73b9d6519c26",
      "45cfc92ef71d4dcb8d562e6eede7e29d",
      "d7fdbf13549240879be55b7f0b8ddbff",
      "936b07ba5d2c4cf59d217cf99a3698b2",
      "bdc3db4828ce497b9f9cff6254d74acd",
      "208818f9241f4b298ebc1e756c8b6faf",
      "e6f0a509db51421793264b529cbf0d77",
      "c684af3b9ea041d48a8f9ca73a17e9b2",
      "c0c242ccad4d454c8b7ce6152aeeb04b"
     ]
    },
    "id": "eXNLu_-nIrJI",
    "outputId": "31e32842-4e94-4808-f4f3-18992e48a855"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e32ea0f0df12457fb401e89457a4df48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=647.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d55529a706484c28bf664f3757575729",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=209528.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "972d29583fba4f8a9fa95181728c0aba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=2.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "173fe205f1a141af9466938cba0ec9b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=112.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "45cfc92ef71d4dcb8d562e6eede7e29d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=43.0, style=ProgressStyle(description_w…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "    \n",
    "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vl6IidfdIrJK"
   },
   "source": [
    "The following assertion ensures that our tokenizer is a fast tokenizers (backed by Rust) from the 🤗 Tokenizers library. Those fast tokenizers are available for almost all models, and we will need some of the special features they have for our preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BYqaoVeGJzFQ"
   },
   "outputs": [],
   "source": [
    "import transformers\n",
    "assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A6VXE3g4JzFQ"
   },
   "source": [
    "You can check which type of models have a fast tokenizer available and which don't on the [big table of models](https://huggingface.co/transformers/index.html#bigtable)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rowT4iCLIrJK"
   },
   "source": [
    "You can directly call this tokenizer on two sentences (one for the answer, one for the context):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a5hBlsrHIrJL",
    "outputId": "2f3024af-f79c-498d-e633-565570e7ce89"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 1602, 176, 3196, 136, 102, 3396, 311, 849, 22280, 16775, 256, 147, 119, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new\n",
    "\n",
    "# tokenizer(\"What is your name?\", \"My name is Sylvain.\")\n",
    "tokenizer(\"Como se chama?\", \"Eu me chamo Sylvain.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j9DS8PEmJzFR"
   },
   "source": [
    "Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in [this tutorial](https://huggingface.co/transformers/preprocessing.html) if you're interested.\n",
    "\n",
    "Now one specific thing for the preprocessing in question answering is how to deal with very long documents. We usually truncate them in other tasks, when they are longer than the model maximum sentence length, but here, removing part of the the context might result in losing the answer we are looking for. To deal with this, we will allow one (long) example in our dataset to give several input features, each of length shorter than the maximum length of the model (or the one we set as a hyper-parameter). Also, just in case the answer lies at the point we split a long context, we allow some overlap between the features we generate controlled by the hyper-parameter `doc_stride`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kGM6TNXFJzFS"
   },
   "outputs": [],
   "source": [
    "max_length = 384 # The maximum length of a feature (question and context)\n",
    "doc_stride = 128 # The authorized overlap between two part of the context when splitting it is needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GmDb0AmAJzFS"
   },
   "source": [
    "Let's find one long example in our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LRx8jfi_JzFS"
   },
   "outputs": [],
   "source": [
    "for i, example in enumerate(datasets[\"train\"]):\n",
    "    if len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"]) > 384:\n",
    "        break\n",
    "example = datasets[\"train\"][i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zNq6fOi44Mj8",
    "outputId": "4e6cba65-a1cc-4f92-a925-67c898e921b5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answers': {'answer_start': [512], 'text': ['Junho a agosto']},\n",
       " 'context': 'A cidade geralmente tem um clima com dias quentes, seguido de noites e manhãs frias. Prevê-se um clima imprevisível, uma vez que as temperaturas podem cair para 1 ° C (34 ° F) ou menos durante o inverno. Durante uma frente fria de 2013, as temperaturas de inverno de Katmandu caíram para -4 ° C (25 ° F), e a temperatura mais baixa foi registrada em 10 de janeiro de 2013, a -9,2 ° C (15,4 ° F). As chuvas são principalmente baseadas em monções (cerca de 65% do total concentrado durante os meses das monções de Junho a agosto) e diminuem substancialmente (100 a 200 cm) do leste do Nepal ao oeste do Nepal. As chuvas foram registradas em cerca de 1.400 milímetros (55,1 polegadas) no vale de Katmandu e em média 1.407 milímetros (55,4 polegadas) na cidade de Katmandu. Em média, a umidade é de 75%. O gráfico abaixo é baseado em dados do Bureau de Padrões e Meteorologia do Nepal, \"Meteorologia Meteorológica\" de 2005. O gráfico fornece temperaturas mínimas e máximas durante cada mês. A quantidade anual de precipitação foi de 1.124 milímetros (44,3 polegadas) em 2005, conforme dados mensais incluídos na tabela acima. A década de 2000-2010 viu anomalias de precipitação altamente variáveis e sem precedentes em Katmandu. Isso ocorreu principalmente devido à variação anual das monções do sudoeste. [Citação necessário] Por exemplo, 2003 foi o ano mais chuvoso de todos os tempos em Katmandu, totalizando mais de 2.900 mm (114 in) de precipitação devido a uma estação de monções excepcionalmente forte. Em contraste, 2001 registrou apenas 356 mm (14 pol) de precipitação devido a uma estação de monções extraordinariamente fraca.',\n",
       " 'id': '5735aaaedc94161900571ef6',\n",
       " 'question': 'Em que meses a monção ocorre no Nepal?',\n",
       " 'title': 'Kathmandu'}"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new\n",
    "\n",
    "example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9WqXSqRxJzFS"
   },
   "source": [
    "Without any truncation, we get the following length for the input IDs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "u6HCftuAJzFS",
    "outputId": "5790f4c6-98e2-4b5e-e8ea-4af7dd050dd1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "407"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L0-BLo9KJzFT"
   },
   "source": [
    "Now, if we just truncate, we will lose information (and possibly the answer to our question):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FueJrxeZJzFT",
    "outputId": "fd9ce232-a13a-41a4-a5f8-703da4192abd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokenizer(example[\"question\"], example[\"context\"], max_length=max_length, truncation=\"only_second\")[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Dj0YzrvJzFT"
   },
   "source": [
    "Note that we never want to truncate the question, only the context, else the `only_second` truncation picked. Now, our tokenizer can automatically return us a list of features capped by a certain maximum length, with the overlap we talked above, we just have to tell it with `return_overflowing_tokens=True` and by passing the stride:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iw6OQ1N4JzFU"
   },
   "outputs": [],
   "source": [
    "tokenized_example = tokenizer(\n",
    "    example[\"question\"],\n",
    "    example[\"context\"],\n",
    "    max_length=max_length,\n",
    "    truncation=\"only_second\",\n",
    "    return_overflowing_tokens=True,\n",
    "    stride=doc_stride\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7hP_LccQJzFU"
   },
   "source": [
    "Now we don't have one list of `input_ids`, but several: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Tks4gYoUJzFU",
    "outputId": "4963f954-3bab-4e4e-cb00-3b5d592c1381"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[384, 165]"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[len(x) for x in tokenized_example[\"input_ids\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GN_GMrg9JzFV"
   },
   "source": [
    "And if we decode them, we can see the overlap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yGZ_xtmhJzFV",
    "outputId": "b8369908-4833-4c12-ee27-3137fe71db50"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CLS] Em que meses a monção ocorre no Nepal? [SEP] A cidade geralmente tem um clima com dias quentes, seguido de noites e manhãs frias. Prevê - se um clima imprevisível, uma vez que as temperaturas podem cair para 1 ° C ( 34 ° F ) ou menos durante o inverno. Durante uma frente fria de 2013, as temperaturas de inverno de Katmandu caíram para - 4 ° C ( 25 ° F ), e a temperatura mais baixa foi registrada em 10 de janeiro de 2013, a - 9, 2 ° C ( 15, 4 ° F ). As chuvas são principalmente baseadas em monções ( cerca de 65 % do total concentrado durante os meses das monções de Junho a agosto ) e diminuem substancialmente ( 100 a 200 cm ) do leste do Nepal ao oeste do Nepal. As chuvas foram registradas em cerca de 1. 400 milímetros ( 55, 1 polegadas ) no vale de Katmandu e em média 1. 407 milímetros ( 55, 4 polegadas ) na cidade de Katmandu. Em média, a umidade é de 75 %. O gráfico abaixo é baseado em dados do Bureau de Padrões e Meteorologia do Nepal, \" Meteorologia Meteorológica \" de 2005. O gráfico fornece temperaturas mínimas e máximas durante cada mês. A quantidade anual de precipitação foi de 1. 124 milímetros ( 44, 3 polegadas ) em 2005, conforme dados mensais incluídos na tabela acima. A década de 2000 - 2010 viu anomalias de precipitação altamente variáveis e sem precedentes em Katmandu. Isso ocorreu principalmente devido à variação anual das monções do sudoeste. [ Citação necessário ] Por exemplo, 2003 foi o ano mais chuvoso de todos os tempos em Katmandu, totalizando mais de 2. 900 mm ( 114 in ) de precipitação devido a uma estação de monções excepcionalmente forte. Em contraste, 2001 registrou [SEP]\n",
      "[CLS] Em que meses a monção ocorre no Nepal? [SEP] fornece temperaturas mínimas e máximas durante cada mês. A quantidade anual de precipitação foi de 1. 124 milímetros ( 44, 3 polegadas ) em 2005, conforme dados mensais incluídos na tabela acima. A década de 2000 - 2010 viu anomalias de precipitação altamente variáveis e sem precedentes em Katmandu. Isso ocorreu principalmente devido à variação anual das monções do sudoeste. [ Citação necessário ] Por exemplo, 2003 foi o ano mais chuvoso de todos os tempos em Katmandu, totalizando mais de 2. 900 mm ( 114 in ) de precipitação devido a uma estação de monções excepcionalmente forte. Em contraste, 2001 registrou apenas 356 mm ( 14 pol ) de precipitação devido a uma estação de monções extraordinariamente fraca. [SEP]\n"
     ]
    }
   ],
   "source": [
    "for x in tokenized_example[\"input_ids\"][:2]:\n",
    "    print(tokenizer.decode(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v7M3ZBTLJzFV"
   },
   "source": [
    "Now this will give us some work to properly treat the answers: we need to find in which of those features the answer actually is, and where exactly in that feature. The models we will use require the start and end positions of these answers in the tokens, so we will also need to to map parts of the original context to some tokens. Thankfully, the tokenizer we're using can help us with that by returning an `offset_mapping`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z3CAD0SDJzFV",
    "outputId": "3405263c-70c8-4b1c-996b-a1af0c5bc2c6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, 0), (0, 2), (3, 6), (7, 12), (13, 14), (15, 18), (18, 21), (22, 28), (29, 31), (32, 34), (34, 37), (37, 38), (0, 0), (0, 1), (2, 8), (9, 19), (20, 23), (24, 26), (27, 32), (33, 36), (37, 41), (42, 49), (49, 50), (51, 58), (59, 61), (62, 68), (69, 70), (71, 76), (76, 77), (78, 82), (82, 83), (83, 84), (85, 88), (88, 90), (90, 91), (91, 93), (94, 96), (97, 102), (103, 108), (108, 115), (115, 116), (117, 120), (121, 124), (125, 128), (129, 131), (132, 144), (145, 150), (151, 155), (156, 160), (161, 162), (163, 164), (165, 166), (167, 168), (168, 170), (171, 172), (173, 174), (174, 175), (176, 178), (179, 184), (185, 192), (193, 194), (195, 202), (202, 203), (204, 211), (212, 215), (216, 222), (223, 227), (228, 230), (231, 235), (235, 236), (237, 239), (240, 252), (253, 255), (256, 263), (264, 266), (267, 270), (270, 274), (274, 275), (276, 282), (283, 287), (288, 289), (289, 290), (291, 292), (293, 294), (295, 296), (296, 298), (299, 300), (301, 302), (302, 303), (303, 304), (305, 306), (307, 308), (309, 320), (321, 325), (326, 331), (332, 335), (336, 346), (347, 349), (350, 352), (353, 355)]\n"
     ]
    }
   ],
   "source": [
    "tokenized_example = tokenizer(\n",
    "    example[\"question\"],\n",
    "    example[\"context\"],\n",
    "    max_length=max_length,\n",
    "    truncation=\"only_second\",\n",
    "    return_overflowing_tokens=True,\n",
    "    return_offsets_mapping=True,\n",
    "    stride=doc_stride\n",
    ")\n",
    "print(tokenized_example[\"offset_mapping\"][0][:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P-77RrvOJzFW"
   },
   "source": [
    "This gives, for each index of our input IDS, the corresponding start and end character in the original text that gave our token. The very first token (`[CLS]`) has (0, 0) because it doesn't correspond to any part of the question/answer, then the second token is the same as the characters 0 to 3 of the question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Pl98Zd4sJzFW",
    "outputId": "c3b90d92-c8b2-4e59-9bb0-25160710ec0b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Em Em\n"
     ]
    }
   ],
   "source": [
    "first_token_id = tokenized_example[\"input_ids\"][0][1]\n",
    "offsets = tokenized_example[\"offset_mapping\"][0][1]\n",
    "print(tokenizer.convert_ids_to_tokens([first_token_id])[0], example[\"question\"][offsets[0]:offsets[1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lfLpSn-pJzFW"
   },
   "source": [
    "So we can use this mapping to find the position of the start and end tokens of our answer in a given feature. We just have to distinguish which parts of the offsets correspond to the question and which part correspond to the context, this is where the `sequence_ids` method of our `tokenized_example` can be useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dZ3-2IUQJzFW",
    "outputId": "4a8a0338-2461-4339-a8d7-437aa5044afd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[None, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, None, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, None]\n"
     ]
    }
   ],
   "source": [
    "sequence_ids = tokenized_example.sequence_ids()\n",
    "print(sequence_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QwHedbQ2JzFX"
   },
   "source": [
    "It returns `None` for the special tokens, then 0 or 1 depending on whether the corresponding token comes from the first sentence past (the question) or the second (the context). Now with all of this, we can find the first and last token of the answer in one of our input feature (or if the answer is not in this feature):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Ltduc4QcJzFX",
    "outputId": "76a30104-bd91-4e8b-b60c-e3a1c674fd19"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "143 145\n"
     ]
    }
   ],
   "source": [
    "answers = example[\"answers\"]\n",
    "start_char = answers[\"answer_start\"][0]\n",
    "end_char = start_char + len(answers[\"text\"][0])\n",
    "\n",
    "# Start token index of the current span in the text.\n",
    "token_start_index = 0\n",
    "while sequence_ids[token_start_index] != 1:\n",
    "    token_start_index += 1\n",
    "\n",
    "# End token index of the current span in the text.\n",
    "token_end_index = len(tokenized_example[\"input_ids\"][0]) - 1\n",
    "while sequence_ids[token_end_index] != 1:\n",
    "    token_end_index -= 1\n",
    "\n",
    "# Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
    "offsets = tokenized_example[\"offset_mapping\"][0]\n",
    "if (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
    "    # Move the token_start_index and token_end_index to the two ends of the answer.\n",
    "    # Note: we could go after the last offset if the answer is the last word (edge case).\n",
    "    while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
    "        token_start_index += 1\n",
    "    start_position = token_start_index - 1\n",
    "    while offsets[token_end_index][1] >= end_char:\n",
    "        token_end_index -= 1\n",
    "    end_position = token_end_index + 1\n",
    "    print(start_position, end_position)\n",
    "else:\n",
    "    print(\"The answer is not in this feature.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YJ2BW8NUJzFX"
   },
   "source": [
    "And we can double check that it is indeed the theoretical answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mmJlRJuCJzFX",
    "outputId": "ac3363f9-c5e5-4da3-b63d-c64d92cbceff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Junho a agosto\n",
      "Junho a agosto\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.decode(tokenized_example[\"input_ids\"][0][start_position: end_position+1]))\n",
    "print(answers[\"text\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xfpRw4hoJzFY"
   },
   "source": [
    "For this notebook to work with any kind of models, we need to account for the special case where the model expects padding on the left (in which case we switch the order of the question and the context):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cU-OwGlPJzFY"
   },
   "outputs": [],
   "source": [
    "pad_on_right = tokenizer.padding_side == \"right\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YWZueJf6JzFY"
   },
   "source": [
    "Now let's put everything together in one function we will apply to our training set. In the case of impossible answers (the answer is in another feature given by an example with a long context), we set the cls index for both the start and end position. We could also simply discard those examples from the training set if the flag `allow_impossible_answers` is `False`. Since the preprocessing is already complex enough as it is, we've kept is simple for this part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "R50AU7q2JzFY"
   },
   "outputs": [],
   "source": [
    "def prepare_train_features(examples):\n",
    "    # Tokenize our examples with truncation and padding, but keep the overflows using a stride. This results\n",
    "    # in one example possible giving several features when a context is long, each of those features having a\n",
    "    # context that overlaps a bit the context of the previous feature.\n",
    "    tokenized_examples = tokenizer(\n",
    "        examples[\"question\" if pad_on_right else \"context\"],\n",
    "        examples[\"context\" if pad_on_right else \"question\"],\n",
    "        truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
    "        max_length=max_length,\n",
    "        stride=doc_stride,\n",
    "        return_overflowing_tokens=True,\n",
    "        return_offsets_mapping=True,\n",
    "        padding=\"max_length\",\n",
    "    )\n",
    "\n",
    "    # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
    "    # its corresponding example. This key gives us just that.\n",
    "    sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
    "    # The offset mappings will give us a map from token to character position in the original context. This will\n",
    "    # help us compute the start_positions and end_positions.\n",
    "    offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
    "\n",
    "    # Let's label those examples!\n",
    "    tokenized_examples[\"start_positions\"] = []\n",
    "    tokenized_examples[\"end_positions\"] = []\n",
    "\n",
    "    for i, offsets in enumerate(offset_mapping):\n",
    "        # We will label impossible answers with the index of the CLS token.\n",
    "        input_ids = tokenized_examples[\"input_ids\"][i]\n",
    "        cls_index = input_ids.index(tokenizer.cls_token_id)\n",
    "\n",
    "        # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
    "        sequence_ids = tokenized_examples.sequence_ids(i)\n",
    "\n",
    "        # One example can give several spans, this is the index of the example containing this span of text.\n",
    "        sample_index = sample_mapping[i]\n",
    "        answers = examples[\"answers\"][sample_index]\n",
    "        # If no answers are given, set the cls_index as answer.\n",
    "        if len(answers[\"answer_start\"]) == 0:\n",
    "            tokenized_examples[\"start_positions\"].append(cls_index)\n",
    "            tokenized_examples[\"end_positions\"].append(cls_index)\n",
    "        else:\n",
    "            # Start/end character index of the answer in the text.\n",
    "            start_char = answers[\"answer_start\"][0]\n",
    "            end_char = start_char + len(answers[\"text\"][0])\n",
    "\n",
    "            # Start token index of the current span in the text.\n",
    "            token_start_index = 0\n",
    "            while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n",
    "                token_start_index += 1\n",
    "\n",
    "            # End token index of the current span in the text.\n",
    "            token_end_index = len(input_ids) - 1\n",
    "            while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n",
    "                token_end_index -= 1\n",
    "\n",
    "            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
    "            if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
    "                tokenized_examples[\"start_positions\"].append(cls_index)\n",
    "                tokenized_examples[\"end_positions\"].append(cls_index)\n",
    "            else:\n",
    "                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n",
    "                # Note: we could go after the last offset if the answer is the last word (edge case).\n",
    "                while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
    "                    token_start_index += 1\n",
    "                tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
    "                while offsets[token_end_index][1] >= end_char:\n",
    "                    token_end_index -= 1\n",
    "                tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
    "\n",
    "    return tokenized_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0lm8ozrJIrJR"
   },
   "source": [
    "This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-b70jh26IrJS"
   },
   "outputs": [],
   "source": [
    "features = prepare_train_features(datasets['train'][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zS-6iXTkIrJT"
   },
   "source": [
    "To apply this function on all the sentences (or pairs of sentences) in our dataset, we just use the `map` method of our `dataset` object we created earlier. This will apply the function on all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed in one single command. Since our preprocessing changes the number of samples, we need to remove the old columns when applying it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 116,
     "referenced_widgets": [
      "144445eae3434ba097c8239f0a0344e6",
      "0bc1b64d71dd4089b9293e6ce4186759",
      "1b58b1af29de49108441b8bd777939b8",
      "5018820ecf1c4a5ea52fab036c8640ef",
      "4b0709971ff74ca89308105d23202e85",
      "97a48d4d9ad44a87991ab631fc1b69bd",
      "af6b275af17a4991a1ff3f946c0a2e61",
      "a7ea8914997e484f8a828aaf18eaa005",
      "fdc3e3f5e6284380a869f98491e1f7d9",
      "2f148877ca3d46e7a925633ed2467b69",
      "92a8a2d600bc455684ef015d9a0f13a7",
      "2bb323d86d914e03abe9d3677c3406a8",
      "6e1f01f6954844cf80f938d89984166f",
      "0759c72d6d814541981cb138ff68812e",
      "5eeb19fda3b54c4f9919194a1f47ee66",
      "3847cc0c9521407bac1f5f7fb55f2506"
     ]
    },
    "id": "DDtsaJeVIrJT",
    "outputId": "ee848864-40ca-43e3-8def-a54ca2188830"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "144445eae3434ba097c8239f0a0344e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=88.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdc3e3f5e6284380a869f98491e1f7d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=11.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenized_datasets = datasets.map(prepare_train_features, batched=True, remove_columns=datasets[\"train\"].column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "voWiw8C7IrJV"
   },
   "source": [
    "Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and force the preprocessing to be applied again.\n",
    "\n",
    "Note that we passed `batched=True` to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "545PP3o8IrJV"
   },
   "source": [
    "## Fine-tuning the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FBiW8UpKIrJW"
   },
   "source": [
    "Now that our data is ready for training, we can download the pretrained model and fine-tune it. Since our task is question answering, we use the `AutoModelForQuestionAnswering` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 174,
     "referenced_widgets": [
      "fb873d9a42cc41ea9b933278bafecbda",
      "6be924a67eb2410ea17092a2f412653b",
      "4ca8b82e45ad43d2ae71989bf50d35ef",
      "5d36626d92464b11a5e02469b7523656",
      "73b0887e0fda440ea974626fa0c7eb29",
      "647eac33b4904425b14bfdabaec816c3",
      "10bd817dd0c44d6095681f2ac9c7c35c",
      "40c62e2b380941719c75f3cec63bb497"
     ]
    },
    "id": "TlqNaB8jIrJW",
    "outputId": "44a88f89-83cb-445e-cc6e-0bd1e5d4b757"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb873d9a42cc41ea9b933278bafecbda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=438235074.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at neuralmind/bert-base-portuguese-cased were not used when initializing BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at neuralmind/bert-base-portuguese-cased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForQuestionAnswering, TrainingArguments, Trainer\n",
    "\n",
    "model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CczA5lJlIrJX"
   },
   "source": [
    "The warning is telling us we are throwing away some weights (the `vocab_transform` and `vocab_layer_norm` layers) and randomly initializing some other (the `pre_classifier` and `classifier` layers). This is absolutely normal in this case, because we are removing the head used to pretrain the model on a masked language modeling objective and replacing it with a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_N8urzhyIrJY"
   },
   "source": [
    "To instantiate a `Trainer`, we will need to define three more things. The most important is the [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments), which is a class that contains all the attributes to customize the training. It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PECDE4UqzRrX"
   },
   "outputs": [],
   "source": [
    "# new \n",
    "\n",
    "file_name = str(model_checkpoint).replace('/','-') + '-finetuned-squad-v1.1-pt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BjF1xBekzRrX"
   },
   "outputs": [],
   "source": [
    "# modified \n",
    "\n",
    "# args = TrainingArguments(\n",
    "#     f\"test-squad\",\n",
    "#     evaluation_strategy = \"epoch\",\n",
    "#     learning_rate=2e-5,\n",
    "#     per_device_train_batch_size=batch_size,\n",
    "#     per_device_eval_batch_size=batch_size,\n",
    "#     num_train_epochs=3,\n",
    "#     weight_decay=0.01,\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Bliy8zgjIrJY"
   },
   "outputs": [],
   "source": [
    "# new \n",
    "\n",
    "args = TrainingArguments(\n",
    "    f\"{file_name}\",\n",
    "    evaluation_strategy = \"epoch\",\n",
    "    learning_rate=3e-5,\n",
    "    per_device_train_batch_size=batch_size,\n",
    "    per_device_eval_batch_size=batch_size,\n",
    "    num_train_epochs=3.0,\n",
    "    weight_decay=0.01,\n",
    "    save_total_limit=1,\n",
    "    fp16 = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "km3pGVdTIrJc"
   },
   "source": [
    "Here we set the evaluation to be done at the end of each epoch, tweak the learning rate, use the `batch_size` defined at the top of the notebook and customize the number of epochs for training, as well as the weight decay."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KWATSilVJzFb"
   },
   "source": [
    "Then we will need a data collator that will batch our processed examples together, here the default one will work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3PyXlPhUJzFc"
   },
   "outputs": [],
   "source": [
    "from transformers import default_data_collator\n",
    "\n",
    "data_collator = default_data_collator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rXuFTAzDIrJe"
   },
   "source": [
    "We will evaluate our model and compute metrics in the next section (this is a very long operation, so we will only compute the evaluation loss during training).\n",
    "\n",
    "Then we just need to pass all of this along with our datasets to the `Trainer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "imY1oC3SIrJf"
   },
   "outputs": [],
   "source": [
    "trainer = Trainer(\n",
    "    model,\n",
    "    args,\n",
    "    train_dataset=tokenized_datasets[\"train\"],\n",
    "    eval_dataset=tokenized_datasets[\"validation\"],\n",
    "    data_collator=data_collator,\n",
    "    tokenizer=tokenizer,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CdzABDVcIrJg"
   },
   "source": [
    "We can now finetune our model by just calling the `train` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 185
    },
    "id": "VRwhM9gnzRrZ",
    "outputId": "d32b8a1f-37d0-4922-dbea-83a1c5539002",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='16734' max='16734' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [16734/16734 2:38:52, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.024800</td>\n",
       "      <td>1.075474</td>\n",
       "      <td>291.106300</td>\n",
       "      <td>37.502000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.756400</td>\n",
       "      <td>1.104146</td>\n",
       "      <td>291.190400</td>\n",
       "      <td>37.491000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.524000</td>\n",
       "      <td>1.225836</td>\n",
       "      <td>291.215800</td>\n",
       "      <td>37.488000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=16734, training_loss=0.8457311714822507, metrics={'train_runtime': 9533.1723, 'train_samples_per_second': 1.755, 'total_flos': 66822624992549376, 'epoch': 3.0})"
      ]
     },
     "execution_count": 39,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sxCpm-uAJzFc"
   },
   "source": [
    "Since this training is particularly long, let's save the model just in case we need to restart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "P3CXKzvXJzFc"
   },
   "outputs": [],
   "source": [
    "trainer.save_model(file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TBh7XxXNJzFd"
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PNP1Z5XUJzFd"
   },
   "source": [
    "Evaluating our model will require a bit more work, as we will need to map the predictions of our model back to parts of the context. The model itself predicts logits for the start and en position of our answers: if we take a batch from our validation datalaoder, here is the output our model gives us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pEd46ewTJzFd",
    "outputId": "513cddd8-a938-400f-f9b7-7dd9ef59b03b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "odict_keys(['loss', 'start_logits', 'end_logits'])"
      ]
     },
     "execution_count": 41,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "for batch in trainer.get_eval_dataloader():\n",
    "    break\n",
    "batch = {k: v.to(trainer.args.device) for k, v in batch.items()}\n",
    "with torch.no_grad():\n",
    "    output = trainer.model(**batch)\n",
    "output.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5YOqrA-qJzFd"
   },
   "source": [
    "The output of the model is a dict-like object that contains the loss (since we provided labels), the start and end logits. We won't need the loss for our predictions, let's have a look a the logits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZERm67vfJzFd",
    "outputId": "f1f20faa-9988-4a01-a3f0-4daac1d8dec9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 384]), torch.Size([16, 384]))"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.start_logits.shape, output.end_logits.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S6B7BmLhJzFe"
   },
   "source": [
    "We have one logit for each feature and each token. The most obvious thing to predict an answer for each featyre is to take the index for the maximum of the start logits as a start position and the index of the maximum of the end logits as an end position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zHOMtUrYJzFe",
    "outputId": "3217429d-8a06-4ca1-93d3-c56348ed5b14"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 15,  80, 114,  76, 183,  73,  22,  40,  40, 128,  47,  53,  53,  39,\n",
       "          39,  97], device='cuda:0'),\n",
       " tensor([ 18,  84, 116,  80, 190,  80,  22,  41,  42, 149,  52,  42,  42,  51,\n",
       "          51, 115], device='cuda:0'))"
      ]
     },
     "execution_count": 43,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output.start_logits.argmax(dim=-1), output.end_logits.argmax(dim=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d1Qnue4MJzFe"
   },
   "source": [
    "This will work great in a lot of cases, but what if this prediction gives us something impossible: the start position could be greater than the end position, or point to a span of text in the question instead of the answer. In that case, we might want to look at the second best prediction to see if it gives a possible answer and select that instead.\n",
    "\n",
    "However, picking the second best answer is not as easy as picking the best one: is it the second best index in the start logits with the best index in the end logits? Or the best index in the start logits with the second best index in the end logits? And if that second best answer is not possible either, it gets even trickier for the third best answer.\n",
    "\n",
    "\n",
    "To classify our answers, we will use the score obtained by adding the start and end logits. We won't try to order all the possible answers and limit ourselves to with a hyper-parameter we call `n_best_size`. We'll pick the best indices in the start and end logits and gather all the answers this predicts. After checking if each one is valid, we will sort them by their score and keep the best one. Here is how we would do this on the first feature in the batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8XW-qLvjJzFe"
   },
   "outputs": [],
   "source": [
    "n_best_size = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2SNcXwx_JzFf"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "start_logits = output.start_logits[0].cpu().numpy()\n",
    "end_logits = output.end_logits[0].cpu().numpy()\n",
    "# Gather the indices the best start/end logits:\n",
    "start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "valid_answers = []\n",
    "for start_index in start_indexes:\n",
    "    for end_index in end_indexes:\n",
    "        if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
    "            valid_answers.append(\n",
    "                {\n",
    "                    \"score\": start_logits[start_index] + end_logits[end_index],\n",
    "                    \"text\": \"\" # We need to find a way to get back the original substring corresponding to the answer in the context\n",
    "                }\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S62bJf2EJzFf"
   },
   "source": [
    "And then we can sort the `valid_answers` according to their `score` and only keep the best one. The only point left is how to check a given span is inside the context (and not the question) and how to get back the text inside. To do this, we need to add two things to our validation features:\n",
    "- the ID of the example that generated the feature (since each example can generate several features, as seen before);\n",
    "- the offset mapping that will give us a map from token indices to character positions in the context.\n",
    "\n",
    "That's why we will re-process the validation set with the following function, slightly different from `prepare_train_features`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9bp22w9BJzFf"
   },
   "outputs": [],
   "source": [
    "def prepare_validation_features(examples):\n",
    "    # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n",
    "    # in one example possible giving several features when a context is long, each of those features having a\n",
    "    # context that overlaps a bit the context of the previous feature.\n",
    "    tokenized_examples = tokenizer(\n",
    "        examples[\"question\" if pad_on_right else \"context\"],\n",
    "        examples[\"context\" if pad_on_right else \"question\"],\n",
    "        truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
    "        max_length=max_length,\n",
    "        stride=doc_stride,\n",
    "        return_overflowing_tokens=True,\n",
    "        return_offsets_mapping=True,\n",
    "        padding=\"max_length\",\n",
    "    )\n",
    "\n",
    "    # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
    "    # its corresponding example. This key gives us just that.\n",
    "    sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
    "\n",
    "    # We keep the example_id that gave us this feature and we will store the offset mappings.\n",
    "    tokenized_examples[\"example_id\"] = []\n",
    "\n",
    "    for i in range(len(tokenized_examples[\"input_ids\"])):\n",
    "        # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
    "        sequence_ids = tokenized_examples.sequence_ids(i)\n",
    "        context_index = 1 if pad_on_right else 0\n",
    "\n",
    "        # One example can give several spans, this is the index of the example containing this span of text.\n",
    "        sample_index = sample_mapping[i]\n",
    "        tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n",
    "\n",
    "        # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token\n",
    "        # position is part of the context or not.\n",
    "        tokenized_examples[\"offset_mapping\"][i] = [\n",
    "            (o if sequence_ids[k] == context_index else None)\n",
    "            for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n",
    "        ]\n",
    "\n",
    "    return tokenized_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BMn4uyp-JzFg"
   },
   "source": [
    "And like before, we can apply that function to our validation set easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 66,
     "referenced_widgets": [
      "3721168b79c74e78be32f631ce4aa817",
      "80d2dbb1b3864d00a503eac66ba2f744",
      "1f2c79f67ebc48b9bddb3a9f9a0ad518",
      "248d0d39fbb840ab816d1138de6b3100",
      "e4ade6af00934247bb5f8c149bfcdca1",
      "7343e8a262174994994911aeffd87a90",
      "9b1f5b7ff0e843d6a4d68cd1812dd354",
      "0d36d17c2adc41f68cd563cf2340402b"
     ]
    },
    "id": "OdU72AnTJzFg",
    "outputId": "9c20313a-6167-4ba6-abb2-cbe33bf02fc6"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3721168b79c74e78be32f631ce4aa817",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=11.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "validation_features = datasets[\"validation\"].map(\n",
    "    prepare_validation_features,\n",
    "    batched=True,\n",
    "    remove_columns=datasets[\"validation\"].column_names\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qBkuz2NyJzFg"
   },
   "source": [
    "Now we can grab the predictions for all features by using the `Trainer.predict` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 37
    },
    "id": "YHSoboEqJzFg",
    "outputId": "4e9895db-887e-4da9-b795-26353a10fb5d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='683' max='683' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [683/683 02:07]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "raw_predictions = trainer.predict(validation_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oQIm5apaJzFh"
   },
   "source": [
    "The `Trainer` *hides* the columns that are not used by the model (here `example_id` and `offset_mapping` which we will need for our post-processing), so we set them back:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OMKvhnEUJzFh"
   },
   "outputs": [],
   "source": [
    "validation_features.set_format(type=validation_features.format[\"type\"], columns=list(validation_features.features.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hAobXlekJzFh"
   },
   "source": [
    "We can now refine the test we had before: since we set `None` in the offset mappings when it corresponds to a part of the question, it's easy to check if an answer is fully inside the context. We also eliminate very long answers from our considerations (with an hyper-parameter we can tune)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BqV4JmFSJzFh"
   },
   "outputs": [],
   "source": [
    "max_answer_length = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# new\n",
    "num_qa = 0\n",
    "datasets[\"validation\"][num_qa]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_logits = output.start_logits[0].cpu().numpy()\n",
    "end_logits = output.end_logits[0].cpu().numpy()\n",
    "offset_mapping = validation_features[0][\"offset_mapping\"]\n",
    "# The first feature comes from the first example. For the more general case, we will need to be match the example_id to\n",
    "# an example index\n",
    "context = datasets[\"validation\"][num_qa][\"context\"]\n",
    "\n",
    "# Gather the indices the best start/end logits:\n",
    "start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "valid_answers = []\n",
    "for start_index in start_indexes:\n",
    "    for end_index in end_indexes:\n",
    "        # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n",
    "        # to part of the input_ids that are not in the context.\n",
    "        if (\n",
    "            start_index >= len(offset_mapping)\n",
    "            or end_index >= len(offset_mapping)\n",
    "            or offset_mapping[start_index] is None\n",
    "            or offset_mapping[end_index] is None\n",
    "        ):\n",
    "            continue\n",
    "        # Don't consider answers with a length that is either < 0 or > max_answer_length.\n",
    "        if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
    "            continue\n",
    "        if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
    "            start_char = offset_mapping[start_index][0]\n",
    "            end_char = offset_mapping[end_index][1]\n",
    "            valid_answers.append(\n",
    "                {\n",
    "                    \"score\": start_logits[start_index] + end_logits[end_index],\n",
    "                    \"text\": context[start_char: end_char]\n",
    "                }\n",
    "            )\n",
    "\n",
    "valid_answers = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[:n_best_size]\n",
    "valid_answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Og-r3j9_JzFh"
   },
   "source": [
    "We can compare to the actual ground-truth answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# modified\n",
    "datasets[\"validation\"][num_qa][\"answers\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pb0kTNF1JzFi"
   },
   "source": [
    "Our model picked the right as the most likely answer!\n",
    "\n",
    "As we mentioned in the code above, this was easy on the first feature because we knew it comes from the first example. For the other features, we will need a map between examples and their corresponding features. Also, since one example can give several features, we will need to gather together all the answers in all the features generated by a given example, then pick the best one. The following code builds a map from example index to its corresponding features indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gBnSB_7uJzFi"
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "\n",
    "examples = datasets[\"validation\"]\n",
    "features = validation_features\n",
    "\n",
    "example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
    "features_per_example = collections.defaultdict(list)\n",
    "for i, feature in enumerate(features):\n",
    "    features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2OBAVTbcJzFi"
   },
   "source": [
    "We're almost ready for our post-processing function. The last bit to deal with is the impossible answer (when `squad_v2 = True`). The code above only keeps answers that are inside the context, we need to also grab the score for the impossible answer (which has start and end indices corresponding to the index of the CLS token). When one example gives several features, we have to predict the impossible answer when all the features give a high score to the impossible answer (since one feature could predict the impossible answer just because the answer isn't in the part of the context it has access too), which is why the score of the impossible answer for one example is the *minimum* of the scores for the impossible answer in each feature generated by the example.\n",
    "\n",
    "We then predict the impossible answer when that score is greater than the score of the best non-impossible answer. All combined together, this gives us this post-processing function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NXYKPkgOJzFi"
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "\n",
    "def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n",
    "    all_start_logits, all_end_logits = raw_predictions\n",
    "    # Build a map example to its corresponding features.\n",
    "    example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
    "    features_per_example = collections.defaultdict(list)\n",
    "    for i, feature in enumerate(features):\n",
    "        features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n",
    "\n",
    "    # The dictionaries we have to fill.\n",
    "    predictions = collections.OrderedDict()\n",
    "\n",
    "    # Logging.\n",
    "    print(f\"Post-processing {len(examples)} example predictions split into {len(features)} features.\")\n",
    "\n",
    "    # Let's loop over all the examples!\n",
    "    for example_index, example in enumerate(tqdm(examples)):\n",
    "        # Those are the indices of the features associated to the current example.\n",
    "        feature_indices = features_per_example[example_index]\n",
    "\n",
    "        min_null_score = None # Only used if squad_v2 is True.\n",
    "        valid_answers = []\n",
    "        \n",
    "        context = example[\"context\"]\n",
    "        # Looping through all the features associated to the current example.\n",
    "        for feature_index in feature_indices:\n",
    "            # We grab the predictions of the model for this feature.\n",
    "            start_logits = all_start_logits[feature_index]\n",
    "            end_logits = all_end_logits[feature_index]\n",
    "            # This is what will allow us to map some the positions in our logits to span of texts in the original\n",
    "            # context.\n",
    "            offset_mapping = features[feature_index][\"offset_mapping\"]\n",
    "\n",
    "            # Update minimum null prediction.\n",
    "            cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n",
    "            feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n",
    "            if min_null_score is None or min_null_score < feature_null_score:\n",
    "                min_null_score = feature_null_score\n",
    "\n",
    "            # Go through all possibilities for the `n_best_size` greater start and end logits.\n",
    "            start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "            end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
    "            for start_index in start_indexes:\n",
    "                for end_index in end_indexes:\n",
    "                    # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n",
    "                    # to part of the input_ids that are not in the context.\n",
    "                    if (\n",
    "                        start_index >= len(offset_mapping)\n",
    "                        or end_index >= len(offset_mapping)\n",
    "                        or offset_mapping[start_index] is None\n",
    "                        or offset_mapping[end_index] is None\n",
    "                    ):\n",
    "                        continue\n",
    "                    # Don't consider answers with a length that is either < 0 or > max_answer_length.\n",
    "                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
    "                        continue\n",
    "\n",
    "                    start_char = offset_mapping[start_index][0]\n",
    "                    end_char = offset_mapping[end_index][1]\n",
    "                    valid_answers.append(\n",
    "                        {\n",
    "                            \"score\": start_logits[start_index] + end_logits[end_index],\n",
    "                            \"text\": context[start_char: end_char]\n",
    "                        }\n",
    "                    )\n",
    "        \n",
    "        if len(valid_answers) > 0:\n",
    "            best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n",
    "        else:\n",
    "            # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n",
    "            # failure.\n",
    "            best_answer = {\"text\": \"\", \"score\": 0.0}\n",
    "        \n",
    "        # Let's pick our final answer: the best one or the null answer (only for squad_v2)\n",
    "        if not squad_v2:\n",
    "            predictions[example[\"id\"]] = best_answer[\"text\"]\n",
    "        else:\n",
    "            answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n",
    "            predictions[example[\"id\"]] = answer\n",
    "\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ALYCEestJzFj"
   },
   "source": [
    "And we can apply our post-processing function to our raw predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 84,
     "referenced_widgets": [
      "d5e7b71782524247912d54b7619b068b",
      "152b10e1df7b4633b408f41b194bb6b5",
      "68c13311ca82469d89c7014efe6d78d8",
      "271edb820d30444da7cebf1459835127",
      "c8bd76dd65c5405b83dca2d009d7534a",
      "33dbb879b59b41bba07e948d88332de2",
      "b335f773b4d2401c89670034e5ccb805",
      "1dab2b3aae54428e952695f3daeb8acb"
     ]
    },
    "id": "uF4tgDAnJzFj",
    "outputId": "aa1a3d04-0b6b-45cb-b8c3-a1350f374b60"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Post-processing 10570 example predictions split into 10917 features.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d5e7b71782524247912d54b7619b068b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=10570.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "final_predictions = postprocess_qa_predictions(datasets[\"validation\"], validation_features, raw_predictions.predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uuBVINfNJzFj"
   },
   "source": [
    "Then we can load the metric from the datasets library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 116,
     "referenced_widgets": [
      "aeec65122d2b4869993cb14bdfc3155c",
      "72817fee75b84eac8273dd51d92bb5ac",
      "c799404686804653bde731972befd439",
      "47ca5cb5929a4f7eac1e4f9609eb3cde",
      "30edc0d0984a497c88e291ce1a73d8e7",
      "a3db05eee0e64d4c93481ba0c41f3868",
      "4ae29b19ed414fa29b16b1d88c9895a1",
      "5cca3b2b9e874b02a0d6435916e51bc5",
      "20a6004dbe504c999a682ceb8b796103",
      "6cf82e60621c451bb8935300228095b4",
      "0ee2aeb9d1224776a12d49ce0605eeef",
      "6d55d907528640a4bb7f1b0930318e20",
      "1d9319a9d2214466b44fef6fb2a74dd1",
      "28401da0477b46e18febfb1c8bb29e5b",
      "b7968d32762543c3b3833c2415f7000f",
      "b81edb02885b48d0a5e0cbcf6d3de685"
     ]
    },
    "id": "YkezY7VtJzFj",
    "outputId": "ccecc340-f277-4a90-e2d0-d1e536e767bc"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aeec65122d2b4869993cb14bdfc3155c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1553.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20a6004dbe504c999a682ceb8b796103",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1132.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dz2TN415JzFj"
   },
   "source": [
    "Then we can call compute on it. We just need to format predictions and labels a bit as it expects a list of dictionaries and not one big dictionary. In the case of squad_v2, we also have to set a `no_answer_probability` argument (which we set to 0.0 here as we have already set the answer to empty if we picked it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AE_nCmKHzRri",
    "outputId": "4232ac03-315e-4228-915c-942d16c5226e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'exact_match': 70.49195837275307, 'f1': 82.50069751388098}"
      ]
     },
     "execution_count": 57,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if squad_v2:\n",
    "    formatted_predictions = [{\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} for k, v in predictions.items()]\n",
    "else:\n",
    "    formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()]\n",
    "references = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]]\n",
    "metric.compute(predictions=formatted_predictions, references=references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TF4B13hhJzFk"
   },
   "source": [
    "Don't forget to [update your model](https://huggingface.co/transformers/model_sharing.html) on the [🤗 Model Hub](https://huggingface.co/models). You can then use it only to generate results like the one shown in the first picture of this notebook!|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xdRgcXa3Mp7t"
   },
   "source": [
    "# Save our QA model to HF format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "kGw0HmeqzRrk"
   },
   "outputs": [],
   "source": [
    "import pathlib\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "qW4NTt4KzRrk"
   },
   "outputs": [],
   "source": [
    "dest = pathlib.Path.cwd()/'HFmodels'\n",
    "fname_HF = file_name\n",
    "\n",
    "path_to_awesome_name_you_picked = dest/fname_HF\n",
    "path_to_awesome_name_you_picked.mkdir(exist_ok=True, parents=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-3HZKe12zRrk",
    "outputId": "d6f1fdc2-cb52-4c3a-bce3-f89afc72d6e0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('/content/HFmodels/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/tokenizer_config.json',\n",
       " '/content/HFmodels/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/special_tokens_map.json',\n",
       " '/content/HFmodels/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/vocab.txt',\n",
       " '/content/HFmodels/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/added_tokens.json')"
      ]
     },
     "execution_count": 60,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.save_model(f\"{str(path_to_awesome_name_you_picked)}\")\n",
    "tokenizer.save_pretrained(f\"{str(path_to_awesome_name_you_picked)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W8gG8eQczRrl",
    "outputId": "2324e1f5-1179-44f3-92bc-9f9e6338163d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "( 413  M) neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt\n",
      "\n",
      "model saved!\n"
     ]
    }
   ],
   "source": [
    "# model size\n",
    "path_file = path_to_awesome_name_you_picked/'pytorch_model.bin'\n",
    "print('(',int(round(path_file.stat().st_size/(1024*1024),0)),' M)',path_file.parent.name)\n",
    "        \n",
    "print('\\nmodel saved!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PcSWu1ROfvoO"
   },
   "source": [
    "# Download the model files to your local folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MkmTHEI-fsNg"
   },
   "outputs": [],
   "source": [
    "# file_to_download = '/content/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/vocab.txt'\n",
    "# file_to_download = '/content/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/training_args.bin'\n",
    "# file_to_download = '/content/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/tokenizer_config.json'\n",
    "# file_to_download = '/content/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/special_tokens_map.json'\n",
    "# file_to_download = '/content/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/pytorch_model.bin'\n",
    "file_to_download = '/content/neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt/config.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "J5mmToRiiXaD"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download(file_to_download)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DcmsZKPizRrl"
   },
   "source": [
    "# Use our QA model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "kGw0HmeqzRrk"
   },
   "outputs": [],
   "source": [
    "import pathlib\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "qW4NTt4KzRrk"
   },
   "outputs": [],
   "source": [
    "dest = pathlib.Path.cwd()/'HFmodels'\n",
    "fname_HF = 'neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt'\n",
    "\n",
    "path_to_awesome_name_you_picked = dest/fname_HF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "C0QQt5eVzRrl"
   },
   "outputs": [],
   "source": [
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "RfcFr-58zRrl"
   },
   "outputs": [],
   "source": [
    "# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19\n",
    "context = r\"\"\"\n",
    "A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, \n",
    "uma doença respiratória aguda causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). \n",
    "A doença foi identificada pela primeira vez em Wuhan, na província de Hubei, República Popular da China, \n",
    "em 1 de dezembro de 2019, mas o primeiro caso foi reportado em 31 de dezembro do mesmo ano. \n",
    "Acredita-se que o vírus tenha uma origem zoonótica, porque os primeiros casos confirmados \n",
    "tinham principalmente ligações ao Mercado Atacadista de Frutos do Mar de Huanan, que também vendia animais vivos. \n",
    "Em 11 de março de 2020, a Organização Mundial da Saúde declarou o surto uma pandemia. Até 8 de fevereiro de 2021, \n",
    "pelo menos 105 743 102 casos da doença foram confirmados em pelo menos 191 países e territórios, \n",
    "com cerca de 2 308 943 mortes e 58 851 440 pessoas curadas.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_qa = str(path_to_awesome_name_you_picked)\n",
    "nlp = pipeline(\"question-answering\", model=model_qa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nVKERZzRzRrm",
    "outputId": "935ff7f2-0881-4027-b870-0ee1a1392d23"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: '1 de dezembro de 2019', score: 0.713, start: 328, end: 349\n"
     ]
    }
   ],
   "source": [
    "question = \"Quando começou a pandemia de Covid-19 no mundo?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PcD1mpfOYWd7",
    "outputId": "ac5369e4-2560-4176-a228-4b91864a1880"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: '1 de dezembro de 2019', score: 0.4922, start: 328, end: 349\n"
     ]
    }
   ],
   "source": [
    "question = \"Qual é a data de início da pandemia Covid-19 em todo o mundo?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cI6LqkvbzRrn",
    "outputId": "f2a8f0b9-5004-4682-909a-59b03e5f9332"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: 'uma doença respiratória aguda', score: 0.0429, start: 110, end: 139\n"
     ]
    }
   ],
   "source": [
    "question = \"O qué a pandemia de Covid-19?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NecR00-wzRrn",
    "outputId": "21dc3c8d-1ade-4486-eb04-e82d535e08b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: 'Acredita-se que o vírus tenha uma origem zoonótica', score: 0.3732, start: 418, end: 468\n"
     ]
    }
   ],
   "source": [
    "question = \"A Covid-19 tem algo a ver com animais?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RcK4pn1hbLhL",
    "outputId": "483fe01f-3495-4865-9412-332ddde4a9fc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: 'Wuhan, na província de Hubei, República Popular da China', score: 0.4265, start: 266, end: 322\n"
     ]
    }
   ],
   "source": [
    "question = \"Onde foi descoberta a Covid-19?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7rFEmmsjzRrn",
    "outputId": "732e3a61-910f-48a6-a353-469ab5da2f23"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: '105 743 102', score: 0.4607, start: 750, end: 761\n"
     ]
    }
   ],
   "source": [
    "question = \"Quantos casos houve?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0v1TTQXDzRrn",
    "outputId": "5b621973-91ff-47c8-b253-3f7dfb78f7d2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: '2 308 943', score: 0.7313, start: 850, end: 859\n"
     ]
    }
   ],
   "source": [
    "question = \"Quantos mortes?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f78AggHLzRro",
    "outputId": "dffdd628-fa54-4c80-f941-329c9eb2b0b2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: '191', score: 0.4053, start: 810, end: 813\n"
     ]
    }
   ],
   "source": [
    "question = \"Quantos paises tiveram casos?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YSQnntVgcHHq",
    "outputId": "c0b8748c-7fcb-454c-c600-54ae968b20b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: '58 851 440', score: 0.9288, start: 869, end: 879\n"
     ]
    }
   ],
   "source": [
    "question = \"Quantas pessoas foram curadas?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Iv42ZZ5zRro"
   },
   "source": [
    "# Upload our QA model to the Model Hub of HF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NcqgkPfAzRrp"
   },
   "source": [
    "Source: [update your model](https://huggingface.co/transformers/model_sharing.html) on the [🤗 Model Hub](https://huggingface.co/models). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "kGw0HmeqzRrk"
   },
   "outputs": [],
   "source": [
    "import pathlib\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dest = pathlib.Path.cwd()/'HFmodels'\n",
    "\n",
    "# new name of 'neuralmind-bert-base-portuguese-cased-finetuned-squad-v1.1-pt'\n",
    "fname_HF = 'bert-base-cased-squad-v1.1-portuguese'\n",
    "\n",
    "path_to_awesome_name_you_picked = dest/fname_HF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "de4-VUjkzRrp"
   },
   "source": [
    "## Save locally "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4Rhdih1IzRrp"
   },
   "source": [
    "Done in the paragraph \"Save our QA model to HF format\" above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h40n9YTrzRrp"
   },
   "source": [
    "## Upload your model with the CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WmIhUBrbzRrq"
   },
   "source": [
    "Now go in a terminal and run the following command. It should be in the virtual enviromnent where you installed 🤗 Transformers, since that command transformers-cli comes from the library.\n",
    "\n",
    "`transformers-cli login`\n",
    "\n",
    "Then log in using the same credentials as on huggingface.co. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "96cXSMmLzRrq"
   },
   "source": [
    "Once you are logged in with your model hub credentials, you can start building your repositories. To create a repo:\n",
    "    \n",
    "`transformers-cli repo create your-model-name --organization your-org-name`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xwvu9p_RzRrq"
   },
   "source": [
    "This creates a repo on the model hub, which can be cloned.\n",
    "\n",
    "```\n",
    "# Make sure you have git-lfs installed\n",
    "# (https://git-lfs.github.com/)\n",
    "git lfs install\n",
    "\n",
    "git clone https://huggingface.co/username/your-model-name\n",
    "```\n",
    "\n",
    "This creates a repo on the model hub, which can be cloned.\n",
    "\n",
    "```\n",
    "\n",
    "# Make sure you have git-lfs installed\n",
    "# (https://git-lfs.github.com/)\n",
    "git lfs install\n",
    "\n",
    "git clone https://huggingface.co/username/your-model-name\n",
    "```\n",
    "\n",
    "When you have your local clone of your repo and lfs installed, you can then add/remove from that clone as you would with any other git repo.\n",
    "\n",
    "```\n",
    "\n",
    "# Commit as usual\n",
    "cd your-model-name\n",
    "echo \"hello\" >> README.md\n",
    "git add . && git commit -m \"Update from $USER\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-gflbn6ozRrq"
   },
   "source": [
    "## Make your model work on all frameworks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install --upgrade tensorflow\n",
    "# !pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "o5zNhpWrzRrq"
   },
   "outputs": [],
   "source": [
    "from transformers import TFBertForQuestionAnswering\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "nuYWUWjCzRrr"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the PyTorch model were not used when initializing the TF 2.0 model TFBertForQuestionAnswering: ['bert.embeddings.position_ids']\n",
      "- This IS expected if you are initializing TFBertForQuestionAnswering from a PyTorch model trained on another task or with another architecture (e.g. initializing a TFBertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFBertForQuestionAnswering from a PyTorch model that you expect to be exactly identical (e.g. initializing a TFBertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the weights of TFBertForQuestionAnswering were initialized from the PyTorch model.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForQuestionAnswering for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "tf_model = TFBertForQuestionAnswering.from_pretrained(str(path_to_awesome_name_you_picked), from_pt=True)\n",
    "tf_model.save_pretrained(str(path_to_awesome_name_you_picked))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-VXyD3PHzRrr"
   },
   "source": [
    "## Check the directory before pushing to the model hub."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rrkJukn8zRrr"
   },
   "source": [
    "Make sure there are no garbage files in the directory you’ll upload. It should only have:\n",
    "\n",
    "- a config.json file, which saves the configuration of your model ;\n",
    "- a pytorch_model.bin file, which is the PyTorch checkpoint (unless you can’t have it for some reason) ;\n",
    "- a tf_model.h5 file, which is the TensorFlow checkpoint (unless you can’t have it for some reason) ;\n",
    "- a special_tokens_map.json, which is part of your tokenizer save;\n",
    "- a tokenizer_config.json, which is part of your tokenizer save;\n",
    "- files named vocab.json, vocab.txt, merges.txt, or similar, which contain the vocabulary of your tokenizer, part of your tokenizer save;\n",
    "- maybe a added_tokens.json, which is part of your tokenizer save.\n",
    "\n",
    "Other files can safely be deleted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Uploading your files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the repo is cloned, you can add the model, configuration and tokenizer files. For instance, saving the model and tokenizer files:\n",
    "\n",
    "```\n",
    ">>> model.save_pretrained(\"path/to/repo/clone/your-model-name\")\n",
    ">>> tokenizer.save_pretrained(\"path/to/repo/clone/your-model-name\")\n",
    "```\n",
    "\n",
    "Or, if you’re using the Trainer API\n",
    "\n",
    "```\n",
    ">>> trainer.save_model(\"path/to/awesome-name-you-picked\")\n",
    ">>> tokenizer.save_pretrained(\"path/to/repo/clone/your-model-name\")\n",
    "```\n",
    "\n",
    "You can then add these files to the staging environment and verify that they have been correctly staged with the git status command:\n",
    "\n",
    "```\n",
    "git add --all\n",
    "git status\n",
    "```\n",
    "\n",
    "Finally, the files should be committed:\n",
    "\n",
    "```\n",
    "git commit -m \"First version of the your-model-name model and tokenizer.\"\n",
    "```\n",
    "\n",
    "And pushed to the remote:\n",
    "\n",
    "```\n",
    "git push\n",
    "```\n",
    "\n",
    "This will upload the folder containing the weights, tokenizer and configuration we have just prepared."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add a model card"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make sure everyone knows what your model can do, what its limitations, potential bias or ethical considerations are, please add a README.md model card to your model repo. You can just create it, or there’s also a convenient button titled “Add a README.md” on your model page. A model card template can be found [here](https://github.com/huggingface/model_card) (meta-suggestions are welcome). model card template (meta-suggestions are welcome)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Cópia de question_answering_BERT_base_cased_squad_v11_pt.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "hf",
   "language": "python",
   "name": "hf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "06d0cdf758404a99b3bfa0df48c9e348": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_204e637901534b95b6ec73b9d6519c26",
      "placeholder": "​",
      "style": "IPY_MODEL_56abcb61373e4d9e976a6dc6cd1560da",
      "value": " 112/112 [00:00&lt;00:00, 246B/s]"
     }
    },
    "06d7277c8e574430a065a9a85f88eea6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "06dae62919924bc0a560b0dc8185cafe": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0722ba5db9a940cba9fc3ec42b42efc8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_feaef1aea58e446e847a45d38d8366c9",
      "max": 2,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_684135d8e3a041559a98fac8af25d72d",
      "value": 2
     }
    },
    "0759c72d6d814541981cb138ff68812e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0a38ad48054b4793af78af528164f84d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "0bc1b64d71dd4089b9293e6ce4186759": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0d36d17c2adc41f68cd563cf2340402b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0d84ebc014b54a1b8fb1b35ea917e36c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0ee2aeb9d1224776a12d49ce0605eeef": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_28401da0477b46e18febfb1c8bb29e5b",
      "max": 1132,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1d9319a9d2214466b44fef6fb2a74dd1",
      "value": 1132
     }
    },
    "10bd817dd0c44d6095681f2ac9c7c35c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "11fffa69385e429f82a472a7dc2f8dba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "12742631aec9417881e3e7c471b1dd77": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_06dae62919924bc0a560b0dc8185cafe",
      "placeholder": "​",
      "style": "IPY_MODEL_f224c0c64f994806a13847d32c34f714",
      "value": " 647/647 [00:00&lt;00:00, 2.68kB/s]"
     }
    },
    "144445eae3434ba097c8239f0a0344e6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1b58b1af29de49108441b8bd777939b8",
       "IPY_MODEL_5018820ecf1c4a5ea52fab036c8640ef"
      ],
      "layout": "IPY_MODEL_0bc1b64d71dd4089b9293e6ce4186759"
     }
    },
    "152b10e1df7b4633b408f41b194bb6b5": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "173fe205f1a141af9466938cba0ec9b3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_cdc77948b42d40f698b1b150b2eca9f4",
       "IPY_MODEL_06d0cdf758404a99b3bfa0df48c9e348"
      ],
      "layout": "IPY_MODEL_abc26b3c8a044f18896650f1c4f36d8b"
     }
    },
    "1b58b1af29de49108441b8bd777939b8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_97a48d4d9ad44a87991ab631fc1b69bd",
      "max": 88,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4b0709971ff74ca89308105d23202e85",
      "value": 88
     }
    },
    "1baccf373ec24ac2ae16d761479e9944": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "1d9319a9d2214466b44fef6fb2a74dd1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "1dab2b3aae54428e952695f3daeb8acb": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1f2c79f67ebc48b9bddb3a9f9a0ad518": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7343e8a262174994994911aeffd87a90",
      "max": 11,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e4ade6af00934247bb5f8c149bfcdca1",
      "value": 11
     }
    },
    "204e637901534b95b6ec73b9d6519c26": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "208818f9241f4b298ebc1e756c8b6faf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "20a6004dbe504c999a682ceb8b796103": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0ee2aeb9d1224776a12d49ce0605eeef",
       "IPY_MODEL_6d55d907528640a4bb7f1b0930318e20"
      ],
      "layout": "IPY_MODEL_6cf82e60621c451bb8935300228095b4"
     }
    },
    "20aa40592cb7449d93cd22c37274e621": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "21e75a90d0cb4d778e7bee589c1e375b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "22091b5d454f44e693ee9f0a0396a447": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2246016d810d4f25b5cf9e696252752a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "248d0d39fbb840ab816d1138de6b3100": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0d36d17c2adc41f68cd563cf2340402b",
      "placeholder": "​",
      "style": "IPY_MODEL_9b1f5b7ff0e843d6a4d68cd1812dd354",
      "value": " 11/11 [03:07&lt;00:00, 17.03s/ba]"
     }
    },
    "271edb820d30444da7cebf1459835127": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1dab2b3aae54428e952695f3daeb8acb",
      "placeholder": "​",
      "style": "IPY_MODEL_b335f773b4d2401c89670034e5ccb805",
      "value": " 10570/10570 [00:24&lt;00:00, 436.46it/s]"
     }
    },
    "28401da0477b46e18febfb1c8bb29e5b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2bb323d86d914e03abe9d3677c3406a8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3847cc0c9521407bac1f5f7fb55f2506",
      "placeholder": "​",
      "style": "IPY_MODEL_5eeb19fda3b54c4f9919194a1f47ee66",
      "value": " 11/11 [00:06&lt;00:00,  1.77ba/s]"
     }
    },
    "2f148877ca3d46e7a925633ed2467b69": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "30edc0d0984a497c88e291ce1a73d8e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "33dbb879b59b41bba07e948d88332de2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "35c8fdad45674aec8fca9007ac63242f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "36b27f842246446d951e90d22ceece63": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6ded78516c3d4d5ba0aca74f89ac930e",
       "IPY_MODEL_ad3303422ce84d8597a9291dba86c0fc"
      ],
      "layout": "IPY_MODEL_5404a9f0aab54897b2c6e2eeaaa5c636"
     }
    },
    "3721168b79c74e78be32f631ce4aa817": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1f2c79f67ebc48b9bddb3a9f9a0ad518",
       "IPY_MODEL_248d0d39fbb840ab816d1138de6b3100"
      ],
      "layout": "IPY_MODEL_80d2dbb1b3864d00a503eac66ba2f744"
     }
    },
    "3847cc0c9521407bac1f5f7fb55f2506": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3f8faf6e856c460b8bcde6fe430d1c74": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "40c62e2b380941719c75f3cec63bb497": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "45cfc92ef71d4dcb8d562e6eede7e29d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_936b07ba5d2c4cf59d217cf99a3698b2",
       "IPY_MODEL_bdc3db4828ce497b9f9cff6254d74acd"
      ],
      "layout": "IPY_MODEL_d7fdbf13549240879be55b7f0b8ddbff"
     }
    },
    "47ca5cb5929a4f7eac1e4f9609eb3cde": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5cca3b2b9e874b02a0d6435916e51bc5",
      "placeholder": "​",
      "style": "IPY_MODEL_4ae29b19ed414fa29b16b1d88c9895a1",
      "value": " 4.02k/? [00:00&lt;00:00, 7.46kB/s]"
     }
    },
    "4ae29b19ed414fa29b16b1d88c9895a1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4b0709971ff74ca89308105d23202e85": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "4c48350b493f4edea02d8b2d3f2608e8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4ca8b82e45ad43d2ae71989bf50d35ef": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_647eac33b4904425b14bfdabaec816c3",
      "max": 438235074,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_73b0887e0fda440ea974626fa0c7eb29",
      "value": 438235074
     }
    },
    "4dc1e1d8ff9c434d81fc0cab52fefbcf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0d84ebc014b54a1b8fb1b35ea917e36c",
      "placeholder": "​",
      "style": "IPY_MODEL_7d2eceabf1784b219db412785212c4b5",
      "value": " 210k/210k [00:01&lt;00:00, 118kB/s]"
     }
    },
    "4fa075949df84e0390bfdfc4cb87fc19": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "5018820ecf1c4a5ea52fab036c8640ef": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a7ea8914997e484f8a828aaf18eaa005",
      "placeholder": "​",
      "style": "IPY_MODEL_af6b275af17a4991a1ff3f946c0a2e61",
      "value": " 88/88 [00:47&lt;00:00,  1.85ba/s]"
     }
    },
    "5404a9f0aab54897b2c6e2eeaaa5c636": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "551640d85a3b42f5b8b75c184cbfed45": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "56abcb61373e4d9e976a6dc6cd1560da": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5cca3b2b9e874b02a0d6435916e51bc5": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5d36626d92464b11a5e02469b7523656": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_40c62e2b380941719c75f3cec63bb497",
      "placeholder": "​",
      "style": "IPY_MODEL_10bd817dd0c44d6095681f2ac9c7c35c",
      "value": " 438M/438M [00:20&lt;00:00, 21.4MB/s]"
     }
    },
    "5eeb19fda3b54c4f9919194a1f47ee66": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "647eac33b4904425b14bfdabaec816c3": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "684135d8e3a041559a98fac8af25d72d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "68c13311ca82469d89c7014efe6d78d8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_33dbb879b59b41bba07e948d88332de2",
      "max": 10570,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c8bd76dd65c5405b83dca2d009d7534a",
      "value": 10570
     }
    },
    "6be924a67eb2410ea17092a2f412653b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6cf82e60621c451bb8935300228095b4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6d55d907528640a4bb7f1b0930318e20": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b81edb02885b48d0a5e0cbcf6d3de685",
      "placeholder": "​",
      "style": "IPY_MODEL_b7968d32762543c3b3833c2415f7000f",
      "value": " 3.35k/? [00:00&lt;00:00, 58.8kB/s]"
     }
    },
    "6ded78516c3d4d5ba0aca74f89ac930e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fcf5c3c66b9b49258d0f1769d3081dd1",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4fa075949df84e0390bfdfc4cb87fc19",
      "value": 1
     }
    },
    "6e1f01f6954844cf80f938d89984166f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "6fc6cf356dbb47e2b69e42cf205ab824": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_35c8fdad45674aec8fca9007ac63242f",
      "placeholder": "​",
      "style": "IPY_MODEL_b34d85775ba14711a038d153c97d4802",
      "value": " 2.00/2.00 [00:00&lt;00:00, 2.20B/s]"
     }
    },
    "72817fee75b84eac8273dd51d92bb5ac": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7343e8a262174994994911aeffd87a90": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "73b0887e0fda440ea974626fa0c7eb29": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "78cc7c33fc664fb8b45019a2d8d1a81c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7d2eceabf1784b219db412785212c4b5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8031f77075de48c6bdcf2a879f404fec": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c1e3222db9bb4aa0a7a4e895e1e69eac",
       "IPY_MODEL_bb2b7b66bbf64ee290774aa96c0f4c00"
      ],
      "layout": "IPY_MODEL_e3bc9a84092049009c39758400895ff8"
     }
    },
    "80d2dbb1b3864d00a503eac66ba2f744": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "84edc8ef4721471c823c4502c503d923": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f44bf3eef0e840f99f982fbf0910d108",
      "max": 647,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1baccf373ec24ac2ae16d761479e9944",
      "value": 647
     }
    },
    "8a51db15df8f48e197cc8cb717325044": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_21e75a90d0cb4d778e7bee589c1e375b",
      "max": 1202,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_94ca5160b56145ea91b95e69ca47dce7",
      "value": 1202
     }
    },
    "92a8a2d600bc455684ef015d9a0f13a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0759c72d6d814541981cb138ff68812e",
      "max": 11,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6e1f01f6954844cf80f938d89984166f",
      "value": 11
     }
    },
    "936b07ba5d2c4cf59d217cf99a3698b2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e6f0a509db51421793264b529cbf0d77",
      "max": 43,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_208818f9241f4b298ebc1e756c8b6faf",
      "value": 43
     }
    },
    "94ca5160b56145ea91b95e69ca47dce7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "972d29583fba4f8a9fa95181728c0aba": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0722ba5db9a940cba9fc3ec42b42efc8",
       "IPY_MODEL_6fc6cf356dbb47e2b69e42cf205ab824"
      ],
      "layout": "IPY_MODEL_c742e996d874416184c0699b242cfd05"
     }
    },
    "97a48d4d9ad44a87991ab631fc1b69bd": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9b1f5b7ff0e843d6a4d68cd1812dd354": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a3db05eee0e64d4c93481ba0c41f3868": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a7b82c77d9d94135bd5f2977585670af": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a7ea8914997e484f8a828aaf18eaa005": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "abc26b3c8a044f18896650f1c4f36d8b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ad3303422ce84d8597a9291dba86c0fc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3f8faf6e856c460b8bcde6fe430d1c74",
      "placeholder": "​",
      "style": "IPY_MODEL_06d7277c8e574430a065a9a85f88eea6",
      "value": " 1/? [00:00&lt;00:00,  3.28 tables/s]"
     }
    },
    "ad995c2518444bc2b8d64289d41edc0d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4c48350b493f4edea02d8b2d3f2608e8",
      "max": 209528,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_11fffa69385e429f82a472a7dc2f8dba",
      "value": 209528
     }
    },
    "aeec65122d2b4869993cb14bdfc3155c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c799404686804653bde731972befd439",
       "IPY_MODEL_47ca5cb5929a4f7eac1e4f9609eb3cde"
      ],
      "layout": "IPY_MODEL_72817fee75b84eac8273dd51d92bb5ac"
     }
    },
    "af6b275af17a4991a1ff3f946c0a2e61": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b335f773b4d2401c89670034e5ccb805": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b34d85775ba14711a038d153c97d4802": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b4962780811d4ecd9a80b8ad2703cc0b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b7968d32762543c3b3833c2415f7000f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b81edb02885b48d0a5e0cbcf6d3de685": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bb2b7b66bbf64ee290774aa96c0f4c00": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_20aa40592cb7449d93cd22c37274e621",
      "placeholder": "​",
      "style": "IPY_MODEL_f04b87a736a5439998716ec2dc6ba670",
      "value": " 1/? [00:02&lt;00:00,  2.40s/ tables]"
     }
    },
    "bc63c8a02c1040f68c28825b5acfd60c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2246016d810d4f25b5cf9e696252752a",
      "placeholder": "​",
      "style": "IPY_MODEL_d58718e51dbd4dc4bf7dd9ef6bde7951",
      "value": " 3.63k/? [00:00&lt;00:00, 15.3kB/s]"
     }
    },
    "bdc3db4828ce497b9f9cff6254d74acd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c0c242ccad4d454c8b7ce6152aeeb04b",
      "placeholder": "​",
      "style": "IPY_MODEL_c684af3b9ea041d48a8f9ca73a17e9b2",
      "value": " 43.0/43.0 [00:00&lt;00:00, 580B/s]"
     }
    },
    "c0c242ccad4d454c8b7ce6152aeeb04b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c1e3222db9bb4aa0a7a4e895e1e69eac": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_78cc7c33fc664fb8b45019a2d8d1a81c",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0a38ad48054b4793af78af528164f84d",
      "value": 1
     }
    },
    "c684af3b9ea041d48a8f9ca73a17e9b2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c742e996d874416184c0699b242cfd05": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c799404686804653bde731972befd439": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a3db05eee0e64d4c93481ba0c41f3868",
      "max": 1553,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_30edc0d0984a497c88e291ce1a73d8e7",
      "value": 1553
     }
    },
    "c8bd76dd65c5405b83dca2d009d7534a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "cdc77948b42d40f698b1b150b2eca9f4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_22091b5d454f44e693ee9f0a0396a447",
      "max": 112,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cf1b41ef505646738f705629b40b1c5d",
      "value": 112
     }
    },
    "cf1b41ef505646738f705629b40b1c5d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "d55529a706484c28bf664f3757575729": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ad995c2518444bc2b8d64289d41edc0d",
       "IPY_MODEL_4dc1e1d8ff9c434d81fc0cab52fefbcf"
      ],
      "layout": "IPY_MODEL_551640d85a3b42f5b8b75c184cbfed45"
     }
    },
    "d58718e51dbd4dc4bf7dd9ef6bde7951": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d5e7b71782524247912d54b7619b068b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_68c13311ca82469d89c7014efe6d78d8",
       "IPY_MODEL_271edb820d30444da7cebf1459835127"
      ],
      "layout": "IPY_MODEL_152b10e1df7b4633b408f41b194bb6b5"
     }
    },
    "d7fdbf13549240879be55b7f0b8ddbff": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e32ea0f0df12457fb401e89457a4df48": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_84edc8ef4721471c823c4502c503d923",
       "IPY_MODEL_12742631aec9417881e3e7c471b1dd77"
      ],
      "layout": "IPY_MODEL_a7b82c77d9d94135bd5f2977585670af"
     }
    },
    "e3bc9a84092049009c39758400895ff8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e4ade6af00934247bb5f8c149bfcdca1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "e6f0a509db51421793264b529cbf0d77": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f04b87a736a5439998716ec2dc6ba670": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f224c0c64f994806a13847d32c34f714": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f44bf3eef0e840f99f982fbf0910d108": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f9c22d5144824c518fad4ec9e0acc2b1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8a51db15df8f48e197cc8cb717325044",
       "IPY_MODEL_bc63c8a02c1040f68c28825b5acfd60c"
      ],
      "layout": "IPY_MODEL_b4962780811d4ecd9a80b8ad2703cc0b"
     }
    },
    "fb873d9a42cc41ea9b933278bafecbda": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4ca8b82e45ad43d2ae71989bf50d35ef",
       "IPY_MODEL_5d36626d92464b11a5e02469b7523656"
      ],
      "layout": "IPY_MODEL_6be924a67eb2410ea17092a2f412653b"
     }
    },
    "fcf5c3c66b9b49258d0f1769d3081dd1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fdc3e3f5e6284380a869f98491e1f7d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_92a8a2d600bc455684ef015d9a0f13a7",
       "IPY_MODEL_2bb323d86d914e03abe9d3677c3406a8"
      ],
      "layout": "IPY_MODEL_2f148877ca3d46e7a925633ed2467b69"
     }
    },
    "feaef1aea58e446e847a45d38d8366c9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}