The_Annotated_"Attention_is_All_You_Need".ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "The Annotated \"Attention is All You Need\".ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/elvinaqa/Hands-MediumON-ML/blob/master/The_Annotated_%22Attention_is_All_You_Need%22.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wArePITKUgQG"
      },
      "source": [
        "<img src=\"aiayn.png\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tSWEk4ttUgQH"
      },
      "source": [
        "> When teaching, I emphasize implementation as a way to understand recent developments in ML. This post is an attempt to keep myself honest along this goal. The recent [\"Attention is All You Need\"]\n",
        "(https://arxiv.org/abs/1706.03762) paper from NIPS 2017 has been instantly impactful paper as a new method for machine translation and potentiall NLP generally. The paper is very clearly written, but the conventional wisdom has been that it is quite difficult to implement correctly. \n",
        ">\n",
        "> In this post I follow the paper through from start to finish and try to implement each component in code. \n",
        "(I have done some minor reordering and skipping from the original paper). This document itself is a working notebook, and should be a completely usable and efficient implementation. To follow along you will first need to install [PyTorch](http://pytorch.org/) and [torchtext](https://github.com/pytorch/text). The complete code is available on [github](https://github.com/harvardnlp/annotated-transformer).\n",
        ">- Alexander \"Sasha\" Rush ([@harvardnlp](https://twitter.com/harvardnlp))\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZaYyfFUqUnGY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1855
        },
        "outputId": "f283707d-138b-4a18-8008-eb1857a88623"
      },
      "source": [
        "!pip install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl numpy matplotlib spacy torchtext "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting torch==0.3.0.post4 from http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl\n",
            "  Downloading http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl (592.3MB)\n",
            "\u001b[K    100% |████████████████████████████████| 592.3MB 52.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages\n",
            "Collecting spacy\n",
            "  Downloading spacy-2.0.10.tar.gz (17.5MB)\n",
            "\u001b[K    31% |██████████▎                     | 5.6MB 35.0MB/s eta 0:00:01"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[K    100% |████████████████████████████████| 17.5MB 79kB/s \n",
            "\u001b[?25hCollecting torchtext\n",
            "  Downloading torchtext-0.2.1-py3-none-any.whl (41kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 8.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from torch==0.3.0.post4)\n",
            "Requirement already satisfied: pytz in /usr/local/lib/python3.6/dist-packages (from matplotlib)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib)\n",
            "Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib)\n",
            "Collecting cymem<1.32,>=1.30 (from spacy)\n",
            "  Downloading cymem-1.31.2.tar.gz\n",
            "Requirement already satisfied: dill<0.3,>=0.2 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Collecting ftfy<5.0.0,>=4.4.2 (from spacy)\n",
            "  Downloading ftfy-4.4.3.tar.gz (50kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 7.7MB/s \n",
            "\u001b[?25hCollecting html5lib==1.0b8 (from spacy)\n",
            "  Downloading html5lib-1.0b8.tar.gz (889kB)\n",
            "\u001b[K    100% |████████████████████████████████| 890kB 1.3MB/s \n",
            "\u001b[?25hCollecting msgpack-numpy==0.4.1 (from spacy)\n",
            "  Downloading msgpack_numpy-0.4.1-py2.py3-none-any.whl\n",
            "Collecting msgpack-python==0.5.4 (from spacy)\n",
            "  Downloading msgpack-python-0.5.4.tar.gz\n",
            "Collecting murmurhash<0.29,>=0.28 (from spacy)\n",
            "  Downloading murmurhash-0.28.0.tar.gz\n",
            "Collecting pathlib (from spacy)\n",
            "  Downloading pathlib-1.0.1.tar.gz (49kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 7.6MB/s \n",
            "\u001b[?25hCollecting plac<1.0.0,>=0.9.6 (from spacy)\n",
            "  Downloading plac-0.9.6-py2.py3-none-any.whl\n",
            "Collecting preshed<2.0.0,>=1.0.0 (from spacy)\n",
            "  Downloading preshed-1.0.0.tar.gz (89kB)\n",
            "\u001b[K    100% |████████████████████████████████| 92kB 7.7MB/s \n",
            "\u001b[?25hCollecting regex==2017.4.5 (from spacy)\n",
            "  Downloading regex-2017.04.05.tar.gz (601kB)\n",
            "\u001b[K    100% |████████████████████████████████| 604kB 2.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Collecting thinc<6.11.0,>=6.10.1 (from spacy)\n",
            "  Downloading thinc-6.10.2.tar.gz (1.2MB)\n",
            "\u001b[K    100% |████████████████████████████████| 1.2MB 1.0MB/s \n",
            "\u001b[?25hCollecting ujson>=1.35 (from spacy)\n",
            "  Downloading ujson-1.35.tar.gz (192kB)\n",
            "\u001b[K    100% |████████████████████████████████| 194kB 5.6MB/s \n",
            "\u001b[?25hCollecting tqdm (from torchtext)\n",
            "  Downloading tqdm-4.19.9-py2.py3-none-any.whl (52kB)\n",
            "\u001b[K    100% |████████████████████████████████| 61kB 9.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from ftfy<5.0.0,>=4.4.2->spacy)\n",
            "Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy)\n",
            "Collecting cytoolz<0.9,>=0.8 (from thinc<6.11.0,>=6.10.1->spacy)\n",
            "  Downloading cytoolz-0.8.2.tar.gz (386kB)\n",
            "\u001b[K    100% |████████████████████████████████| 389kB 2.5MB/s \n",
            "\u001b[?25hRequirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from thinc<6.11.0,>=6.10.1->spacy)\n",
            "Collecting wrapt (from thinc<6.11.0,>=6.10.1->spacy)\n",
            "  Downloading wrapt-1.10.11.tar.gz\n",
            "Collecting toolz>=0.8.0 (from cytoolz<0.9,>=0.8->thinc<6.11.0,>=6.10.1->spacy)\n",
            "  Downloading toolz-0.9.0.tar.gz (45kB)\n",
            "\u001b[K    100% |████████████████████████████████| 51kB 4.2MB/s \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25hBuilding wheels for collected packages: spacy, cymem, ftfy, html5lib, msgpack-python, murmurhash, pathlib, preshed, regex, thinc, ujson, cytoolz, wrapt, toolz\n",
            "  Running setup.py bdist_wheel for spacy ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/a3/a3/c0/07e3ef38ab89f9339e73393d0fb04f5eccd1a0334d78b55b52\n",
            "  Running setup.py bdist_wheel for cymem ... \u001b[?25l-\b \b\\\b \b|\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/4b/2a/0e/dce3ff7a6f0f916906ef978afe512a7b5aef8abfd9ba988acf\n",
            "  Running setup.py bdist_wheel for ftfy ... \u001b[?25l-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/ae/d7/4c/339066248431397227741c7fdc80ad85826188ee9b0c24b4c7\n",
            "  Running setup.py bdist_wheel for html5lib ... \u001b[?25l-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/d4/d1/0b/a6b6f9f204af55c9bb8c97eae2a78b690b7150a7b850bb9403\n",
            "  Running setup.py bdist_wheel for msgpack-python ... \u001b[?25l-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/49/fc/5d/07243edfc74536e0776f3f8901418e7596de0e3b887dd17d86\n",
            "  Running setup.py bdist_wheel for murmurhash ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/67/30/e2/ef52a408eda4b23581669abfd1c1516a931d9e53f2e7616cb9\n",
            "  Running setup.py bdist_wheel for pathlib ... \u001b[?25l-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/2a/23/a5/d8803db5d631e9f391fe6defe982a238bf5483062eeb34e841\n",
            "  Running setup.py bdist_wheel for preshed ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/97/64/22/20fabf1f51039b799e64e46d0381b023cfdbe159c349d7c135\n",
            "  Running setup.py bdist_wheel for regex ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/28/20/68/c71f468f76d9bd81730a7633ace0ad30507cf99166314109a1\n",
            "  Running setup.py bdist_wheel for thinc ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/f8/fc/92/3bb08540cc5ac05df781005e686273adcd97af91ca2c032154\n",
            "  Running setup.py bdist_wheel for ujson ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/9e/9b/d0/df92653bb5b2664c15d8ee5b99e3f2eb08a034444db8922b2f\n",
            "  Running setup.py bdist_wheel for cytoolz ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/52/92/ed/661ecb7a67b42b21fc3dea140abb9ae9b8e94e72f0b3aff6c1\n",
            "  Running setup.py bdist_wheel for wrapt ... \u001b[?25l-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/56/e1/0f/f7ccf1ed8ceaabccc2a93ce0481f73e589814cbbc439291345\n",
            "  Running setup.py bdist_wheel for toolz ... \u001b[?25l-\b \b\\\b \bdone\n",
            "\u001b[?25h  Stored in directory: /content/.cache/pip/wheels/57/51/8a/433a9c0a2c65fc1b2a795ae036b932f3339a02e9ae88367659\n",
            "Successfully built spacy cymem ftfy html5lib msgpack-python murmurhash pathlib preshed regex thinc ujson cytoolz wrapt toolz\n",
            "Installing collected packages: torch, cymem, html5lib, ftfy, msgpack-python, msgpack-numpy, murmurhash, pathlib, plac, preshed, regex, toolz, cytoolz, tqdm, wrapt, thinc, ujson, spacy, torchtext\n",
            "  Found existing installation: html5lib 0.9999999\n",
            "    Uninstalling html5lib-0.9999999:\n",
            "      Successfully uninstalled html5lib-0.9999999\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4LTc4HW7UgQI"
      },
      "source": [
        "# Standard PyTorch imports\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import math, copy\n",
        "from torch.autograd import Variable\n",
        "\n",
        "# For plots\n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_R749nLNUgQL"
      },
      "source": [
        "* Table of Contents                               \n",
        "{:toc}      "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "esxhOQubUgQL"
      },
      "source": [
        "# Background"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1M-PiEMOUgQM"
      },
      "source": [
        "The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU\n",
        "[16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building\n",
        "block, computing hidden representations in parallel for all input and output positions. In these models,\n",
        "the number of operations required to relate signals from two arbitrary input or output positions grows\n",
        "in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes\n",
        "it more difficult to learn dependencies between distant positions [12]. In the Transformer this is\n",
        "reduced to a constant number of operations, albeit at the cost of reduced effective resolution due\n",
        "to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as\n",
        "described in section 3.2.\n",
        "\n",
        "Self-attention, sometimes called intra-attention is an attention mechanism relating different positions\n",
        "of a single sequence in order to compute a representation of the sequence. Self-attention has been\n",
        "used successfully in a variety of tasks including reading comprehension, abstractive summarization,\n",
        "textual entailment and learning task-independent sentence representations [4, 27, 28, 22].\n",
        "End-to-end memory networks are based on a recurrent attention mechanism instead of sequencealigned\n",
        "recurrence and have been shown to perform well on simple-language question answering and\n",
        "language modeling tasks [34].\n",
        "\n",
        "To the best of our knowledge, however, the Transformer is the first transduction model relying\n",
        "entirely on self-attention to compute representations of its input and output without using sequencealigned\n",
        "RNNs or convolution. In the following sections, we will describe the Transformer, motivate\n",
        "self-attention and discuss its advantages over models such as [17, 18] and [9]."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84vTAA5TUgQM"
      },
      "source": [
        "# Model Architecture"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-f9BuNsUgQN"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VBOvyU9BUgQN"
      },
      "source": [
        "Most competitive neural sequence transduction models have an encoder-decoder structure [(cite)](cho2014learning,bahdanau2014neural,sutskever14). Here, the encoder maps an input sequence of symbol representations $(x_1, ..., x_n)$ to a sequence of continuous representations $\\mathbf{z} = (z_1, ..., z_n)$. Given $\\mathbf{z}$, the decoder then generates an output sequence $(y_1,...,y_m)$ of symbols one element at a time. At each step the model is auto-regressive [(cite)](graves2013generating), consuming the previously generated symbols as additional input when generating the next. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1AC8KeDJUgQO"
      },
      "source": [
        "class EncoderDecoder(nn.Module):\n",
        "    \"\"\"\n",
        "    A standard Encoder-Decoder architecture. Base model for this and many \n",
        "    other models.\n",
        "    \"\"\"\n",
        "    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):\n",
        "        super(EncoderDecoder, self).__init__()\n",
        "        self.encoder = encoder\n",
        "        self.decoder = decoder\n",
        "        self.src_embed = src_embed\n",
        "        self.tgt_embed = tgt_embed\n",
        "        self.generator = generator\n",
        "        \n",
        "    def forward(self, src, tgt, src_mask, tgt_mask):\n",
        "        \"Take in and process masked src and target sequences.\"\n",
        "        memory = self.encoder(self.src_embed(src), src_mask)\n",
        "        output = self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)\n",
        "        return output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ip0EXqvEUgQQ"
      },
      "source": [
        "The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q9dznVhQUgQQ"
      },
      "source": [
        "<img src=\"ModalNet-21.png\" width=400px>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "euyRXbaMUgQR"
      },
      "source": [
        "## Encoder and Decoder Stacks   \n",
        "\n",
        "### Encoder: \n",
        "\n",
        "The encoder is composed of a stack of $N=6$ identical layers. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6yy7pY85UgQR"
      },
      "source": [
        "def clones(module, N):\n",
        "    \"Produce N identical layers.\"\n",
        "    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "psiq5idJUgQT"
      },
      "source": [
        "class Encoder(nn.Module):\n",
        "    \"Core encoder is a stack of N layers\"\n",
        "    def __init__(self, layer, N):\n",
        "        super(Encoder, self).__init__()\n",
        "        self.layers = clones(layer, N)\n",
        "        self.norm = LayerNorm(layer.size)\n",
        "        \n",
        "    def forward(self, x, mask):\n",
        "        \"Pass the input (and mask) through each layer in turn.\"\n",
        "        for layer in self.layers:\n",
        "            x = layer(x, mask)\n",
        "        return self.norm(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mie9sUeSUgQV"
      },
      "source": [
        "We employ a residual connection [(cite)](he2016deep) around each of the two sub-layers, followed by layer normalization [(cite)](layernorm2016).  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sEz9kLClUgQV"
      },
      "source": [
        "class LayerNorm(nn.Module):\n",
        "    \"Construct a layernorm module (See citation for details).\"\n",
        "    def __init__(self, features, eps=1e-6):\n",
        "        super(LayerNorm, self).__init__()\n",
        "        self.a_2 = nn.Parameter(torch.ones(features))\n",
        "        self.b_2 = nn.Parameter(torch.zeros(features))\n",
        "        self.eps = eps\n",
        "\n",
        "    def forward(self, x):\n",
        "        mean = x.mean(-1, keepdim=True)\n",
        "        std = x.std(-1, keepdim=True)\n",
        "        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nx4On5PCUgQY"
      },
      "source": [
        "That is, the output of each sub-layer is $\\mathrm{LayerNorm}(x + \\mathrm{Sublayer}(x))$, where $\\mathrm{Sublayer}(x)$ is the function implemented by the sub-layer itself.  We apply dropout [(cite)](srivastava2014dropout) to the output of each sub-layer, before it is added to the sub-layer input and normalized.  \n",
        "\n",
        "To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension $d_{\\text{model}}=512$.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zx9JBwAcUgQY"
      },
      "source": [
        "class SublayerConnection(nn.Module):\n",
        "    \"\"\"\n",
        "    A residual connection followed by a layer norm.\n",
        "    Note for code simplicity we apply the norm first as opposed to last.\n",
        "    \"\"\"\n",
        "    def __init__(self, size, dropout):\n",
        "        super(SublayerConnection, self).__init__()\n",
        "        self.norm = LayerNorm(size)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, x, sublayer):\n",
        "        \"Apply residual connection to any sublayer function that maintains the same size.\"\n",
        "        return x + self.dropout(sublayer(self.norm(x)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZduB6mIlUgQa"
      },
      "source": [
        "Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0mEBw9tIUgQb"
      },
      "source": [
        "class EncoderLayer(nn.Module):\n",
        "    \"Encoder is made up of two sublayers, self-attn and feed forward (defined below)\"\n",
        "    def __init__(self, size, self_attn, feed_forward, dropout):\n",
        "        super(EncoderLayer, self).__init__()\n",
        "        self.self_attn = self_attn\n",
        "        self.feed_forward = feed_forward\n",
        "        self.sublayer = clones(SublayerConnection(size, dropout), 2)\n",
        "        self.size = size\n",
        "\n",
        "    def forward(self, x, mask):\n",
        "        \"Follow Figure 1 (left) for connections.\"\n",
        "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))\n",
        "        return self.sublayer[1](x, self.feed_forward)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iHOZnpnBUgQd"
      },
      "source": [
        "### Decoder:\n",
        "\n",
        "The decoder is also composed of a stack of $N=6$ identical layers.  \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3o_ZB42sUgQd"
      },
      "source": [
        "class Decoder(nn.Module):\n",
        "    \"Generic N layer decoder with masking.\"\n",
        "    def __init__(self, layer, N):\n",
        "        super(Decoder, self).__init__()\n",
        "        self.layers = clones(layer, N)\n",
        "        self.norm = LayerNorm(layer.size)\n",
        "        \n",
        "    def forward(self, x, memory, src_mask, tgt_mask):\n",
        "        for layer in self.layers:\n",
        "            x = layer(x, memory, src_mask, tgt_mask)\n",
        "        return self.norm(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dOi_W1qaUgQf"
      },
      "source": [
        "In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack.  Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMm6xHWVUgQg"
      },
      "source": [
        "class DecoderLayer(nn.Module):\n",
        "    \"Decoder is made up of three sublayers, self-attn, src-attn, and feed forward (defined below)\"\n",
        "    def __init__(self, size, self_attn, src_attn, feed_forward, dropout):\n",
        "        super(DecoderLayer, self).__init__()\n",
        "        self.size = size\n",
        "        self.self_attn = self_attn\n",
        "        self.src_attn = src_attn\n",
        "        self.feed_forward = feed_forward\n",
        "        self.sublayer = clones(SublayerConnection(size, dropout), 3)\n",
        " \n",
        "    def forward(self, x, memory, src_mask, tgt_mask):\n",
        "        \"Follow Figure 1 (right) for connections.\"\n",
        "        m = memory\n",
        "        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, tgt_mask))\n",
        "        x = self.sublayer[1](x, lambda x: self.src_attn(x, m, m, src_mask))\n",
        "        return self.sublayer[2](x, self.feed_forward)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Nen9h7wUgQi"
      },
      "source": [
        "We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions.  This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position $i$ can depend only on the known outputs at positions less than $i$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RyQKI9AgUgQj"
      },
      "source": [
        "def subsequent_mask(size):\n",
        "    \"Mask out subsequent positions.\"\n",
        "    attn_shape = (1, size, size)\n",
        "    subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')\n",
        "    return torch.from_numpy(subsequent_mask) == 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fgQMtvM-UgQl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 339
        },
        "outputId": "07611646-88f8-4410-8bd9-77bbe59b86cd"
      },
      "source": [
        "# The attention mask shows the position each tgt word (row) is allowed to look at (column).\n",
        "# Words are blocked for attending to future words during training. \n",
        "plt.figure(figsize=(5,5))\n",
        "plt.imshow(subsequent_mask(20)[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.image.AxesImage at 0x7f181c187cc0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEvCAYAAADRrN1JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAE4BJREFUeJzt3XvMXHWdx/H3WCSYKnIRKLDGBiXf\nSJ7nD0ESGpVWIaKEhESqG1PJihiy4RKypCGY3YAVRRGxCWAkRi6RDeFiuVRtECwGokJAg2YU+C5L\nxDQthK5EpIjQy+wfcyrDOPNczlyY59f3KyHM/H5n5nx7Zp7P/H5zLtNotVpIUone8mYXIEmjYsBJ\nKpYBJ6lYBpykYhlwkoplwEkq1l5vdgG7NRqNvserNJtNpqene/Z5mIskoNGzcVICYqaAa7VaNBo9\n6zfgJEGfgHOKKqlYtaeoEbEWOA5oAedn5qMdfScClwE7gQ2ZeemghUrSfNUawUXEcuDIzFwGnAlc\n1bXIVcBpwIeAj0fEUQNVKUk11J2ingDcBZCZTwD7R8S+ABFxBPBCZm7KzF3Ahmp5SRqrugG3BNja\ncX9r1dar73ng0JrrkaTahrWTofcuztn7/qHZbNJqtXr+B8zYJ0m91N3JsIXXR2wAhwHP9uk7vGqb\nUb/j3MDDRCTVU3cEdy+wEiAijga2ZOZLAJn5DLBvRCyNiL2AU6rlJWmsah/oGxHfAI4HdgHnAB8A\nXszMOyPieODyatF1mfmtWQvxQF9J9Xkmg6RieSaDpD2LASepWBNzNZG6+k1dZ+K0VtozOIKTVCwD\nTlKxDDhJxTLgJBXLgJNULANOUrEMOEnFMuAkFcuAk1QsA05SsQw4ScUy4CQVa8GfbF9HnRP0wZP0\npYXGEZykYhlwkoplwEkqlgEnqVgGnKRiGXCSimXASSqWASepWAacpGLVPpMhIr4JfKR6jq9n5h0d\nfc8Am4CdVdOqzNxcv0xJmr9aARcRHwWmMnNZRBwIPAbc0bXYJzNz26AFSlJddaeoDwKfrm7/BVgc\nEYuGU5IkDUetEVxm7gReru6eCWyo2jpdGxFLgV8AX8pMz1SXNFYD7WSIiFNpB9y5XV0XAxcAK4Ap\n4LTZnqvZbNJqtXr+B/TtG+d/khaWRt0/3Ig4CbgU+ERmvjDDcmcDh2TmJTMW0mj0LaTVatW+xNEw\n1anDYJTGoucfZq0RXES8E7gCOKU73CLinRHx04jYu2paDvy+znokaRB1DxP5V+BdwG0RsbvtfqCZ\nmXdGxAbg4Yh4hfYe1h8OXKkkzVPtKeqwOUWVNIDhTVElaSEw4CQVy4CTVCwDTlKxDDhJxTLgJBXL\ngJNULANOUrFqX/BSc1P3AGUPEJYG5whOUrEMOEnFMuAkFcuAk1QsA05SsQw4ScUy4CQVy4CTVCwD\nTlKxDDhJxTLgJBXLgJNULANOUrG8msiEmukqJDP9fKFXIZFe5whOUrEMOEnFqjVFjYgVwO3AH6qm\nZmae19F/InAZsBPYkJmXDlinJM3bIN/BPZCZK/v0XQWcBGwGHoiIdZn5+ADrkqR5G/oUNSKOAF7I\nzE2ZuQvYAJww7PVI0mwGGcEdFRHrgQOANZl5X9W+BNjasdzzwHsHWI8k1VI34J4C1gC3AUcAP4+I\n92Xmaz2WndOvrjSbTaampvr2T8rhD9YhLRy1Ai4zNwO3VnefjojngMOBPwJbaI/idju8apvR9PR0\n376Zjvsap4VQh8Enva7Wd3ARsSoiVle3lwCH0N6hQGY+A+wbEUsjYi/gFODe4ZQrSXPXqPOJHxHv\nAG4G9gP2pj1dPRh4MTPvjIjjgcurxddl5rdmLaTR6FvIQhg5TUodjuC0h+r5B1Er4EbBgBtOHZPy\nekpj1vMPwjMZJBXLgJNULK8mUpg6U2intSqVIzhJxTLgJBXLgJNULANOUrEMOEnFMuAkFcuAk1Qs\nA05SsQw4ScUy4CQVy4CTVCwDTlKxPNleta9x50n6mnSO4CQVy4CTVCwDTlKxDDhJxTLgJBXLgJNU\nLANOUrEMOEnFMuAkFavWmQwRcSZwekfTBzPz7R3924FfdvSfkJk765UoSfXUCrjMvA64DiAilgOf\n6VrkxcxcMVhpkjSYYZyLejGwagjPI0lDNVDARcSxwKbMfK6ra5+IuBl4D7AuM789yHokqY5BR3Bf\nBG7s0b4a+G+gBTwYEQ9m5q9neqJms8nU1FTf/km5coV1SAvHoAG3AjivuzEzr919OyI2AtPAjAE3\nPT3dt6/VatW+pM8wWcfgdRjMGqfaARcRhwHbMvO1rvYALqH9vdwi4EPADwcpUpLqGGQEdyjw/O47\nEXER8EBmPhQRm4BHgF3A+sx8ZLAyJWn+GpMyZWg0Gn0LWchTMuv458dII9DzjeiZDJKKZcBJKpYB\nJ6lYBpykYhlwkoplwEkqlgEnqVgGnKRiDeNySdKc1T1A2QOEVYcjOEnFMuAkFcuAk1QsA05SsQw4\nScUy4CQVy4CTVCwDTlKxDDhJxTLgJBXLgJNULANOUrEMOEnF8moiWhBmugrJTD9f6FVI9myO4CQV\ny4CTVKw5TVEjYgq4G1ibmddExLuBm4BFwLPA6Zn5atdj1gLHAS3g/Mx8dKiVS9IsZh3BRcRi4Gpg\nY0fzV4DvZOZHgP8FvtD1mOXAkZm5DDgTuGpoFUvSHM1livoqcDKwpaNtBbC+uv0j4MSux5wA3AWQ\nmU8A+0fEvgNVKknzNGvAZeaOzHylq3lxx5T0eeDQrv4lwNaO+1urNkkam2EcJjKXXxGZdZlms8nU\n1FTf/knZ3W8db2QdmmR1A25bRLytGtkdzhunr1T3O0dsh9HeGdHX9PR0376ZjnMaJ+tYeHUYfHu2\nuoeJ/Aw4rbp9GnBPV/+9wEqAiDga2JKZL9VclyTV0pjtEy4ijgGuBJYC24HNwCrgRmAf4E/AGZm5\nPSJuqW6/EhHfAI4HdgHnZObvZiyk0ehbyEIYKVjHZNbhCG6P0fMNMGvAjYsBZx2jqGNS3t8auZ5v\nAM9kkFQsA05SsbyaiIpWZwrttLYcjuAkFcuAk1QsA05SsQw4ScUy4CQVy4CTVCwDTlKxDDhJxTLg\nJBXLgJNULANOUrEMOEnF8mR7qYsn6JfDEZykYhlwkoplwEkqlgEnqVgGnKRiGXCSimXASSqWASep\nWAacpGLN6UyGiJgC7gbWZuY1EfFu4AbgrcB24HOZ+VzH8iuA24E/VE3NzDxvmIVL0mxmDbiIWAxc\nDWzsaP4q8L3MvC0izgEuAC7seugDmblyaJVK0jzNZYr6KnAysKWj7WxgXXV7K3DgkOuSpIHNOoLL\nzB3AjojobHsZICIWAecAX+nx0KMiYj1wALAmM+8bSsWSNEe1ryZShdtNwP2ZubGr+ylgDXAbcATw\n84h4X2a+1u/5ms0mU1NTfdc3KVdrsI43sg5NskEul3QD8FRmrunuyMzNwK3V3acj4jngcOCP/Z5s\nenq674parVatS9gMm3VYx7BrMJhHq9ZhIhGxCngtMy/p1x8Rq6vbS4BDgM21q5SkGhqzfYJExDHA\nlcBS2oeEbAYOBv4O/LVa7PHMPDsibgHOoD0yvBnYD9ib9ndwG2YspNHoW8gkfEJbh3WMogZHcEPT\nc+PPGnDjYsBZx0Kuw4B70/Xc+J7JIKlYBpykYhlwkoplwEkqlgEnqVgGnKRiGXCSimXASSrWIOei\nShpQ3QOUPUB4bhzBSSqWASepWAacpGIZcJKKZcBJKpYBJ6lYBpykYhlwkoplwEkqlgEnqVgGnKRi\nGXCSimXASSqWVxORFqB+VyGZ7ecL97SrkDiCk1QsA05SseY0RY2IKeBuYG1mXhMRNwLHAH+uFrki\nM3/S9Zi1wHFACzg/Mx8dWtWSNAezBlxELAauBjZ2dX0pM3/c5zHLgSMzc1lEvB+4Hlg2aLGSNB9z\nmaK+CpwMbJnH854A3AWQmU8A+0fEvvMvT5LqmzXgMnNHZr7So+vciLg/Im6JiHd19S0Btnbc31q1\nSdLY1D1M5Cbgz5n524i4CPgycO4My8/6yxrNZpOpqam+/ZOye9s63sg6JqsGmJw6JkGtgMvMzu/j\n1gPf7VpkC28csR0GPDvTc05PT/ftm+3YnnGxDuuY5BrmUseeFn61DhOJiHURcUR1dwXw+65F7gVW\nVsseDWzJzJfqFilJdcxlL+oxwJXAUmB7RKykvVf11oj4G7ANOKNa9hbgjMz8VUT8JiJ+BewCzhlR\n/ZLUV2NShqyNRqNvIQtl+G8de24dk1DDXOqYlL/3Eej5j/ZMBknFMuAkFcuriUh7kDrT6IU8rXUE\nJ6lYBpykYhlwkoplwEkqlgEnqVgGnKRiGXCSimXASSqWASepWAacpGIZcJKKZcBJKpYn20ua0UI+\nQd8RnKRiGXCSimXASSqWASepWAacpGIZcJKKZcBJKpYBJ6lYBpykYs3pTIaImALuBtZm5jURcTtw\nUNV9APBwZp7VsfzngUuBp6um+zLza0OrWpLmYNaAi4jFwNXAxt1tmfnpjv7rge/3eOitmbl6GEVK\nUh1zmaK+CpwMbOnuiIgA9svMR4ZdmCQNatYRXGbuAHa0s+yfnE97dNfL8oi4B3grsDozH6tdpSTV\nUPtqIhGxN/DhzDy7R/fDwNbM/ElELAN+AEzP9HzNZpOpqam+/ZNydQLreCPrmKwaYHLqmASDXC5p\nOdBzapqZTwJPVrcfioiDImJRZu7s92TT0/3zr9Vq1bpky7BZh3VMcg0l1DHscB7kMJFjgd/16oiI\nCyPis9XtKdqjub7hJkmjMJe9qMcAVwJLge0RsRL4FHAorx8GsnvZuzPzVOBm4KaI+PdqHWcOuW5J\nmlVjUubrjUajbyELfdhtHeXXMQk1lFDHAHnUc2WeySCpWAacpGIZcJKKZcBJKpYBJ6lYBpykYhlw\nkoplwEkq1iDnokrSUNU9SLnfAcKO4CQVy4CTVCwDTlKxDDhJxTLgJBXLgJNULANOUrEMOEnFMuAk\nFcuAk1QsA05SsQw4ScUy4CQVa2J+NlCShs0RnKRiGXCSimXASSqWASepWAacpGIZcJKKNVE/OhMR\na4HjgBZwfmY+2tF3InAZsBPYkJmXjrCObwIfob19vp6Zd3T0PQNsquoAWJWZm0dQwwrgduAPVVMz\nM8/r6B/L9oiIM4HTO5o+mJlv7+jfDvyyo/+EzNzJkETEFHA3sDYzr4mIdwM3AYuAZ4HTM/PVrsf0\nfR8NuY4bgLcC24HPZeZzHcuvYIbXb4h13AgcA/y5WuSKzPxJ12PGsT1uBw6qug8AHs7MszqW/zxw\nKfB01XRfZn5t0DpmMzEBFxHLgSMzc1lEvB+4HljWschVwEnAZuCBiFiXmY+PoI6PAlNVHQcCjwF3\ndC32yczcNux19/BAZq7s0zeW7ZGZ1wHXwT9eo890LfJiZq4Y9nqr9S0GrgY2djR/BfhOZt4eEZcB\nXwC+2/GY2d5Hw6rjq8D3MvO2iDgHuAC4sOuhM71+w6oD4EuZ+eM+jxnL9sjMT3f0Xw98v8dDb83M\n1YOse74maYp6AnAXQGY+AewfEfsCRMQRwAuZuSkzdwEbquVH4UFg94v1F2BxRCwa0bpqGfP26HQx\n7U/hcXkVOBnY0tG2Alhf3f4RcGLXY/q+j4Zcx9nAuur2VuDAAddRt47ZjGt7ABARAeyXmY8MuI6h\nmJgRHLAE+E3H/a1V21+r/2/t6HseeO8oiqimVy9Xd8+kPf3rnnJdGxFLgV/Q/vQc1ekgR0XEetpD\n/jWZeV/VPrbtsVtEHAts6pyGVfaJiJuB9wDrMvPbw1pnZu4AdrT/Zv5hcceU9Hng0K6HzfQ+Glod\nmfkyQPXhdw7tkWW3fq/f0OqonBsRF9DeHudm5v919I1le3Q4n/borpflEXEP7Wn96sx8rG4NczVJ\nI7huM/0CbL1fh52HiDiVdsCd29V1Me3pyApgCjhtRCU8BawBTgX+DbguIvbus+zItwfwReDGHu2r\ngbOAjwOrIuKDY6hlt7n8u0e2bapwuwm4PzO7p43zef0GcRNwUWZ+DPgt8OVZlh/l9tgb+HBm/rxH\n98PAlzPzE8B/AT8YVR2dJmkEt4X2J8tuh9H+ErlX3+HMb5g+LxFxEvCfwCcy88XOvsz8QcdyG4Bp\n4IfDrqHacXFrdffpiHiO9r/7j4x5e1RWAP/0JXlmXrv7dkRspL09fj3COrZFxNsy8xV6/7tneh8N\n2w3AU5m5prtjltdvaLqCdT0d30dWxrk9lgM9p6aZ+STwZHX7oYg4KCIWDXOHVC+TNIK7F1gJEBFH\nA1sy8yWAzHwG2DcilkbEXsAp1fJDFxHvBK4ATsnMF7r7IuKnHZ/Ey4Hfj6iOVRGxurq9BDiE9g6F\nsW6Pav2HAdsy87Wu9oiImyOiUdXxIV7fazgqP+P1UfNpwD1d/X3fR8MUEauA1zLzkn79/V6/Idex\nrvpOFtofQt3vx7Fsj8qxwO/61HlhRHy2uj0FbB11uMGEXU0kIr4BHA/sov29xgdo76W7MyKOBy6v\nFl2Xmd8aUQ1n0R7m/09H8/20d/PfGRHn055yvEJ7D+t5o/gOLiLeAdwM7AfsTXu6czBj3h5VLccA\nX83MT1b3L6K9h/ChiLgc+Bjt12z9MHf9V+u9ElhK+1CMzcAq2lPlfYA/AWdk5vaIuKW6/Ur3+ygz\ne/7RDVjHwcDfef27rMcz8+zdddCeHb3h9cvMDSOo42rgIuBvwDba2+D5N2F7fIr2e/QXmXlrx7J3\nZ+apEfEvtKfTb6G9bf5jHDsiJirgJGmYJmmKKklDZcBJKpYBJ6lYBpykYhlwkoplwEkqlgEnqVgG\nnKRi/T+ZPco/IHY/EwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f181c2322e8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fl2E1IaqUgQq"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3twSbimFUgQq"
      },
      "source": [
        "### Attention:                                                                                                                                                                                                                                                                               \n",
        "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.  The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.                                                                                                                                                                                                                                                                                           \n",
        "\n",
        "We call our particular attention \"Scaled Dot-Product Attention\".   The input consists of queries and keys of dimension $d_k$, and values of dimension $d_v$.  We compute the dot products of the query with all keys, divide each by $\\sqrt{d_k}$, and apply a softmax function to obtain the weights on the values.                                                                                                         \n",
        "<img width=\"220px\" src=\"ModalNet-19.png\">\n",
        "\n",
        "In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix $Q$.   The keys and values are also packed together into matrices $K$ and $V$.  We compute the matrix of outputs as:                      \n",
        "                                                                 \n",
        "$$                                                                         \n",
        "   \\mathrm{Attention}(Q, K, V) = \\mathrm{softmax}(\\frac{QK^T}{\\sqrt{d_k}})V               \n",
        "$$                                                                                                                                                                                                        \n",
        "                                                                                                                                                                     "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wlZ8zw9PUgQr"
      },
      "source": [
        "def attention(query, key, value, mask=None, dropout=0.0):\n",
        "    \"Compute 'Scaled Dot Product Attention'\"\n",
        "    d_k = query.size(-1)\n",
        "    scores = torch.matmul(query, key.transpose(-2, -1)) \\\n",
        "             / math.sqrt(d_k)\n",
        "    if mask is not None:\n",
        "        scores = scores.masked_fill(mask == 0, -1e9)\n",
        "    p_attn = F.softmax(scores, dim = -1)\n",
        "    # (Dropout described below)\n",
        "    p_attn = F.dropout(p_attn, p=dropout)\n",
        "    return torch.matmul(p_attn, value), p_attn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AV7cIqbrUgQs"
      },
      "source": [
        "The two most commonly used attention functions are additive attention [(cite)](bahdanau2014neural), and dot-product (multiplicative) attention.  Dot-product attention is identical to our algorithm, except for the scaling factor of $\\frac{1}{\\sqrt{d_k}}$. Additive attention computes the compatibility function using a feed-forward network with a single hidden layer.  While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.                                                                                             \n",
        "\n",
        "                                                                        \n",
        "While for small values of $d_k$ the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of $d_k$ [(cite)](DBLP:journals/corr/BritzGLL17). We suspect that for large values of $d_k$, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients  (To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean $0$ and variance $1$.  Then their dot product, $q \\cdot k = \\sum_{i=1}^{d_k} q_ik_i$, has mean $0$ and variance $d_k$.). To counteract this effect, we scale the dot products by $\\frac{1}{\\sqrt{d_k}}$.          "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OV7kNMbKUgQt"
      },
      "source": [
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uiaCxaGGUgQt"
      },
      "source": [
        "### Multi-Head Attention                                                                                                                                                                                                                                                                                                                                                                                                                               \n",
        "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
        "Instead of performing a single attention function with $d_{\\text{model}}$-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values $h$ times with different, learned linear projections to $d_k$, $d_k$ and $d_v$ dimensions, respectively.                                                                                                                                                                                                   \n",
        "On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding $d_v$-dimensional output values. These are concatenated and once again projected, resulting in the final values:\n",
        "\n",
        "<img width=\"270px\" src=\"ModalNet-20.png\">\n",
        "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
        "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.                                                                                                                                                                                                                                                                                             \n",
        "    \n",
        "    \n",
        "   \n",
        "$$    \n",
        "\\mathrm{MultiHead}(Q, K, V) = \\mathrm{Concat}(\\mathrm{head_1}, ..., \\mathrm{head_h})W^O    \\\\                                           \n",
        "    \\text{where}~\\mathrm{head_i} = \\mathrm{Attention}(QW^Q_i, KW^K_i, VW^V_i)                                \n",
        "$$                                                                                                                                                                                                                                                                                                                                                                         \n",
        "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
        "Where the projections are parameter matrices $W^Q_i \\in \\mathbb{R}^{d_{\\text{model}} \\times d_k}$, $W^K_i \\in \\mathbb{R}^{d_{\\text{model}} \\times d_k}$, $W^V_i \\in \\mathbb{R}^{d_{\\text{model}} \\times d_v}$ and $W^O \\in \\mathbb{R}^{hd_v \\times d_{\\text{model}}}$.                                                                                                                                                                                                                                                        \n",
        "   \n",
        "\n",
        "   \n",
        "In this work we employ $h=8$ parallel attention layers, or heads. For each of these we use $d_k=d_v=d_{\\text{model}}/h=64$. Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality.   "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_ea0UrEgUgQt"
      },
      "source": [
        "class MultiHeadedAttention(nn.Module):\n",
        "    def __init__(self, h, d_model, dropout=0.1):\n",
        "        \"Take in model size and number of heads.\"\n",
        "        super(MultiHeadedAttention, self).__init__()\n",
        "        assert d_model % h == 0\n",
        "        # We assume d_v always equals d_k\n",
        "        self.d_k = d_model // h\n",
        "        self.h = h\n",
        "        self.p = dropout\n",
        "        self.linears = clones(nn.Linear(d_model, d_model), 4)\n",
        "        self.attn = None\n",
        "        \n",
        "    def forward(self, query, key, value, mask=None):\n",
        "        \"Implements Figure 2\"\n",
        "        if mask is not None:\n",
        "            # Same mask applied to all h heads.\n",
        "            mask = mask.unsqueeze(1)\n",
        "        nbatches = query.size(0)\n",
        "        \n",
        "        # 1) Do all the linear projections in batch from d_model => h x d_k \n",
        "        query, key, value = [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)\n",
        "                             for l, x in zip(self.linears, (query, key, value))]\n",
        "        \n",
        "        # 2) Apply attention on all the projected vectors in batch. \n",
        "        x, self.attn = attention(query, key, value, mask=mask, dropout=self.p)\n",
        "        \n",
        "        # 3) \"Concat\" using a view and apply a final linear. \n",
        "        x = x.transpose(1, 2).contiguous().view(nbatches, -1, self.h * self.d_k)\n",
        "        return self.linears[-1](x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVdpQ_KwUgQv"
      },
      "source": [
        "### Applications of Attention in our Model                                                                                                                                                      \n",
        "The Transformer uses multi-head attention in three different ways:                                                        \n",
        "1) In \"encoder-decoder attention\" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder.   This allows every position in the decoder to attend over all positions in the input sequence.  This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [(cite)](wu2016google, bahdanau2014neural,JonasFaceNet2017).    \n",
        "\n",
        "\n",
        "2) The encoder contains self-attention layers.  In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder.   Each position in the encoder can attend to all positions in the previous layer of the encoder.                                                   \n",
        "\n",
        "\n",
        "3) Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position.  We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.  We implement this inside of scaled dot-product attention by masking out (setting to $-\\infty$) all values in the input of the softmax which correspond to illegal connections.                                                                                                                                                                                                                                                      "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gERLhK-FUgQw"
      },
      "source": [
        "## Position-wise Feed-Forward Networks                                                                                                                                                                                                                                                                                                                                                             \n",
        "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
        "In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically.  This consists of two linear transformations with a ReLU activation in between.\n",
        "\n",
        "$$\\mathrm{FFN}(x)=\\max(0, xW_1 + b_1) W_2 + b_2$$                                                                                                                                                                                                                                                         \n",
        "                                                                                                                                                                                                                                                        \n",
        "While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1.  The dimensionality of input and output is $d_{\\text{model}}=512$, and the inner-layer has dimensionality $d_{ff}=2048$. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HuDPthO2UgQx"
      },
      "source": [
        "class PositionwiseFeedForward(nn.Module):\n",
        "    \"Implements FFN equation.\"\n",
        "    def __init__(self, d_model, d_ff, dropout=0.1):\n",
        "        super(PositionwiseFeedForward, self).__init__()\n",
        "        # Torch linears have a `b` by default. \n",
        "        self.w_1 = nn.Linear(d_model, d_ff)\n",
        "        self.w_2 = nn.Linear(d_ff, d_model)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.w_2(self.dropout(F.relu(self.w_1(x))))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68VLwifsUgQz"
      },
      "source": [
        "## Embeddings and Softmax                                                                                                                                                                                                                                                                                           \n",
        "Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{\\text{model}}$.  We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities.  In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [(cite)](press2016using). In the embedding layers, we multiply those weights by $\\sqrt{d_{\\text{model}}}$.                                                                                                                                 "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sl5JzPeGUgQz"
      },
      "source": [
        "class Embeddings(nn.Module):\n",
        "    def __init__(self, d_model, vocab):\n",
        "        super(Embeddings, self).__init__()\n",
        "        self.lut = nn.Embedding(vocab, d_model)\n",
        "        self.d_model = d_model\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.lut(x) * math.sqrt(self.d_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F_hw5TyCUgQ1"
      },
      "source": [
        "## Positional Encoding                                                                                                                             \n",
        "Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence.  To this end, we add \"positional encodings\" to the input embeddings at the bottoms of the encoder and decoder stacks.  The positional encodings have the same dimension $d_{\\text{model}}$ as the embeddings, so that the two can be summed.   There are many choices of positional encodings, learned and fixed [(cite)](JonasFaceNet2017). \n",
        "\n",
        "In this work, we use sine and cosine functions of different frequencies:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  \n",
        "$$                                                                                                                                                                                                                                                                                                                                                                                                                                                                            \n",
        "    PE_{(pos,2i)} = sin(pos / 10000^{2i/d_{\\text{model}}}) \\\\                                                                                                                                                                                                                                                                                                                                                                                                                                      \n",
        "    PE_{(pos,2i+1)} = cos(pos / 10000^{2i/d_{\\text{model}}})                                                                                                                                                                                                                                                                                                                                                                                                                                       \n",
        "$$                                                                                                                                                                                                                                                        \n",
        "where $pos$ is the position and $i$ is the dimension.  That is, each dimension of the positional encoding corresponds to a sinusoid.  The wavelengths form a geometric progression from $2\\pi$ to $10000 \\cdot 2\\pi$.  We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$. \n",
        "\n",
        "In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks.  For the base model, we use a rate of $P_{drop}=0.1$. \n",
        "                                                                                                                                                                                                                                                    \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MVsjhp6uUgQ1"
      },
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    \"Implement the PE function.\"\n",
        "    def __init__(self, d_model, dropout, max_len=5000):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "        self.dropout = nn.Dropout(p=dropout)\n",
        "        \n",
        "        # Compute the positional encodings once in log space.\n",
        "        pe = torch.zeros(max_len, d_model)\n",
        "        position = torch.arange(0, max_len).unsqueeze(1)\n",
        "        div_term = torch.exp(torch.arange(0, d_model, 2) *\n",
        "                             -(math.log(10000.0) / d_model))\n",
        "        pe[:, 0::2] = torch.sin(position * div_term)\n",
        "        pe[:, 1::2] = torch.cos(position * div_term)\n",
        "        pe = pe.unsqueeze(0)\n",
        "        self.register_buffer('pe', pe)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)\n",
        "        return self.dropout(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMsBRCuLUgQ3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 321
        },
        "outputId": "0ba52f87-93d6-441b-b6c6-b724981758db"
      },
      "source": [
        "# The positional encoding will add in a sine wave based on position.\n",
        "# The frequency and offset of the wave is different for each dimension.\n",
        "plt.figure(figsize=(15, 5))\n",
        "pe = PositionalEncoding(20, 0)\n",
        "y = pe.forward(Variable(torch.zeros(1, 100, 20)))\n",
        "plt.plot(np.arange(100), y[0, :, 4:8].data.numpy())\n",
        "plt.legend([\"dim %d\"%p for p in [4,5,6,7]])\n",
        "None"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3MAAAEvCAYAAADvmpjfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd0W8eZ+P0vOggSAEEC7L1BlCiR\n6tUqtixZ7t2O457ilPXGybacd3+7qbvZmmyK0+M4TlxlW7Zsy01WtXphLyDF3nsBiQ7c9w/KimxL\nlkSCBEjO5xwdisAlMCAHg3nmPncemSRJCIIgCIIgCIIgCDOLPNQNEARBEARBEARBEK6cCOYEQRAE\nQRAEQRBmIBHMCYIgCIIgCIIgzEAimBMEQRAEQRAEQZiBRDAnCIIgCIIgCIIwA4lgThAEQRAEQRAE\nYQZShroBl9Lbaw/L2gkmk47BQUeomyHMYqKPCdNB9DNhOoh+JkwH0c+EqRbKPmax6GUXul2cmZsg\npVIR6iYIs5zoY8J0EP1MmA6inwnTQfQzYaqFYx8TwZwgCIIgCIIgCMIMJII5QRAEQRAEQRCEGUgE\nc4IgCIIgCIIgCDOQCOYEQRAEQRAEQRBmIBHMCYIgCIIgCIIgzEAimBMEQRAEQRAEQZiBRDAnCIIg\nCIIgCIIwA02qaLjVai0AXgd+YrPZfvGJ+zYD/w74gV02m+0HZ2//CbAKkIBv2Gy2E5NpgyAIgiAI\ngiAIwlw04WDOarVGAj8HPrjIIT8DtgLtwH6r1foKYAFybTbbaqvVmg88BayeaBsEQRAEQRAEQRDm\nqsmcmXMD1wP/9Mk7rFZrFjBgs9laz36/C7iG8WDuNQCbzVZttVpNVqvVYLPZRibRjmn35v7D4JUT\nqYrCFGFApVKiUMpRKuUoFHJUajkROjW6KDXaCBUymSzUTQ4Jry9A94CDjv4xeoecxJt05KZGY4xU\nh7ppwidIgQB+ux3/yAi+kWECDgeSz4fk8yL5/B//6veBTI5crUGmUSNXq5Gp1ePfq9XINRoUBgNK\nYzRytfhbh4uAFMDtd+P0uc79c/lcxEvRxEhxKOSKUDcxbLi9fry+AH5/AJ9fwh8Y/+rzB/AHJKKj\nNJj0mlA3UxDmDK/fi23wDC6fiyh1FHp1FFGqKKJUOjF2XUTPkJOa5kFkQKxRS6xRS4xei0oprrAC\n8PsDuF0+3C7v+Ffn+P99vgBZVgvaCFWom3jZJhzM2Ww2H+CzWq0XujsB6D3v+x4gGzADp867vffs\nsRcN5kwmHUpl+LxRRxxjtBx1IZPkgAvo+8zj5XIZkXoNUXoNUQYtUXoNRlME5rgozHFRxFgiw+r1\nTVTPoIOapgFauuy0dNtp6bLT2T9GICB96thkSyTzM2OZnxnLgqxYEmJ1czbgvRSLRR+Ux/G7XDg7\nOnC2deDs6MDV2YlnYBDv8DDeoWG8djsEAkF5rvMpo6JQx8agNplQx8SgjjGhNpvRpaWgS01FZTAE\n/TmFccOuEd6u28ehlpOMuO24vG4kPv1+BDBq9KxKXcLatGXkmbOQy+beh73PH+BoRSdvfthIZUP/\nZx4rk0FRroXNK9JYVZCIWjXzx/DpEKzxTJgbHB4npzvLOd5WSnFXJW6f+4LH6dWRGDR6oiMMrE5d\nwsbMNagVM2ciHiwuj4+K+n5O23o4XdNNe+/Yp46RycCk12Ax6Ygz6Ug0R3LtijQSYiND0OKp5XJ6\nGex3MDQwxmC/49y/oQEHo3YXHrf/oj+r1apYeVXWRe8Pt7FsUtfMXYGLzdQvOYMfHHQEuSmTt+0h\nK62DnTT2dNA72s+QYwQpIEMekCOT5KglDRmaTGLlFpxjXhyjHnq67HS2DX/qsWQyMERHEB2rw3T2\nnzleT4wlErk8/AOcUaeXnR82sre4Hf95gZtOoyQryUBSbCTJ5kjMRi3tfWPUtg1R3z7M+8dbeP94\nCwDGKDXz02O47apMzNERoXopYcdi0dPba7+in5F8PtxtbbgaG3B3tOPt6sLT3YlvYOCCx8sjIlAY\njETExaMwGMbPqBmMyCMjkSmVyJUqUCqQKVXIlApkCiUypRICAQJeD5LHQ8B99qvHffZ7N/6RYXyD\nQ/iGB3H19uFobrng8yv0BtRJSaiTktF89DU1FYVu9n2wTJceRy8ftBzgWNcpvAEfEUotMVoTWp2W\nCOVf/2nPfnUyxuGWU7x7Zj/vntlPtMbI0rhClsYXkqZPmfULLcNjHvaXtLOvuJ2hUQ8AOSlGDDo1\nSoUMhVyGQiFHqZCP/18uo6FjhOLaXopre9FplKycH8+6RYlkJOhn/e9roiYynglzz4jHTllvJaW9\nldgGz+CXxifclohYCpNWEauNYdQ7it0zht07yqhnFLtnlGGXnQ57N5U9tWwv38Xm9A2sS1qJWjG7\ns0MG7W5OVHdT3jiArWUIn398UVajVrA418yCzBhUCjn9Iy76h130j7joG3ZxpnUIW/MgAK/sqeOq\nwiRuWpMxIzMOAgGJoQEHfV12ertH6euy0987htvlu+DxEToVBmMEmgglGq0SjVaFNmL8q0arJEKn\nIjUr5qLjVSjHsosFkTJJuvBK7eWyWq3fBfrO3wDFarVmAM/bbLbVZ7//DtDP+Jm5TpvN9puztzcA\nhTab7aK/ld5e++QaOEXO/2N6Az46RjtpsbfRMtJOeV8Vdu8olohY7sq7lQWxViRJwuP24RjzMDLk\nYuj8VYL+MVzOj3c6lVpBXKKe+GQDCUlG4pMNYXXK1+cPsOd0O28camTM5SPOFMHVS1JIsUSSZI7E\nGKm+6KQmEJBo7Rmlrm2I2rZh6lqHGB7zoFbJuXVdFtcuT0Ehn3tnBj7pUgOGJEn4BvpxNTTgaqjH\n2VCPu6UZyev92HFKkwlVfALqxETU8YmoExJQxyegiDYiV03PB13A7cY3PIxvaBBvby+ezg48He14\nOjrw9vV+6nh1UjIROblos3OIyMlFFRcnJsmX0Djcwu6WfZT2ViIhYdbGcHXaelYnLvvMCY3Foqer\ne4jawXpO9pRQ2luB0+cCwBwRyzWp67kqedWs+/3Xdwzzwak2TlT34A9IaNUK1i1MZNOSZBIvY5W6\ns3+MD8s7OVzRxfDZIDDZEsm6hYlctSgJnXa61kpnBhHMCZ/F5XOxvW4nxzpPncsgSIlKotCygEJL\nAUmRCZccg0Y8do70HeWduv14/B6iVJFck7ae9cmr0Sq10/Eypo0kSRws6+T5D+pwe8YD3tS4KAqy\nYliYGUtOihGl4uLzqEBAYmjUTW3rEK9/2Ej3oBOVUs7VS5K5flU6el34BsEjQ046WoboPRu89feM\n4vN+PLPIaIrAaIrAEK3FEP3Xr3qjFrVmcmNziIO5C74JpiSYO3t7JXAD0AYcAT7PeDD3PZvNdq3V\nal0C/Mxms637rMefCcHcJzl9Tt5qeJ99bYeQkCg0L+CO3JuIjYi56OM5HZ7x4K7PQU/nCN3tIwz2\nf/yspDEmgoRkIykZJlIzTUSE4M0mSRIldX28tPcM3YNOdBolN6/N4OqlKZ85cFzqMY9WdvP8B3WM\nOr2kx+t5eNs80hPC6zT2dLtQH/MODOCoLGessgJnrQ3/yHkZyjIZmpQUtFnZaDOz0aSmoo5PQK4N\n7w+xgNuNp7MTT2c77vZ2XE2NuBrqkTyec8co9Aa0OTlEZOegm78ATWrarAsuJqqq38Y7TXuoH24E\nIE2fwua0DRRZCi7rWpJP9jNvwEd1v41TPaWU9VbiCXhZZF7A/fl3EanSTdnrmC6d/WM89VY19R3j\n753EWB1XL0lhTUECERP4kPcHAlQ2DvBhWSfFdX34AxJmo5av37Zwzo9h5xPBnHAxjcPNPF35PH2u\nAZIiE1iduIxFlgLMnzFnuhiLRU9jRxf7Wj9kX9shnD4XOmUEm1LXsTFlLbpZMIYN2t08/XYN5Q39\nRGiU3HZVJkutcRM+q+YPBDhc3sXrhxoZGHGjUSvYsiyVrSvSwmJRamzUTXvzEO3Ng7Q3D2Efdp27\nTyYDkzkSS4IeS3wU5gQ95rhIVOqpa/esCuasVutS4H+BDMDL+K6VO4FGm822w2q1rgf+8+zhr9hs\ntv85+3P/AawHAsDXbTZb6Wc9z0wM5j7SPtrJi7Yd1A83oZKr2Jp+NZvT1qO6zFxut8tLd8d4YNfd\nMf7v/BzfuEQ9qVkxpGXFEJdomPK0zOYuOy/uqaOmZQi5TMamxcncvC4jaCs4doeHF/ec4XBFF3KZ\njC3LU7llXSYa9dy8HsVi0dPd3o+z1sZYZQWOynI8HR3n7leaTGcDt6zxr+kZyDUzL0XiQiS/H3dr\nK876Olz1Z3CeqftYqqgiOprIhYuIXFhI5Pz5yLVzLz03IAV4q+E93mneA8D8WCvXpm0kNzrrigLd\nzxrLhtzD/KnyBWqH6jFponlkwX1kR2cEo/khUVbfx292VuJ0+ynKMXPNshTmp5uCtjBgd3h493gr\nu442o1TIuX9LHusLk4Ly2DOdCOaET/IH/LzXvJddTbuRJIlr0zdyQ+a1KOUTn4if388cXicH2g+z\np/UgY14HEcoIvlDwefJj8oL1EqbVRwvfz75fi8PtY0FmDI9sm0eMITgLtl5fgP0l7bx5pJmRMQ+R\nWiU3rcng2uWp07p46vP6aW0apK1xgPbmoY+d2FBrlCSnRZOUHk18koFYSyTKab5meVYFc9NlJgdz\nMP7mO951mh31b2H3jKde3mO9bUKDiSRJ9PeM0do4QEt9P13tI+c2GNFolaRmxpCeHUNGrnnSp5E/\n+byvHmhg15FmJGBRdiz3XJ1zWalIE1HZNMAz79TQO+TCbNTy4FYrBVmxU/Jc4chvt2M/fRJPRSnD\nFZXn0iZlajURefOILCggckEBqoTEOXV2yjvQj7OulrGKchzl5fhHz77/FAp0edbx4G5REeqEhNA2\ndBp4/F7+XP0ip3vKMEfE8sWC+0nVJ0/osS41lgWkAO827eGtxveRyWTckLmFLekbZ9QmKZIk8c6x\nFl7eV49CIeeRbfNYXTB1/aSsvo/fvVHFmMvHukWJ3H9t3pzfJEUEc8L5+p0DPF31Ag3DTURrjDw0\n/17yTNmTftwL9TOXz83B9iO82fAuASTus97B6qTlk36u6TQy5uGZd22cru1Fo1Jwz9U5bChKmpI5\ngNvjZ/epVt451jI+hi1M5KFt1im9/MXj9tFc30+DrY+Whv5zaZNKlZzE1GiS06NJSTcRGxcV8v0k\nRDA3ATM9mPvIR6mX+9sPI0kSD+TfzcrEpZNqg8fto61pkJaGAVoaBhizj+/0pFDISMuOJXd+HOnZ\nsZNatQhIEs++V8ve4nbiTBE8sMXKgswrT324Um6vn52HGnn3WCsBSWJ9YRIPbrWG/E08VfxjY4wW\nn8J+4jiO6qpzO0uqk1OILChAt2AhEbm503aNW7iTAgFcTY2MlZUyVl6Gu7np3H2a1DT0K1ehX7ES\nVczsWwQYdtv5TfnTNI+0km3M5MsLHyRKPfGFlcsdy+oGG3i66nmG3MPMM+Xy4Px7MWrCP43Q4/Xz\n9Ns1HK3qJjpKzeN3LCIzcep3Ue0bcvLkaxU0d9lJi4via7cVEGea+SleEyWCOeEjJ7qKecG2A5ff\nxeK4RdxnvT1o6Y+f1c/ODDXym7KncficXJ+xmeszr50RC6Ina3p45l0bo04v1tRoHrkhn7hp2Cxu\nxOHhp9tLaey0U5gdy1duLUATxEUpp8NDY10fjbV9tDUNEvCPT/eNpgiyrGbSs2OJSzKgmOAlPFNF\nBHMTMFuCuY80jbTwZMkfcPpc3DfvTtYEaXVIkiQG+sZorO3jTFXPudPSKrWCjNxYcvLjSM2MuaI3\nhT8Q4Km3ajhS2UVqXBR/d08RhmmuEdfSbeept6pp6RllTUECj16fP2sCOr/DwVhJMfYTxxirqgT/\neAqtJiMT/fIVpG/ZiF0299IHJ8I3NMRYRTmjp08yVllx7ncZkWcdD+yWLkcRFRXiVk5e+2gnvyr9\nI4PuIVYmLOVz8+5ANYmUJLiysWzUM8afq1+ior8avSqKhxbcG9YpSwMjLn7+ajnNXXaykw18/baF\nREdNXyqy1+fn+d117CvpIEKj5Is35rM41zJtzx9ORDAnePwenqt5hRPdxWgUau7Ou5WVCUuDGlBd\nqp91j/XwZOlT9LsGWJmwlPvm3TGptM6ptuNAA28cbkKllHPnhmyuWZaCfBoDUJfHx5M7KqhsHCAn\n2cjf3rmIqElsxufz+mms68NW3kVb0yAfhSDmuCgyrWay8iyYzOFdrkoEcxMw24I5gFZ7Bz8v+S1j\nXgefs97OuuRVQW2bJEkM9I5RV93DmaqecxeLarRKcvLjmF+UhDn+sye2Xl+A3+6s5FRtL1lJBr55\ndyGR2tDspulw+fjfF0to7Bxh3aJEHt42b1oHs2CSJAnXmTMMH9iH/eTxcymUmrR09MuWE7V8BWpL\nHCAmPxPlHx3Ffuok9mNHcNbaxm9UKIhcUIBh9VqiFi8ZL68ww1T21/BUxbO4/G5uytrK1vSrg/KB\nd6X9TJIk9rYe5LX6twlIAe6x3sZVQR7DguFM2zC/2FHOyJiHdYsSeWCLNWTFcg+Vd/LMuza8vgDX\nr0rn9vVZs2ZR6nKJ8Wxu8wf8/Lr8aar6bWQY0nho/r3E6cxBf57L6WcjHju/Ln2aZnsrVlMOX1r4\nABHK8Fs4fftoM9v31RMXHcE37lo0ZZe2XIrPH+Cpt6o5WtVNkjmSb91deEXX6UmSRG+XnZqyLuqq\nevC4x3dvj08ykGW1kGU1Y5hBZalEMDcBszGYg/EV9p8V/5ZR7xh3593KhpQ1QWzdX0mSRE+nnTNV\nPZyp6cFxdgvt+CQD8xcnkTPP8qk0TLfXz5OvllPROMC8tGgev2PRhHZ5CyaHy8t/v1BCc5d9POXy\nOuuMCuj8o6OMHDnE8IH9eDrHNzFRxcVjWLMW/bIVF7zOS0x+Js870I/9xHHsx47ibmkGQGEwYFy3\nHuOGjahigz+ZmAr7Wg/xct1OlHIFD+Tfw9L4wqA99kT7WfNIK78sfYoxr4MvLnyAIktB0No0WYfK\nO3n67RokCe65JofNS0NfL6+1Z5Qnd5TTM+hk87IU7tscvmc0p4IYz+augBTgmaoXOdFdzPxYK48t\nfGjKzoZdbj9z+z38sfI5yvuqSIpM4GuFj2LSRk9JmyZiz+k2/vJeLTEGDd/+/BLMxtAGOwFJ4sUP\nzvD+yVZiDBq+dXcRSebPDi4dYx5qK7qpKe9ksG88W0wXpcZakIB1YQKm2JmZdi6CuQmYrcEcQMdo\nFz8r+S12zyh35t7MptTPrNIwaYFAgOb6AaqKO2hpGN8ZUK1RYl0Yz4KiJEzmSJxuHz99uYza1iEW\nZcfytVsLwubC/VGnl/95oZiW7lE2LU7m/i15IZ+gfRZJknDW1TK8fx+jp04g+XzIlEqilizDuH4D\nEdZ5n9l+MfkJLndHB8MH9zNy6EMCjjGQyYhcVEj0xqvRLShAFqa1DV+vf5v3mveiV0fx2MKHyTSm\nBfXxJ9PPmkda+b/i3xCQAjxe9CVyojOD2raJKKnr4+evlKHTKvnqrQXMz5j6a3wv15jLy3/85TTt\nfWPce00uW5anhrpJ00aMZ3OTJEm8XLeTfW2HyDSk8/jiL6GZwkLeV9LPAlKA7bU7OdB+GKPawFcL\nHyVVH/rdZw9XdPL7N6sx6FR8+/6lJMSER9AjSRK7jjbzyv4GIrVKnrirkOxk46eO6+4YofxkG/U1\nvQQCEnK5jIxcM/MWJZCaaUIepp+1l0sEcxMwm4M5gK6xHn5W/BuGPXZuy7mBzWkbgtC6SxsZclJd\n2kl1WSfOsfFUv7hkA2ccbmyDTpbNi+PLN82fcO24qTLq9PJfzxXT1jvKNUtTuG9zbtgFdJLPx8ix\nIwy++w6ejnYAVAkJRK/fiGH1WhT6y9s0Qkx+pkbA48F+4hjD+/biamwAQGW2YNywEeNVG8Lq2rrD\nHSd4tmY78ToLXy/8IrERpqA/x2T7WWW/jV+X/RGNQsO3lnyVpKjQ7Sba0m3nR385jSRJfPv+JWQk\nTP1GJ1eqf9jFD/98kpFRD1+9tYBl8+JC3aRpIcazuentxg94s/FdEiPj+eaSr055rcqJpI1/0HqA\nHWfeIlKl4x+X/e2E6tsFyylbD798rQKdRsk/3reE1Ljw+Tz6yMGyDv70tg2lQsbf3L6QgqxYAoEA\njbV9lJ1oo6t9vIanyaxjflESufPjQlIXeaqEYzCn+O53vzvNTbkyDofnu6Fuw4VERmpwODyXPvAS\notSRFJjnU9pbSUlvOSqZkuxpWN3WaFWkZJhYuCyF2LhIxsY8dLeNoHH5SVErWV+UjCUMtoD9JLVK\nwdJ5Fsob+ik904/T7acgMyYsAjq/Y4yh3e/T+btfYz96BL/DgX75SuI+/wCWO+8hIif3iurABauP\nCR8nUyjQpqVjXL+ByEVFSFIAV0M9jopyhvbsxm+3o05KQhER2tXQ+qEm/lDxF3TKCL6x+CuYdVMz\nwZhsP4vTmYnVmjjVU0J5XxVL4hYRoZz+QvVDo27++4ViRp1evnLLgrA6I3c+nVZJfpqJI1XdnLL1\nkp9uClqdqHAmxrO552D7EV498yYxWhNPLHkMg3rqd7+90n4mk8nIMmZgUEdR3FNO3VA9KxKWopRP\nf0ZSRUM/T+6oQKVS8K17isJyMQogPV5PWryeE7Yeimt6iHL5OPhOLdWlXYza3aRnx7B+ay6rN2WT\nkGxEFSbZXcESyrEsMlLzvQvdLoK5CQrmHzNKFcki84LxgK6vAo1CTZYxIyiPfSlyuYzoGB1v13RT\nM+QgJVaH5PDSdKaf6tJOJEkixqxDqQyfN6NGpWCZNY6yhn5Kz/Th8QaYnxG8wr9XyjvQz8DO1+n6\nw28ZKy8DCUxXX0Pil7+Ccd16VLHmCbVNTH6mnjI6mqiixURvuhql3oi7tRVHVQVDez7A29uDKj4B\npX76P1AHXIP8vPh3eAJevrLoYdIMKVP2XMHoZyn6JNRyFSW9FVQP1LIsvgiVYvo2TPJ4/fz4pRI6\n+x3csSGLDUUTq7k3XaKjNKTF6Tla2c3p2l6W5FkmtUPcTCDGs7nldE8Zf6neTpQqkieWPEbsNJ3t\nmmg/SzekMuIeobK/hj5nP0WWhdM6p6htHeKnL5chk8l44s5C8lLD5/q9C4lSyZF6xtD2O+lrtxPw\nS8wvSuSam/JZuCwFQ3REWCyyTwURzE3AXAjmACJVOhZZFlDSW05pbwV5phxitMFPqbqQNw41cbCs\nk/nZsXzl/iXkFyYil8voah+hpX6AitMduJw+TObIoBYjnwyNWsFSaxxl9X2UnOlDpZRP++Dn7min\n94Xn6f7z07jO1CGPjCL2xptI/NJjRC1eMukzO2LyM33kKjUR2TlEX30NKksc3s5OHNVVDO/dg6ul\nGVWsGVXM9ExG3H4Pvyj5PX2ufu7KuyWom51cSLD6WZYxHafPRXl/NQ3DzSyNL0IxDavbAUnit29U\nUdU0yNqCBO6+OmdGTCLiY3QYo9ScqOmhvKGflfPjg1rDKdyI8WzuqBmo43flf0atUPH44i+RFJU4\nbc89mX6WH5NH7WA9VQM2VArVtGRJATR2jvDjF0vw+SX+5vaFLMgM3/qo9mEXR/c3sPetGsYGnSjV\nSlr8fiKzTNxx4/xZlU55MSKYm4C5EswB6FQRpBtSOdZ1iuqBWlYmLEU9hRcKA1Q2DvD02zXEGrR8\n654iNCoFao2S1MwYFixOQqNV0dc1SlvTIBWn23E5vJjjo1CpQx/UadUKllotHK/uoaSun/kZMdOS\nruTp7qLn+WfpefYZPG2tqBMTMd9xN/EPP4rOOi9ohb3F5Gf6yeRytGlpGDduQpuWjre/F2d1NSMf\nHsBRU43KbEFlnrodMCVJ4unK56kdqmdd0kpuyNwy5YFJsPqZTCZjXkwu3Y4eqgZsdDl6WBw39avb\nOw42sq+4nbwUI1+9dWHYFZj9LBkJBnz+ACV1fdS1DbFqfvyMav+VEOPZ3NA80sqTpX8A4KuFj05b\nltFHJtPP5DI5C2LncaqnlLLeStINaVNSPuF8fcNO/vPZYpweH4/dUsCSvPCsQzky5OTI3gb27bLR\n02nHEB3BmmtyuPp6K6fbhqloGkSjUpCbEt5nFINBBHMTMJeCOYAYrQmFTE5pXyUdY10siy+assnQ\nwIiL/32xBH9A4pt3FxH/iR2TlEoFiSlGCpYmoTdo6e8Zo7VxkMriDrweH+Z4/afKGkw3rVpJRoKe\nQxWdVDUNsm5hAqopSgn19vXS++ILdD/z9HgQl5JK/AMPY7n3PrTpGcgUwX1eMfkJHZlMhjoxEcO6\n9ejm5eMbGcZZXcXI4Q9x1p9BnZiEMjr4H1pvN+3mQPsRcqIzeWTBfdNyZiuY/Uwmk7HQPJ+GoSaq\nBmzYvaMUxH72rq2TcaSiixc+qMMSreXv710c8hIqE5GfbqJn0El5wwCdAw6WzYubEWcWr5QYz2a/\nYbedH5/+JS6fmy8W3M+C2HnT3obJ9jOtUkNOdCbHuk5R1ldJkaWASNXU1HcLBCR+/koZnQMOHtiS\nx9qF03cG83KNDDk5vKee/W/X0ttlx2CKYO01OWy4Lg9Lgh6FQs7C7FiOVXVRXNdHbooRywyqGTcR\nIpibgLkWzMF4ulLTSAvVA7VoFGqyozOC/hw+f4CfvlxG14CD+zbnsdR68dUguVyOJUHPgiVJROrV\n9HTYaWkYpKqkA79fwhIfhSJExXgBzMYIJEmi5EwfPWd34gzmZMg7MEDfyy/R9fQfcDc3oU5IJO7+\nB4i79/NokpKmbOIlJj+hJ5PJUJnNGFatQVewEF9fH46qSoYP7MPd3oY6OQXlZe5OeiklPeW8ULuD\nWK2Jx4u+NG2biAS7nylkcgotC6jst1HZX4NFZyZ5CtKs6tqGeHJHORq1kn/43OKQ12GaKJlMxqJs\nM3WtQ5Q3DOD2+ikI4zSriRLj2ewmSRJPVT5L+2gnd+TexOrE5SFpRzD6WbTGSIwmmlM9pdgGzrAy\nYcmU1MXbdbSZD8u7WGq1cNem8EoPH7O7Obynnr27aujrHsUYo2Pt5hzWb83DEq//WFu1agU5yUYO\nlXdReqafFfnx6LQzb2Htcol2x+F3AAAgAElEQVRgbgLmYjAnk8nIj8njRNdpyvurmReTE/Riltv3\nneFETQ8r8uO4a2P2ZQ0icrmMuEQDCxYnoY1Q0dUxfk1dVcl4EWxzfFTIUoRyU43UNA9S0TiASa8J\nyi5Q/tFR+l7dTvcffoersQGVJY64z32euPsfRJM89UWIxeQnvKhMMRjWrCUiNw9Pd+d4ULdvD96+\nXjSpaSh0E1+9bbN38KuyP6KQK/nbxV/GHDF9k/mp6GcquYr8mDwOd56gesDG8vjFQQ1Oe4ec/Pfz\nJXi8Af72zkVkJ3261tFMopDLWJxnpqSuj5Iz/WQlGYg3hUdtqWAR49ns9mHHMfa2HmSeKZe7824J\nWWASrH6Wok/C6XVS0V9N91gPi+MWBfU1NXaO8Ls3qoiO0vDEXYVhc72sx+3j1OFmdr9RRU+HHVOs\njnXX5nDVljzM8VEX/R3EGLToI1SctPVS1zbEmoIEFDO8ntzFiGBuAuZiMAegUahJ06dwtPMU1QN1\nrEhcErTr507Zenn+gzoSYnT87Z2LrjgtUa6Qk5BspGBxEkqVgq62YZrrB7BVdBGhUxFjiZz2gVwu\nkzE/PYbDFZ2UnulncZ4FwwQvxJV8PoZ2v0fHr36B02ZDGROD5e7PEf/gw2jT0qfttYnJT3hSWSwY\n1q1Hm56Bu70NR1UlQ3v34Lfb0WZmIVdfWb+ze0b5afFvGPM5eLTg8+SZsqeo5Rc2Vf1Mp9IRpdJR\n3FtO51g3y+MXB+W9EwhI/PTlUroHnNy/1cqK/PggtDb01EoFuSlGDpZ1Ut08yFWLEqcsZTwUxHg2\ne/U4evld+TNoFBr+ZvEXiVCG7ix5MPuZ1ZRD/XAzVQM2ZDJZ0MZml8fHj18swe708je3LyTZEvpa\ncn5/gKqSDt7dUUlLwwBanYo112Sz4bo8zHH6yxq7MxL09I+4KG8YYGTMQ1FueF7/N1kimJuAuRrM\nAcRGxCBDRllfJd2OHpbGTf76uZ5BB/+3vRS5TMbf3VtE7CQ2DFEo5CSlRrNgcRLIZLQ3DVJv66Ol\ncYAYcyRR01w7SadVEm/ScbSqm9rWIdYtTLyiM4WSJDFWUkzHkz/DfvwYMpUK8+13kviFL6PNyEQ2\nzatMYvITvmQyGeqERIwbNqJOSMDd3IyjspzhA/uRaTTjQf9l9pdnql6kaaSFGzO3cFXyqilu+adN\nZT9L1SfTZG+leqAWvVpPuiF10o+5+2QbB0o7z2YV5AShleHDGDVeh7Kkro/hMU/YboYwEWI8m538\nAT+/LnuaftcgD+TfPe0bnnxSMPuZXCanIHYexT1llPVVkqZPJl43+ffkX96zUdU0yHUr0ti4OLRl\nVCRJoqmun3d3VFBb0Q3A0jXpXHvzfBKSjVc055TJZBRkxlDeMEBZfT/RUeqwrZU3GSKYm4C5HMwB\nZEdn0DDcRNVALVqllixj+oQfy+P18+MXS+kfcfPwtnlBuy5DqVSQkmEid0E8jlEPbY2D1JR1MTzg\nIC5RP63lDJLMkYyMeSir72fM5aMw5/J2onK1NNP1+98w+PZbBJxOojddQ9LXHkeXP3/ag7iPiMlP\n+JPJZGhSUoneuAmFTofTVsNY8WlGT59CFZ+A2hL3mT9f0lvBrsb3yTZmcH/+XSFJTZrqlPE8UzZH\nO09S1V/DkrhCIlUTTx/sGXTwyx0VRGiUfCOMUpOCKTvZSFl9P+UNA6Qn6EmImR3plmI8m53ebdrD\n8e7TLIsv4vrMa0PdnKD3M7VCTW50Nke7TmIbqGNN0gpU8onXhDxl6+XlffWkxUXx5ZsXoJCH7jq5\n3i47779eRcmxVtwuH/MXJ3Hd7QWk58RO+JIZhUJOQWYMhyvGN0RZmB1L9NlFqtlCBHMTMNeDOZlM\nRn5sHse7TlPeV0V+TB4m7cSuD/nze7WUNfSzvjCRm9cGv36KRqsie14cKenR9PeO0to4SFVxBwF/\ngLhEw7RdT5efbqL4TB9l9f2kWKJIMl/8Wibf8BA9LzxHz1+ewdfXS+TCRSR9/W8xrll7xelywSYm\nPzOHTC4nIjsHw7r1BJwOHJUV2I8cwtXSjDY9E0XUp9NoHF4nvyp9Cn/Az9cKv4BeHZpUm6nuZ1ql\nlhitiVM9pbTa21iVuGxCQWtAkvjljgq6B508fP28GX+d3MXI5TJyUowcKO2gunmQdQsTUc+CoFWM\nZ7NP80grf6p+EaPGwFcXPYJKEfrC91PRzwwaPUhQ3l+Fw+dioTl/Qo8zaHfzk5dKAPi7e4pCFuS4\nXV6O7Kln/zu12EfcZOTEct3tBcxblIhKPfmxRqdVkRoXxeGKLho7RriqMBF5GG3uMlkimJuAuR7M\nAWgUGlKikjjedZqawTpWJSy94kGzsnGAFz6oIzUuiq/fNrW1mPRGLfmFieiNWrraRs5dTxdl0GCK\n1U352QeFQo41NZpD5Z3jxXgvsLOSFAgwtPcDOn/5c1wN9aiTkkn44mPE3nRL0HYnnCwx+Zl55BoN\nUYWLiSxajKejY/x6uv17CbjdRGRlIVP+9X27ve516oYauSFrC0VxC0PW5unoZ0lRCXSNdVM1UIt6\ngsV495d08MHpdopyzNy+Piusdn4LNkOkGoVcRnFdH0OjbpZaP/sM70wgxrPZxeP38GTpHxj1jvGl\nhQ+SGJUQ6iYBU9fPMo1plPZWUNVvw2rKIUZruqKfH1+MKqejz8F9m3NZlD219esuRJIkbBXdvP1K\nBR0tw0THRHDtLfNZuiYj6MW+4006+oadVDQOEKlRkp08exbfRDA3ASKYG2eOiCUgSZT3VeH0X9nK\nkNcX4Kcvl+Jw+3jirkJijVN/LZtMJsMcr2d+USIyGbQ3DXKmupeeLjsJyQY02qldwTNEqonSqThZ\n00tT1whrChLOrQy5mpvoePJnjBw8gEytxnL3vcQ/9Ajq+PD4MPqImPzMXEpjNIa169AkJeOqr8dR\nXsbIkcOoLBbUiUnUDtazvW4nSZEJPDj/HuSy0O36NV39LM+UzfGu01T217DIsgCD+vIXTQZGXPz8\n1XJUSgXfvLtwRtaTu1LZyQYqGsbTLdPiokiMnZpaV9NFjGezyytn3qCy38am1HWsT1kT6uacM1X9\nTC6Tk6JP5mjnSRqHm1mTtALFFYzb751oZV9xB4uyY7n3mtxpX4zq6x7lvdcqqTjVDsCK9ZlcfWM+\n0VOYxp2bYuTDsk4qmwZZvSBh1pQrEMHcBIhg7q+yjRkU91ZQ3V/Lgth5RGsub6XjrSNNnLL1cs3S\nFK4qTJraRn6CQiknJcNEdn4cg31jtDUOUl3SebbMgR75FOaLp8fr6egbo6JxAINOTbpJRd/LL9H9\npz/iHxpEv3I1yY8/gW5efsiui/ssYvIzs8lkMjTJyRg3bAK5fDz18thRnC1NvOQ9zYjMw1cLH7ni\nFd5gm65+plaoSdDFcbz7NA3DzaxOXH5ZQawkSfxmZxUdfWM8sDUPa1pof1/TRS6TkZMSzcHSTqqa\nB1m3aGanW4rxbPao6rexvW4nCZHxfGHB/Sjk4dMvp7KfmbTR2D1jVA7UoJDJyb3M3S1buu38Zmcl\nUREqvnVPEVr19AU1bpePo3vr2fe2jdERN1lWM9vuWEh6TuyUzr8ANCoFep2ak7YeeoecrJw/O3Ye\nFsHcBIhg7q/kMjmJkXEc7TpFm72T1UnLL7m60zPo4NevV6HXqfj6bQtRhai4tzZCRV5BPMYYHe0t\nQzTV9dNY14c5PmrKdr2UyWRY00zsL2nHV15M3Dt/wVldhSo+nqSvfJ2YrduQa6d3x80rISY/s4NM\noUA3Lx/90mW429twVlaSVTNIpiWbwsXXhjxdcDr7WZzOwpBrmKoBG5IkYY259G6Uhyu6eOd4Cwsy\nY7jn6vAqrDvVDDo1SsV4uuWA3c2yGZxuKcaz2WHUO8aTJb/HF/Dz9cJHiQlyDdzJmup+lh2dwfGu\n01T321gct5CoS1zr7A8E+MlLZQyNevjqrQWkxU/fZRwNtl52bS+nvXkIoymCzTfPZ+madDTTeIYs\nNS4KW8sQlU0DpM6CDAMIz2BuUn9Rq9X6E2AVIAHfsNlsJ87engw8e96hWcC3ATXwA6D+7O3v22y2\nf5tMG+aaPFMOy+KLONldwpGOE6xNXnnRYyVJ4rnddfj8Ae65Jifkp7hlMhl5C+JJy4rh6L4Gqks7\n2fHnYhYsTmLlhswpSb2McA7zZfuHRLTV4ZMrsNxyG6brtiFXhXZzE2HuUScmEXjs83zw0v+wrniU\npN2ltNb/kPgHH0aTmhbq5k2bO3JvxDZYx3vNeykw53/mDr3Do25e+KAOjVrBQ9dZ51Qg95GtK9I4\nXdvLsapuluZZWDZv5gZ0wsz3St0bDHvs3Jx1Han60G6rHwoRSi13593Kb8v/xHM1r/LEksc+M8Ng\nf0kHbb2jrFuYOG3XyTnGPBx8r44GWy8KhYwVV2VQtDINRQgW82UyGQ9eZ+Vf/3CcZ9+vJT/dNCfS\n5KfbhP+yVqt1A5Brs9lWA18AfvbRfTabrd1ms2202Wwbgc1AC7Dz7N0vfnSfCOQm5racG9Ao1Lze\n8Daj3rGLHne6dnxHx/x0EyvDqLCuNkLFxm1Wbv18ESazjsriDl743Qka6/qC9hySJDG0bw9N3/kX\nIlrr6DSm8PuUmxhZtkkEckJI+AN+nre9SkW2FtU/PY5+5SpcjQ00/+C79G5/kYBnbpy10Cq1PJB/\nNwDPVm/HH/Bf8DhJkvjze7WMuXzctTEbszF0hYhDSS6X8egN+aiUcv78no0RcXZLCJHG4WaOd50m\nVZ/MtekbQ92ckCm0LKDIUkD9cCOHO45f9LhRp5cdBxrQqhXcsSFryts1vsFJFy/87jgNtl4SUgzc\n9ehylq7NCEkg95HE2EhuWJ3OoN3NjoMNIWvHbDaZv+41wGsANputGjBZrdYLVQd8GHjFZrONTuK5\nhPNEa4xcn3ktY14Hb9S/c8FjXB4fz+2uRSGXcf+WvLBc0U5MjeauR5axYn0mLpeXd16p4IM3qnG7\nvJN6XG9/H+0//m96/vIMMoWchEe/RPzj32JAbeDZ92sJSFKQXoEgXL49rQdpHe1gVcIy5qUvJvFL\nXyH5ib9DFRPL4Ltv0/z9f8XZUH/pB5oFck3ZrElaTpejhw87jl3wmJO2Xk7X9pKXGh3ywrqhlhgb\nye3rs7A7vLzwQV2omyPMQZIk8XLdGwDcmXtzSDdtCgd35d2CVqHltfpdDLtHLnjM6wcbGXP5uHlt\nJsYpLkMwOuJi1/Zy9rxZg98fYN3mHG79/GJMseFRp/KG1enEmyL44FQbTV0X/n0JEzeZd2MC0Hve\n971nb/ukLwJ/OO/7DVar9R2r1fqB1WpdPInnn9M2pawjITKeQx3HaR5p/dT9Ow81MWh3s21VWljn\nKCsUcpauSeeuR5ZhSdBTW9nNi78/QXN9/xU/liRJDO3fS9O//j8c1VVELiok4/v/hmHNWuZlxLB8\nXhwNHSMcLu+aglciCBfX4+jlrcb30KuiuD33xnO3RxYsJP17PyT62q14u7tp/dEP6Xv1ZQLeyS1o\nzAQ3Zm1Fq9DwVuN7OLzOj9036vTyl/dsqJVyHrl+3qyqUTRR1y5LJT1ez9HKbho7xWRImF4nu0to\nGmlhcdwiciZQWmS2idYYuTVnG06fi+21r3/q/rbeUfYWtxNvimDzspQpa4ckSVQWt/PC70/Q0jBA\nSoaJe76wnIXLUsJqEV+lVPDgViuSBH9624Y/EAh1k2aVYCaufqrXWK3W1UCNzWb76JPnKNBrs9ne\nOnvfM8BnFlgymXQoleGzU9L5LJbQ1iN7bMV9fG/vT3ilYSf/tvkfz62UNXeO8P6JVuJjdDx880I0\nM2AHNItFT25eHIf21rP/PRu7tpdTtCKVLTcvQBtx6WvpXD09nPnFrxguLUMRGUnWVx7HsmnDxwaz\nr95ZRNl/fcCrBxrYsiaTyMt43FALdR8TJk+SJH657/d4Az6+vvJeMpI+mfKsJ/5vvszwprXU/fQX\nDOx6E1dlGblPPE5U1tSn5kBo+pkFPbcv2MZzZa+xv+cgDxbdce6+114vx+7w8siN8ynIC58U8VD7\n8u0L+edfHebVg4386Gtrw2qydjnEeDYzuXxu3jjyDiq5ki+suBtLZHj/Haern91q3kxxfxnFveU0\nexpYllwIjI/5P32ljIAk8djti0hMmJoaa8ODTl5/oYSmM31otEpuuruQohWpYTsuWCx6Ttb1sfdU\nG8dtfdy8/vJ2Aw1H4TaWTSaY6+DjZ+KSgM5PHHMjsPujb2w2Ww1Qc/b/R6xWq8VqtSpsNtuFL5oA\nBgcdk2ji1LFY9PT22kPahjhZ4rnNUHaW7mFt8kokSeJnL5zGH5C45+ocRobC8/d3MfMKE7AkRrHn\nzRpKjrdypqaHjduspGbGXPB4SZIYPrifvpdeIOByEbmokLgHHkZmMtHX9+nM3htWpfPqgQb+8Fo5\nn9ucO9UvZ1LCoY8Jk3e6p4zKnloWmvPJ0eZe/G8al0bqv3yf3u0vMrx/L6V//21ib7yZmG03IFNO\n3QXjoexnK0zLeUe7n7dr97LMtBSLLpaeISdvfdiIJVrL6vw48R44T6JRS1GOmZIzfbx3uJEleZZQ\nN+myifFs5nqr8X36nYNsTb8amUNNryN8/47T3c/uyr6VH/X/H7898Tzx8iS0Si2na3spreujICuG\ndLNuStpTV9XNgXfr8Lh9pOfEsmFrHpF6zQXnPeHklrUZHK/s4pm3q7EmG4iZot3Mp1Iox7KLBZGT\nSbN8D7gTwGq1LgE6bDbbJ1/dcqD0o2+sVus/Wq3Wz539fwHjZ+kuGsgJl3ZuM5T68c1QDld0Uds2\nzOJcM0U507NzUrDFxkVx+0NLWLYuA8eohzdfLGP/u7V4vR/vKj77CB2/+Ck9zzwNMhnxj3yBpMef\nQGW6eB2qrSvSiIsez9tu7w3vQU+Y+fwBP2/Uv4NcJuf2nBsvuWIq12qJf+Ahkr/59ygNRvpf30HL\nj36Iu719mlo8vVQKFbdmb8Mv+Xmt/i0AXt1fjz8gcceGbJSKuX1dzoXctSkbuUzG9r1n8PlFqpIw\ntQZdQ7zfvA+DWs+WObzpycUkRsazJX0jQ+5hPmg5gNfn58U9dSjkMj43BcXB3S4vu3dWsXtnNYFA\ngA3b8th2RwGR+qm9Ji9YDDo1d23Kwe3x8+z7taFuzqwx4U9Km812GDhltVoPM76T5detVuvDVqv1\ntvMOSwR6zvv+OeDLVqt1P/AbxnfBFCYhWmPkhswtjPkcvGLbxUt7z6BWyblvc16omzYpCoWc5esy\nuP3BJcRYIqkq7uCVp0/R1z0egI1VVtD83X9hrLQEXf580r/3bxjXXnXJgVOllHPv5lwCZ8s2SGIz\nFGEKHeo4To+zj3VJK4nTXf5ZlMgFBaR/7wcY1qzF3dxEyw++w9Ce3bOyvy6JKyTTkE5JbwUHzpRz\nvLqHzES92IL/IhJjI9mwOInuQSf7imdnkC+Ej9fr38Eb8HJz1nVolTPvLMp02Jy2Eb06ig9aD/DG\ncRu9Qy6uWZoS9P0K2psHeempk9RV9RCXpOfuR5cxvzApbNMqL2bdokTyUowU1/VR3TQQ6ubMCrJw\nnxz09trDsoHhlDLiD/j50Yn/o3O0G1fVKu5cvpRtqy5eu2mm8fn8HN3XQPnJduQKGQXGIczHdyBT\nKDDfdgemLdchk1/ZusT/bS+lrL6fr95awPIwnTSGUx8TrpzL5+a7R/8Tt9/D91b/Ewb1xHLsR4tP\n0/WnpwiMjhJZtJiEhx5FoQ9evn449LPG4Rb+59QvUHtMDJes4B8/t4R56Rc/wz7XjTg8fPvXR1Aq\n5PzHY6vQTUGNzmALh34mXJmP3pep+mT+cdnjM2IHy1D1s/1th3mp9jWkngxUPQv50ZeD9770+wIc\nO9BI6fFWZDJYujaDpWvSkF/hvCecNHaO8IM/nSQz0cD/e3DpjApIQ5xmecFf1MztCcI5CrmCLUnX\ngwx02TVcs2x2beOtVCpYtzmXa69OQulzUzZgpCzjBizf/P+Iue76Kw7kAD63ORelQsaLe+pwe0Sm\nrxB8e1sPYveMsjl1/YQDOYCoxUtI/84PiJiXz1hJMU3f+xccNdVBbGnoZRrTyNbl41EPkpk/IgK5\nSzDo1NywOp1Rp5c3jzSHujnCLCRJEq/UjZcHFqUILm1t0grUAT2Ym9myNjZogdxA3xiv/OkUpcdb\nMZoiuO2BJSxflzGjAzmAzEQDS60WGjtHKA5ijeG5amb3BuGc0hIJX38CAe0QFQNVoW5OUEmSxNDe\nPUh//G9WNO0gTuOgT2nh9fe6aGm48hIGAPEmHVtXpDEw4uato2IyJASX3TPK+y37iFJFck3a+kk/\nnspkIuVb/4D59jvxj4zQ9r//Rd+rLyP5fEFobej5AwH6qjOQAnJGTRW4/aIw9qVcuyyVWIOG3Sdb\n6RtyXvoHBOEKnOwuoVGUIrhszV1jjDZkIpNL9GpLL/0DlyBJEjVlnbzy9Cn6e8eYX5TIXY8sJT7p\nQuWcZ6bbrspCJoNXDzQQCIRlEt6MIYK5WaC9b4yjlV2YHUXIZXLeaniPgDQ7Loz3j43R8eTP6Hn2\nGWRqNRmPfZHbn9jGmmuycbt9vPVSOYd2n8Hvu/LXe+PqDEx6De8eb2Fo1D0FrRfmqrebPsDt97At\nc3PQrjORyeXEXH8jqd/+Z1SxZgZ2vUnrf/073t7eS/9wmDtU3kVXt0RSoIBRn53dLftD3aSwp1Yp\nuH1DNj6/xMv750axeWF6ePweXqvfhVKu5Nbs60PdnLAXkCSee78O/0AiFk08J7tLaB/95Obul8/r\n8bHnzRr27rIhV8jYetsCNlxnRaWeul2NQyHJHMnagkQ6+sY4Uinq/06GCOZmgdcONiABd65ZyKqE\npXQ5ejjZXRLqZk2aq7GB5h98h7GSYiLm5ZP+3R+iXzKeW124PJXbH1hCdKyOspNtvPrn04xc4eq0\nRq3g5rUZeH0BdolUJSFIeh39fNh+FHNELOuSVgb98SOyskn7zvfRr1yFq6GB5u//K/bjx4L+PNPF\n7fGz42ADaqWcL664GYNaz+7mfQy5h0PdtLC3cn48GQl6jlf3UN8hfl9CcOxu2c+Qe5irU6/CHHHh\nskDCXx2t7KKxc4QV+fHcPe9GJCRer397Qo/V3zPKy0+foraym7hEPXc9sows68wpQXKlblmXiVIh\n4/UPG8XuvJMggrkZrrnLzilbL5mJBopyzFyXsRmFTMFbje/jD8zMa8EkSWJwz25a/uPf8PX3E3PT\nLaR86x8+VXLAkqDnzoeWMm9hAn3do2z/4ykarzD3eu3CRMxGLftKOhgYcQXzZQhz1JuN7+KX/Nyc\ntRWlfGpWUhURESR88TESHv0SUkCi87e/oue5v8zItMv3TrQwPOphy4o0EqIN3JS1FU/Ay876d0Ld\ntLAnl8m45+ocAF7cc2ZW7nYqTK8h9/C5UgRb0zeFujlhzx8IsPPDJpQKGXduzCY/Jo/c6Cwq+2uo\nG2y47MeRJImqkg5eeeY0QwNOCpencOv9izFER0xh60Mv1qhl4+Jk+oZd7C/pCHVzZiwRzM1wrx4Y\nHyxu35CFTCYjNsLE2qQV9Dn7OdZ1KsStu3J+p5PO3/yK3uf+giJCR/ITf4f5ltsuusmJSq1g0w3z\n2HS9Fb8/wDuvVHBkbz2BwOWt8CgVcm5am4HPHxDXzgmT1mJv42R3CWn6ZBbHLZrS55LJZBjWrCX9\n//0r6qRkhvbsHk+77J/YdaShMDLmYdexFvQ6FdtWpgGwKnEZyVGJHOs6RctIW4hbGP6saSYW55o5\n0zbM6dqZn3IrhNY7TXvwBLzclLVVlCK4DEcquukZcnJVYRJmYwQymYxbzqamvl6/67IWWDxuH7t3\nVrP/nVqUSjnb7ihgzTU5KOZInc0bV2egUSl443CT2JBuguZGT5ml6tqGKG/oZ15aNPPP2/1ta8bV\nKOVKdjXuxhuYOSv17tYWWn74XUZPHkebk0vad75P5IKCy/rZeYsSuf2BJRhNEZQca2Xnc6WM2S/v\nOrg1BQnERUdwoKSD/mFxdk6YuNfPjKfW3JJ9/bTt/qZOTCLtn/8V/arV42mXP/gOYxXl0/Lck7Xz\nUCNuj5+b12YSoRk/i/lRgXWANxvfC2XzZoy7NuWgkMvYvq9epCoJEzboGuJIx3HMEbGsTFga6uaE\nPX8gwJuHx8/K3XBeOahMYxpFlgIaR1oo6/vsDek+Sqs8U91DfLKBux5ZRkaueaqbHlYMkWq2rkhl\nZMzD+ydbQ92cGUkEczOUJEm8uv/sWbn12R+r0RGtMXJV8ioG3eMDc7iTJInhg/tp+fcf4O3uxnTd\n9aT+/T99Kq3yUszxUdz58FKyrBY624bZ/seTtDUNXvLnFPLxs3P+gMSbR5om9iKEOa96oJaawTry\nY/KYF5M7rc8t12hI+MKXiXvgISSXi/af/pi+13cgXeYZ6lDoGnCwv6SDeFMEG4qSPnbfvJhccqIz\nqeyvEWfnLkNCjI6NRcn0DDo5WCpSlYSJeb9lHz7Jz3XpV6OQK0LdnLB3/lm5GMPHz2LelHUdMmTs\nbHjnohvS1VV18+qfTzM86KRoZSq33FeE3jg3z4ZuXZFGVISKt4+1MOr0hro5M44I5maoqqZBbK1D\nLMqOJSfF+Kn7t6RvQi1XjadM+MP3jRHweuh++im6//RHZCoVSX/zDSx33o1MObFrjdQaJVtunc/a\nzTm4XT7efLGUU4eaLpnqsGpBPPExOj4s6xTbfAtXLCAFeP3MLoBzKTbTTSaTEb1hE6nf/meUMTEM\nvPE67T/9MT77SEjacymv7K/HH5C4Y0M2ygukE23L2AyM7wwqXNqNa9JRKeXsOtoszs4JV2zIPcyh\njuPEamNYkbDk/2fvvMOjuq+8/7lT1XudUaOOhLqEACFEx+Bes06cstndbDa9O3Gym/fd8m4SO8nG\nidN2s5u2sZ04zWDHBVhpprMAACAASURBVINBCEl0FSRgqELSjHqXRjOact8/RgMYAxpJ03U/z8Pz\ngDT33i+jq9/c8zvnfI+/5QQ8t8vKuUiLTKEyvYKeyV6Odr+z5cVud1C3/yL79pxFEJxulZVbli2a\nsspbEa5WcM+6bKYsNl4/KrW8zJXFe+cEMaIo8qdDTivqh6uX3vI1MapoNmVUMTo9xmFDgy/luY11\naIjOp7/JWF0t6uwcsv/PvxBVUrrg8wqCQNHqDB58fwmR0WqO1bbzxh9bmbbcvuRULpPx4Ex27pX6\n9gVrkFhcnOptpnPCSEVqKZnRmtkP8CJhOUvI/vq/EFlUjKmtlY5//WemLrvfiO8LOnqdxk1LNc7B\nsbdCF7+cpbHZtAy00TUuZZtmIzZKzaZiDYNjFhpaJZtvibnx1tWD2Bw2duVIWTl3uFNWzsU9S7aj\nlCn4y5W3sM5sqpsmp3nlt820HO8iLjGCR/+6LKTdKufC1jIt8dFq9p3oYtjNNhkJJ1IwF4Q0XRjg\nSvc4q3XJZKdF3/Z127M3ESZXs/fqwYAbwms6r6fj3/4ZS/sVYiqryPzK11AmeXZBS9PG8tiHy9Fm\nx9F+cXDGJcp029evyUslPTGCutM99A3f/nUSEjdid9h59cpe5IKc+5bu9LccAORRUWg+9VkSH34U\n28gwXc98g9G6w/6WdY3XZsyGHqjKeUeJ+I0IgsCumezcG1J2zi12rc1CLhP4y5Gr2AO4xFYisBi1\njFFnPEpCWLyUlXOD2bJyLuLD4tiUUcWwZYQaQz09hlH+8IsTdHeOslSXxKMfKiM+MdKHygMblfL6\nuChpU31uSMFckOEQRf5cexlBgIduk5VzEaWMZEtmNePWCWq66nyk8M6IosjI2/vo+u4z2CcnSH7v\n+0n9248gU6m8cr3wCBX3PV5EcUUGI4Mm/virk1y9eGu3P5lM4MENS3CIIq/UtXtFj0To0dh/mv6p\nQdalrw6omUyCTEbivfej/ewXEFQqen/x3/S9+Lzfxxf0DJk4fraPrNQoCpcm3vG1qxJWkh2dSWP/\naYwTUrZpNhJiwthQlE7f8BTHz/b5W45EkLCvowarw8bO7C1eG6cSSriTlXNxV/YWwuXhNBzRs/v5\nJkyT06zbspS7HspHpZbe65vZUJROanw4tc1GaVN9DkjBXJBx7GwvXf2TVOanoUmafUdna2Y14Ypw\n9l2tYcrmX6dGV39c3wu/QR4RQcYXv0z89h233Zn3FDKZjPXblrPtvlzsdpHX/nCak/VXb9lHtzo3\nBW1yJPVtPfTcIYsnIQHOzYm9Vw8gILAja7O/5dySyIJCsv7x/6LSaBjZ/xZdz34X+/i43/S81nAV\nEacd9Wy/+4IgcPeSbYCUnXOXe9ZlIxMEXm24ikOaOycxC2PT49QajhCvjmNd+mp/ywl43M3KuQgT\nwijq3kjS5ZUISpH7Hi+mdG2W1597ghW5TMbDG5did4i8fPiKv+UEDVIwF0TYHQ52115BLhN4YMMS\nt46JUIazPWsTkzYTBzprvazw9ryjPy5nCVlf/2cidLk+1bCyII2HP1BKVIyaY4eusPflNqzT78xS\nyASBhzYsQRSdtukSEneibfAcholuylOLSY64c5bJn6hSU8n62teJLC1j6txZrv6/f8bc4fsm84HR\nKRraekhPjKDMzT6RgsQ8MqM0nOproWdSyjbNRnJcOJX5qRgHJjmll+bOSdyZfVdrsDqs3CVl5dxi\nLlk50+Q0e15sYrJdjjlyjK6i46Rl3b41RsLJ6twUMpKjOHrG+V5LzI4UzAURR9p66R12LiIpceFu\nH7c5o4ooZST7O2qZtPo+2zR14cL1/rj1VWR++asoE/zz4JucFs2jf12OJjOWy/oB/vhrpy3wjZSu\nTCYrJYqjbb0YByb9olMi8BFFkTevvg04S2kCHVlYOJqPf4rEBx7CNjhI57f+nfFjR32q4Y2jHdgd\n4rXskTsIgsCuJdsRuf5+S9yZe9fnIACv1s/u5CuxeBmfnqDW0ECcOpZKTYW/5QQ8c8nK9feM88df\nnaTHMMbyvBQydwgMCv0c72n0kdrgRSYI3LMuC1GEN491+FtOUCAFc0GCQxR5/WgHcpnAfZWzp/Zv\nJEyhZkf2Zsx2M/s7DnlJ4a0Za6ij67tPX++P+xvv9ce5S0SkivveW0xhuZbhAWcfneHq9Xl0MkHg\nweoliEjZOYnbc3HkCpdHr1KYlIc2Kt3fctxCkMlIfOAhNJ/8NAgyuv/rJ7T/+jc+mUc3OmHhUHM3\nSbFhrF2VOqdji5JWoYlM43hPI32mAS8pDB3SEiKoyEuho2+C5ku37hGWkNjfcYhph5Ud2ZtRSlm5\nWXFl5TbOkpW7dK6fl59vZGLMwpqNS9j+QB7bcqqRC3Le6jh427lzEtepyEshKTaMwy3djE4GloFf\nICIFc0FCy8VBjAOTrF2VOmtq/1Zs1FYSrYqipqveJ71zosPBwJ/+QM///AxBqUT72S/4pD/OXeRy\nGRt2rGDLPTqs03Ze/V0LZ24YtluyPInstGiOn+2jq2/Cj0olApXrWbmtflYyd6JKy8n62tdRpqRi\n+OOfMf7khzgs3rWC3nu8E5vdwd1rs245V+5OyAQZu3K2Sdm5OXBfZQ4gZeckbs3E9CQ1hnpiVTFU\npa/xt5yA58as3D23ycqJosiJw+3sfbkNgF2P5FO+PhtBEIgPi2NNWhm9pn6a+9t8KT0okctk7Fqb\nhdXmYN+JTn/LCXikYC5IcA1R3LU2a17Hq+QqNmdswGw3U2f0bmmVw2zG+JMfMvTaqyhTZnp18gu8\nes35kluUzv3vLUalllPz+nnq376IwyEiCAIPz2TnXm1o97NKiUCjY7yLs0PnWRG3lKWxc8uUBwpq\nrZasr32d2MICJhtP0fn0N7AODXnlWhNTVt5uNBAbpWJD0fyymKUphaRFpHCs5xQDU97RGUpkpERR\ntjKZy8YxzrQPz36AxKJif+chpu3TzqycXOlvOQHPbFk567Sdt3af4fjhdqJj1DzywTKWrHxnX/CO\nrE0ICOy9ekDaYHGDDYXpREcoOXDKwNQd5gRLSMFcUHCxa5QLXaMULUskIzlq3ufZqF2HSq7iQOdh\nbA7v/GJYhwbpfPobTDaeIjw3j6yvfR1Vun+HKM+GJiuORz5UTlxiBM3Huq4NGC9cmkhmShQnzvUz\nIDXhStzA3vYDAOzMCb6s3I3Io6JY9c9fJ6Z6I5aOq3R8418xt7d7/Dr7T3ZhmbazsyILpWJ+A4ld\n2TmH6GDv1QMeVhia3LfeudEgzWySuJEJ6yQ1XXXEqKKp0qz1t5yAZ7as3MSYmZefb+TSuX7SM2J5\n9MPlJKa8+1ktNTKFkuQCOsa70A9f9IX0oEallLN9dSYmi42aJuPsByxipGAuCHBl5e6eZ1bORYQy\ngqr0NYxYRjnZ2+wJae9g6vJlOv79X7F0dhC7cRMZn/si8qj5B5++JDY+nEc+WErmkniuXhrkz79x\n1rvvWpOFQxTZK6X5JWboneyjqb+VrGgtufEr/C1nwcgUClI/9Dckvedx7KOjdD7zDcZPnvDY+acs\nNvad6CQyTMHm0oVt7JSnFpMSnsSR7hMMmaVs02zkpMVQuDSR850j6Duk90vCyYGOWiz2aXZkbUIl\nZeVm5fjZvts6WPb3jPOnX59ioHeCvOJ07n9fMeERt/cFcJllvSltSLnF1jItapWcvcc7sNqkXsPb\nIQVzAU734CSNFwZYpolhZWbcgs+3JbMamSBjX0eNR9P848eO0vXtb2IfGyP5vU+Q8sEPIyiCq6Fa\nHabknvcUUlCmYah/kj/+6iRZsWHER6upbe5m0mz1t0SJAGBvx0FERHZmbw2YHtCFIggCCTvvRvPJ\nz4Ag0D1TJu2JNeJgk4FJs40dFZmEqRa2JsgEGTtztmIX7bx19eCCtS0G7q/KAZy9cxISJusUB7vq\niFZGsUG7zt9yAh5RFHnzWCeCADvXvHNDvf3iAC8/38jkxDSVW5axaddK5LP0A2fFZJAbv4Lzwxdp\nH5OcGmcjMkzJ5hINIxPTNLT1+FtOwCIFcwHOG0edv+y71mZ75MExMTyespQijJM9nBk6v+DziaLI\n0Guv0v1fP0GQy9F+5vPEb78raB9yZTIZ1XetZMOO5ZinrPzlt81UZsZjsdo52GjwtzwJPzNkHuZY\nzylSI1IoSs73txyPE1VSSuZXvoYiPoGBP/2B3l/8D6Jt/iXZVpudN491EqaSs608wyMaK1JLSQpL\noN54jFHLmEfOGcos18aSlx1PW/swl4yj/pYj4Wfqu49htlvYmlmNSu5fZ+lgQN8xwtXeccpXJr9j\nJNTpE86WDETY+XA+JWsz3X7u2ZnjzM65yvUl7sxdFVnIZQKvH+3A4ZB6DW/FvIM5nU73PZ1O16DT\n6ep1Ol3FTd9r1+l0tTqd7uDMH+1sx0i8m+FxCw1tPaQmRFC6Islj592etRmAfQvc2Rbtdvr+95cM\n/OkPKBISyPzqPxFZWLRwgQFAYXkG97ynCLlCRv+ZPjLlMvYd75TS/Iuc/R2HcIgO7srejEwIzb2w\nsKxssv7x/6DOWcJY/WG6/uPb2CfnN2+xtqWbsclptpZlEBnmmXIuuUzO9uzN2EQ7h7rqPXLOUOe+\n9TkAvFrX7lcdEv7F7rBzsLMOlUzJBq3UK+cOrjlnrqycwyFyeN8FDu+7SFi4kgeeKGGpLvlOp3gX\nK+KWkROTRfNAGz2TvR7XHGrER6upLEijd8hE44V+f8sJSOb1NKLT6TYBK/R6fSXwd8APbvGyu/V6\n/eaZPwY3j5G4gX0nOrHZRe5em4VM5rlMV2a0xpnmH7lEx1jXvM7hME9heO5ZRg/VoM7KJutrX0et\n9czOe6CQtTSBh95fSmS0mjQ7xJlsNLR2+1uWhJ8Yn56gzniMeHUcFaml/pbjVRRxcWQ++RRRZeVM\nndfT+a1/xzo4t/luNruD1490oFTI2FGR6VF9a9PKiVRGUGs4wrRdmkE0G7lZcSzXxtJ8aRBDvzRq\nZbHS2H+aYcsIlZoKIpQR/pYT8BgHJmm+NMhybSzLtLFYp+28+adWTp8wEJ8UwSMfKiNVEzPn8wqC\ncK13bq9ULu4Wd6/NQgBeO3JVcgK9BfPdWt4GvAyg1+vPAvE6nW62O3o+xyxaTGYbB5sMxEaqqMyf\n24Bdd9ievQmAfR01cz7WNjJM59PfxNR6moiCIjK//FUUcfGelhgQJKZE8ciHyohNjCAFgRP7L2GR\nLHIXJQc7D2N1WNmevQm5bH6OjMGETK0m/WOfJG77XUx3G+n4xr9h7rjq9vFH2noZHDOzsVhDbKRn\ny7lUciXV2kombSaO9pz06LlDEUEQrmUW3joxvw08ieBGFEX2dxxCQGBzxgZ/ywkK9h53Gp/tXJPJ\n5ISF3S800n5xkIyceB7+QCkxN5RdzpXCpDzSIlM53tsomTm5QXpiJKUrk7nSPc65jhF/ywk45hvM\npQE35jr7Z752Iz/V6XSHdTrdt3Q6neDmMRIz1DQbmLLY2b46Y95W3nciN34FGVEaTvW1MDA16PZx\nlq5OOv7935yOlZs2o/30Z5GFzX2IeTARFa3msQ+VIUSqCLc6eOmXJzFNStmAxcSUbYoaQz3RyijW\nL6IBu4JMRsp7nyD58fdhHxuj8+lvMtnaMutxTtOADuQyYcEuvLdjo3Y9CkHOgc7DOESp/Hk2Slck\nkRwXRkNbD+Mmaf1abFwabadjvIui5HxSIjzXthGqjE1OU9/aQ0pcODnxEfzp16fo75kgtyiNe95T\niHqBZeMyQcZdWZtxiA72dRzykOrQ5u51zs+S1464v6m4WPCU3eDNNYD/B3gDGMKZjXvUjWNuSXx8\nBAovBDOeIDk52ivntdrs7D/ZRbhawWM7cokK94518MP5O3nu6C9o6D/K35Y/PuvrR5qaufTMd7Cb\nTGR/6ANoH3koaI1O5sMTf7+GZ5+theEpdj/fyBMfWUtSqnfuARfeusck5sbusw1M2cy8r/BBtGkJ\n/pbjcWa7z5KfeIyEHC0XvvcDDD94luWf+AdSd2y/7eubz/djGJhkU2kGumVz6ydxl2SiqcquoKb9\nCAZbB2WaQq9cJ5R4aNNyfra7lePnB3h8h87n15fWM//xS72zv/TRwp0h/3PwxP/vrVPnsNkd3FWY\nzp4XmzFPWdlydy4bti332HPPrsRqXr/6Fg3dx/hg+YPEhIX2z2WhJCdHU7jsKqcvDTBqsbM8Y+EO\n7wvREkjMN5gz8s6smga41kyk1+t/7fq7Tqd7DSic7ZjbMTxsmqdE75KcHE1//7hXzl3bbGRoZsbZ\n1ISZqQmzV66zInwl8eo49l+uY0vaJqJUkbd97Vh9HT2/+jmCIJD20Y+hXrOOgYHF1XsRE6YgMieO\nrvZhGJrif35wmF2PFKDJ8s6C4s17TMJ97A47r+kPoJKrKIsrC7mfidv32YoCtF94EsNzz3Lxhz9h\nqN1A4oMP3/LB5vf79ABUF6Z59f1an7yOmvYj/Kl1L5nKHK9dJ1QoWZpAuFrOK7WXqS5IQ6nwnYmP\ntJ75jz7TACcMLWRHZ5IgpoT0z8ET99m01c4rtZdJU8q5WH8VUYSt9+WiK0jz+HPP5oxqfn9+N39o\nfpP7l+706LlDkR3lWk5fGuDFN87ysQcL/KLBn2vZ7YLI+a7ke4HHAHQ6XRlg1Ov14zP/jtXpdG/q\ndDpXk8QmoPVOx0hcxyGKvDFTnuRp04CbkcvkbM2qxuqwcshwa1c4URQZev0v9Pz8Z8jUYWi/8CQx\naxbvbJpda7OdOxDpUVin7bzyu2YunevztywJL9I80MawZYR1aauJUM6/RyIUCF++gqyv/hPKpGSG\nXt1D7y/++12jC3qHTDRfGmSZNoal8zAHmAsZ0RpWxi/n/PBFusaNXr1WKBCuVlBdpGF0cppjZyUX\nvcXCgc7DiIhsy6peVNU086W+tYeIKSuZVhFBJnDPewrRFXinK2h9egWRiggOG45gtUuzbGcjf0kC\nWSlRHD/XR2+AJnv8wbyCOb1eXw+c1Ol09ThdKT+p0+k+rNPpHtbr9aPAa8ARnU5Xh7M37g+3OsYz\n/4XQovnCAN2DJirz04iPVnv9euvT1xCuCKemq57pmxYS0eGg/3cvMPDH36OITyDzqa8RsdL3pTmB\nxKqceDJTojjZM86Ge3TI5TL2vnyG1lPSDLpQ5UDnYQA2Z6z3s5LAQJWWTuZX/2lmdEEdhueexWG+\nXj2wb8ZgY8dq725GudiWWQ3A2521PrlesLO9PANBgLdOdEqucIuASauJI93HiVfHUZIslSLPht3h\n4EjNZbKRoQ5X8tD7S8lc4r3SepVcRZV2LRPWSU70NnntOqGCIAjcvS4bUYS3ZgxqJBbQM6fX65+6\n6UvNN3zv+8D33ThG4iZenxkSvtNLpgE3E6ZQs1FbyZtX3+ZozwmqtZUAOKxWen/x34wfO4pKo0X7\nuS+iTAi9XqG5IggCu9Zk8bNXz9DcM8aDT5Twl5daqN17AdPENBXVOdLOZwjRMdbF5dF2ViXqSI1M\n8becgEERG0vmk0/R/dMfMXm6ha7vPoP2M5/Hogzj8OluEmLUlM9x9tJ8WZWoIzUimRO9TTy47G5i\n1ZJJ8p1IigunbGUyJ/X9nO8cQZcVmk7EEk7qDEeZdli5N7NqUbjwLgS73cGfX2oh1mwHlZzH/rps\nQY6V7rJRW8m+jhoOdtWxLn219AwxC+W6ZOKj1dS19vDIxmVEhHnK/iN4Cc2pt0HKJeMoFw2jFC9L\nRJt0+/41T7MpowqFIGffzEBkh3kK4w+eZfzYUcKWryDzK1+TArkbqMhLIT5aTW1zNxFxYTz8wTJi\n4sI4WX+VmjfO43BIznqhwoEuZ1Zui2Tl/S5kajWaT36GmMoqzFcu0/H0v1N/+CwWq51tZRnIZb75\neJEJMrZkbsAu2jlkaPDJNYOdu2ZK+PdKO9shjc1h42BXHWFyNVWaxePCOx+s0zZe/8Np+q+OMIHI\nXY8W+CSQA4gPi6M4uYCuCSMXR6745JrBjEIuY2uZFsu0nbrT0uxfkIK5gGL/yZnyJC/3yt1MrDqa\nNWnlDEwN0nL5GJ3PfAvT2TYiS0rJ+MKTyCN9F1gGAwq5jB2rM7FY7RxsNBAbH87DHywjKTWKs83d\n7P3zGWxWu79lSiyQUcs4J3ubSY1IIS9hpb/lBCSCQkHq336E+J13Y+3pIeGP/0m6fZTqYo1PdaxN\nKydSEUGtoeFd5eIS72a5NpYl6dE0XRiQ+k5CmJO9zYxOj7Fe42ynkLg1U6Zpdr/QTOeVYUYQkWfH\nsSzbtxnrzRlVABzsqvPpdYOVjcUaFHIZ+0924ZDKxaVgLlAYnbBw/GwfmqRI8ny8iABsy6omdtyG\n+KNfYOm4SuzGTWg+/ilkKs8O+w0VNpVoCFfL2XeiC6vNQUSkigefKEGbHceVCwO8+rsWLGbpoTKY\nOWxowC7a2ZxRJZW93AFBEEh+z+OYN99LlHWS93e9ibzLt7vLKrmKau06Jq0mjklDxGdFEJwGWyLX\nexwlQgtRFHm7s3ZmSHiVv+UELOOjZl7+TSP9PeMQq+YiIjvXZftcx7LYHDKjtTT3tzI4JQ0Rn43o\nCBXr8lPpG5ni9CX3ZyWHKlIwFyDUNBmxO0S2lWn98uAYN2jmffvGiRyzoNy5lZQPfhhBLtXX345w\ntYJNxdp3uMKp1ArufU8Ry3KT6e4a5eXnm5gct/hZqcR8sDps1BqOEK4IZ216ub/lBAWvsZRXUqpQ\n2Kfp+o9vM9HU6NPrb8xYj1yQ87Y0RNwtVuuc5eKHW7oxSRtPIceFkUt0TRgpTSkkMVxqk7gVwwOT\n/Pk3jYwMTaErSefkqBltchSrcny/oS4IzqBbRLytu7jEO9lengHAvpPShpQUzAUANruDA00GwtUK\nKr1kf3snTOf1dH37W6jMVg6sjqKhMFLKRLjB1jItggBvn7q+kMgVMnY8uIqCMi1D/ZP8+X9PMTIk\nlTEFGyd7mxi3TrBeU4FaLmWnZ6O9Z4zzXaMIJRVoP/05EASMP36O0cOHfKYhVh3D6tQSek19nBnU\n++y6wYpCLmNbeQYWq51DzVLfSaixv8Pp7ro1c6OflQQmfd1jvPx8I5PjFtZtXkqfUoYDkZ1rMv32\n/FOeWkK0Moo64zEs9mm/aAgmslKjWZkZR9uVIboHJ/0tx69IwVwAcFLfz+jENBsK0wlT+daVZ6Kl\nCcP3voNjepq0v/8YXUVajvecwmSVApDZSIoLp2R5Ele6x7lsHLv2dUEQ2LBjOWs2LmF8zMLLv2lk\noHdxDVgPZkRR5GDnYQQENmml8iR3eOu4c0PjrtWZRBYWkfHFLyMLD6f3lz9n6M3XfaZjy8yYAtc4\nCYk7s7FYg0opY//JTuyScVPI0DPZR+vgWZbGZrMk1jfO2MFEV/sQu19owmK2sfluHXllGmpbjMRG\nqVi7KtVvupQyBRu0a5myTXGs55TfdAQTruzc/kWenZOCuQBg/8kuBGBrudan1x07egTjj54DQUD7\n6c8Su2YdG7WVTDusHOk+4VMtwcrWMudCcmN2DpwBXfn6bKrvWsGUycruFxrp7hr1h0SJOXJptJ3O\nCSPFyQUkhku27bMxOmHh2Nle0hMjyJ+ZxxS+bDmZX/lHFPHxDPz+dwz8+Y8+mWmWGa1hZdwyzg1f\nwDAhZZtmIypcSVVBOoNjFk6dH/C3HAkPcWBm5uI2KSv3Li6d6+cvvz+NwyFy10P55BWn09DWy5TF\nzuYSLQq5fx+Lq7WVyAQZB7vqpDmQblC6MomEGDV1p3swmW3+luM3pGDOz1ztGeeiYZTCZYmkxkf4\n7LojB96m57//E5lKRcbnnySyoAiASk0FSpmCGkOD1HfiBnk58aQlRHDsbC9jpneXRRSUadl2fx7W\naTuv/raZjstSo26g48rqbMmUxhG4w4FGA3aHyI7V7yxPUms0zrEmySkM/eUV+l74DaIPsj9bs2aG\niHdIQ8TdYftq54bU3uMdflYi4QlMVmdWJyEsnqLkfH/LCSjONBl5a3cbcrmM+/6qiKW6ZKdRzKku\n5DKBTSW+deG9FbHqGMpSiuiZ7EU/fNHfcgIeuUzGllItFqudw4t4TIEUzPmZfSedc362zaSKvY0o\nigy+uoe+53+NPCqazC9/lfAVK659P0oZyerUUgamBqW+EzeQCQJby7TY7CK1zcZbvmZlfiq7Hi1A\nBF7/QysXz/b5VqSE2wxODdPc30pmlIZlsTn+lhPwWG12DjQaiAy7db+vMimZzKe+hkqbweiB/fT8\n/GeINu/unuYn5pISnsSJviYmphd3H4U7pCdGUrQskUuGMS4ZpeqBYOdoz0mmHVaqteuQCdIjnovG\nIx3UvHEedZiSB95XjHbGNfx85wiG/knKdcnERan9rNKJayPxYJdULu4OG4s1KBXOcnGHY3FmM6Xf\ndD8yZprm6Jk+UuPDr5UneRNRFBl46bcMvvwnFImJZD71NdSZ766n35SxHoCaLslRyR2qCtNRq+Qz\nGYpbZx5ylidx318VIVfIeGv3Gc403Trwk/Avhwz1iIhsztwgmQC5wZEzvYybrGwq0aJW3tr9VhEb\nR+aXv0rY0mWMH2nA+NMf4bB6r7lfJsiozqjE5rDR0H3ca9cJJVxDxN+ShogHNaLodEJUyBSsT5eG\nhIPzPTlac5kjBy8TFaPmoQ+UkpIec+37+08ZgOstE4FATkwWS2KyaB04R79JquaZjegIFetWpdI/\nYqZlkVY/ScGcHznUZMRmd7C1PAOZlx8cRYeD3l/9nOG33kSVriHzK/+IKvXWzpmZ0VqWxuZwZkhP\nn6nfq7pCgXC1gvUFaQyNWWi6cPuFRJMVx4NPlBAWrqTmjfM0HpXKmgIJi32aOuMxopVRlKeW+FtO\nwCOKIm8d77qWnb4T8shIMr7wJBF5+Uw2NWL4/vdwmKe8pm1dWjlKmZJawxGpXNwN8rLj0SZFOs24\nJiUXvWBFP3yReY0YdwAAIABJREFUPtMAZSlFRKki/S3H74iiSN2+i5xq6CA2PpyH3l9KfOL1dpbh\ncQuN5/vJSI5kRUasH5W+G9eYghppiLhbuKrb9p9YnBtSUjDnJ+wOBwcaDahVcjYUpnv1WqLNRvd/\n/ZSxw7Woc5aQ+eWvoky4cyZw80x27lBXg1e1hQq3M0K5meS0aB76QAmR0WqOHLjMkZrLUpNzgHCs\n5yRTtik2aNehlPnWVTYY0XeM0NU/wercZBJiwmZ9vSwsDM1nPkdUWTlT587S+Z1nsE94x+U1QhlB\nRWoJg+YhqVzcDQRBYHOpFrvj9uXiEoHPIYPz83qjdr2flfgfh8PBgdf0nD5pICE5kofeX0J07DvX\nqZomZ7/v1rKMgKvEKE0pIlYVQ0P3ccw2s7/lBDxZqdHoMuNoax/GOLD4yuulYM5PNJ4fYHjcQlVB\nGuFq7z04OqzTGH/8HBMnjhG+YiUZX/wy8ujoWY8rTi4gVhVNQ/cJzDZp8PVsaJMiyc2K4+zV2ReS\n+MRIHv5AKbHx4TQ2dFC376IU0PkZh+jgQGcdckFOtbbS33KCgrcbneVJc+n3lSmVpP/DJ4hZvwFL\n+xU6n/kmttERr+irznD+HGsN0oaUO6wvSEOtlHOwybBo+06CmWHzCC39bWRFa8mJyfS3HL9itzt4\na/dZ9Kd7SEmP5sEnSoi4qR/OZndQ02wkXC1nXb7/xhHcDrnM+Vlktls40n3S33KCApeZ0/5ZNtVD\nESmY8xOuifXeND5xmM0Yvv89JluaicgvQPu5LyIPD3frWIVMQZV2HWa7meO90rwTd3D9LGfLzgFE\nx4bx0PtLiE+K4PRJAzVvnJceoPzI+eFL9Jr6KEspJlY9+2bHYmdkwlWeFMVy7dzKkwS5nNQP/y1x\n23YwbTTQ+cw3sQ55vs8hKzqDJTFZtA3qGZga8vj5Q41wtdPEZmjMQvMlaUxBsHHYcAQRkWrt+oDL\nMvkS67SNN/7YymV9P+mZsdz/3mLCwpXvet2p8875vlUFvp/v6y4btGtRyBTUdNVJ5eJuULIiicQY\nNfWnezCZrf6W41OkYM4PdPZNcL5zhPyceNITvVPXbjdN0vW97zB17iyRJaVoPvVZZOq5OTVt0KxD\nLsip6aqXMkduULIiifhoNXWtPUxZZnfsi4hS8+ATJSSlRnG2uZu3Xz2LQxrc6xdc2ZtNGVJWzh1q\nm43YHSJbyrTzenAUZDKS3/sECffch7W3l86nv8F0n+ddXqu1lYiIHDYc8fi5Q5Etpc7exwMzWVeJ\n4MDmsFFnPEaEIpzVqcX+luM3pi02nv/ZUTouD5G1NIF7/6oI1W0qn96eMT7ZMku/rz+JVkWxOqWE\nvqkBzg6d97ecgEcuk7G1LMM5pqBlcY0pkII5P7D/2jgC75RC2MfH6frOM5gvXSR67To0H/skMuW7\nd6ZmI1YdTWlKId2TvVwYueQFpaHFtXkn03bqW3vcOiY8QsUD7yshTRvDhTN97H35DHabFND5khHL\nKC0DZ8iI0pAT8253V4l34nCI1DQbUavkrFs1//IkQRBIeuQxEh96BNvgIJ3PfIPpbs/2a5WlFBGp\njKCh+zhW++LaqZ0PmSlRLM+IpfXyEH3DJn/LkXCTpr7TjFsnWJe+GpVc5W85fsE8ZeWV3zbTcXmI\npbpkdj1agPI2DrtdMxvqq7y4oe4pXO7itdKGlFtUF2tQKWTsP9W1qKqdpGDOx0xMWTnS1ktSbBhF\nyxI9fn7byAid3/4mlo6rxFRvJO3vPoqgmH8JwaaMKgAOSmMK3GJjsQaFXODtU11uZzPVYQrue7wI\nbXYcV84P8PqfWrFa7V5WKuGi3ngMh+hgg3bdoi5PcpfmSwMMjVlYn++Zft/E+x4g+fH3YR8ZofOZ\nb2Lp9JzLq1KuZH36Giask5zqa/HYeUOZrTPZuYONkhFKsOAyPlms/b5Tpmn2vNBEX/c4xRWZ7Hgw\nD7n89o+31/p9A2gcwe3IiskgKzqD1oGzDJmH/S0n4IkKV7Iu3zmm4PQiGlMgBXM+prbFyLTNwday\nDGQyzz44WgcHnOVKRiNx23eQ+qG/QZAt7Ee8JCaLzGgtLf1t0kLiBjGRKipyU+keNHH2qvvvl1Kl\n4J7HCslalkDn5SFe+/1pLGbvDleWALvDTp3xGGFyNRWppf6WExQccJUnlXquPCl+x05SPvjX2Ccm\n6Pz200xdvuyxc2/QrkNAkIxQ3KRcl0J0hNL5WSVtKgU8holuLo22k5ewkpSIJH/L8TmTExZ2P9/E\nYP8k+aUaHvirYmR3eO4xmW00tPaQGKOmeHlwvF+ucvF64zF/SwkKtpQ6g/SaRTTPVwrmfIhDFDlw\nyoBKIaO62LPjCKb7+uh8+ptY+/tIuPd+kh9/wiNZBkEQ2DQz70RK87vH1nLnQ+7+k3NzVFIo5ex6\npICluiSMHSP85j8bsCyyJl5f0zp4jhHLKGvSyghTzK2ndDHSN2yi9coQyzNiyUiJ8ui54zZtIe1v\nP4JjyoThP57BdN4zIwWSwhPIT9RxZayDjvHF53I2V5QKGdVFGibNNo6f83wfo4RnqZmpmnGV4y0m\nJsbM7H6+ieFBE0UVGVTftQJhlk3yutZuLFY7m0u1Ht9Q9xblqcWEK8KoNx7D7pA2WGYjOy2aJekx\nNF8aYHB0cYx1kII5H3LmyhADo2bWrEolMmzuPWy3Y7qnm65vfxPb0CCJDz9K0sOPerRcrDylmEhl\nBHXGo1LfiRssTY8hJy2aposDDIzObTCyXC5jx4OrWJmfiqFjhD0vNmOekt5zb+HK1mzQrvOzkuDg\n4MxOpyezcjcSU1lF+j98HIfViuHZ7zLZ1uqR87rKz2q7pA0pd9hcokFAMkIJdKZsUxzvOUVCWDz5\nibn+luNTxkamePn5JkaHpyitzGL91mWzPveIMxvqCrlAdbHGR0oXjlquYk1aOaPT47QMnPG3nKBg\nc6kGUYRDi2RuphTM+RDXg9DmEs89CFlmrL1tw8MkvedxEu+932PndqGa6TuZtJpo7D/t8fOHGoIg\nsK08A1GcX9+JTCZj6325lK7NYqB3gj0vNGGanPaC0sVNv2mQs0PnWRqbgzbKs5nyUMRqczqERYUr\nWa1L8dp1olevQfOJT4PDgfG5Z5k8vfBet1WJOhLDEjje24jJOrcNlsVIUlw4RcsSuWwc42rPuL/l\nSNyGI90nmXZYqdasQyYsnse5kSETLz/fxPiomYrqHNZuXOLWBvaZq8P0DJmoyE0hJiK4jGKqZzYc\nJWde91iTl0q4WsGhFiM2e+ibyi2e334/MzxuoenCAFmpUSxJ98wcK0tnJ13f/hb2sTGS3/d+Enbe\n7ZHz3ooqzVpAWkjcZU1eClHhSg41G7Ha5l4WIQgC9z1WRH6ZhsH+Sfa82IRpQhre7knqjEeB6x+S\nEnfmxLl+JqasVBeno1R496MjqrgEzac/B4KA8Uc/YKKpcUHnkwkyqrXrsDqsHO2RBvC6g8uy/UCj\nVJoaiIiiSK2hAYUgp1JT4W85PmNoYJLdzzcxOW5h3ZalrK7KcbsS6e2Z1oetQWB8cjPpkaksj1vC\nueEL9Jn6/S0n4FEr5awvSGN0Yprmi6E/N3Pen8g6ne57Op2uQafT1et0uoqbvrdFp9Md0el0dTqd\n7uc6nU6m0+k263S6fp1Od3Dmz3MLlx881LYYcYgim0vnN5fpZsxX2+n8zrewj4+T8sEPE79thwdU\n3p7kiERy41dwabQd44R7tvuLGaVCzoaidCamrJw8P7+FV5AJVO9YQdHqDIYHTLz8QhMT41JA5wms\nDhsN3ceJUkZSmlLkbzlBwYFGAwKerSy4E5H5BWg/83mQyTD+5IeMnzyxoPNVplegkCk4ZJDmZrpD\nwZJEkmLDONLWu+gG8AYD+uGL9Jr6KU0pJlrl2f7VQGWgd4LdM5UqG7Yvp3St+6NkBkfNNF0cIDs1\nmqWaGC+q9B7VGld27qiflQQHm0ucpbQHF0G5+LyCOZ1OtwlYodfrK4G/A35w00v+C3hMr9dXAdHA\nrpmv1+j1+s0zfz49X9HBhsMhcmhmLtPavPnPZXIxdfkSXd95GofJROrf/B1xmzYvXKQbuPqKXBkN\niTuzaWYhqVmAxbcgCKzftoyStZmMDk2x+/lGxhdJQ683aexrYcI6ybr01ShlC7fXD3U6+ya4aBil\nYGkiyXHhPrtuRN4qtJ/7IoJCSfd//pjxY/Nfe6JUkZSnFNNnGkA/fNGDKkMTmUxgS6mWaZuDOjfn\nZkr4Dtc4gk0Zi2McQX/POHtebMJssrJp10oKV88tu1bTbEAUYWuZZzbU/UFxSiFRykiOdJ+Q/Avc\nQJscxcqMWNrah+kN8bmZ883MbQNeBtDr9WeBeJ1Od+NWR7ler3fVZvQDnh+oFkS0XB5kaMxCpQfm\nMk1dOI/hP76Nw2Ih7SMfJbaq2kMqZ6coaRUxqmiO9pxiWlpIZiU1PoK87Hj0nSN0D07O+zyCILBu\n81LK12czNmJm9/ONjI1IfT8LwVUuvEEjlVi6g8sIw1vGJ3ciYqWOjM9/EZlaTffPfspYw/xnXrqM\nUA5JYwrcoqooHYVc4MApg5TNDCCGzSO09LeRGaUhJ8b97FSw0tc9xp4Xm7GYbWy5N5dVJXMzL7HZ\nHdQ2dxOhVrBm1cI31P2FUqagMr2CSZvkX+Aum2c+sw6F+JiC+QZzaTiDNBf9M18DQK/XjwHodLp0\n4C7gtZlvrdLpdHt0Ot1hnU7n3brAAMKV4t08xwXoZkz6c3Q9+10cVivpH/04MWt9uyMnl8mpTK9g\nyjbFqb5mn147WHEtJAuddyIIAms2LqGiOofxMQsvP9/EyFBo7zR5ixvnMiVHLOp9JreYsthoaHPO\nZSpa5p/3K3z5CrSffxJZeDg9P/8Zo3W18zpPTkzmtbmZw+YRD6sMPWIiVFTkptAzZOLcHOZmSniX\nOuMxRESqMyqDNsvkLr3GMV75bTPWaRvb7ssltzBt9oNuovniAKOT01QWpKFWyr2g0nds0K6V5mbO\ngXKd07+gtqUbqy10jVA8VV/0rtVEp9OlAK8An9Dr9YM6ne4C8C/AS8BS4IBOp1uu1+vvaNMXHx+B\nQhGYv3zJybMbmfQNmTh9eRBdVjzlBfMP5kZaTmP8wffAbif3K18ice2aeZ9rIdwfsZW9Vw9wtO8E\n9xdt8YuGYGJHfCQv7rtAQ1sP//BoMao5fpDcfI/d/VAhMTHh7P/LWV75bTMf+vh6kjw87yvU2d3x\nKgD35m1x63d4MXCn9+G1+itYpu28Z9sKUlP92GuSXEzC//sX2v7vv9D7i/8hKlxB2s675nyae3O3\n8NPjv6FxpInHCz3v/htqPLJ1JQ1tvdSf6WNjRfaCziX9vi0cu8PO0YYThCvD2JVfHdLzMTuvDPHq\n71qwWh08/P4yCtysDLj5Pmv4s3PEycNbVgT9PZhMNMVX8mjqOYNJOUp2XPCZufiau9Zm86eDF7nQ\nPc4mD5nfBNp9NN9gzsgNmThAA3S7/jFTcvk68I96vX4vgF6vNwC/m3nJJZ1O1wNogSt3utBwgNa5\nJidH098/u2Xznw9dRhRhfUGqW6+/FaazZzA89yw4HKR//FM4lubN+1wLRUBFXuJKzgzqabpyXrJ0\nd4PKglReP9LBG3WXqcx3f1fxdvfYysJUpqamqX/7Er/8YR0PPFFMfGKkJyWHLGabhUNXjhKnjiVL\nmeO336NA4k5rmSiK7Dl0CblMoHxZov/fr5hktF/4Cl3ffYZLP/5PxkdNxG3ZNqdT6CJyCVeEse/i\nYTalVCOXBeZmYaCQEKEgMyWKhtPdnL88QHz0/IIHdz8zJe5MS38bQ1MjbNRWMj48zTihObamu3OE\nv/z+NDarnR0PriI1I8at++fm+6x/ZIpGfR/LtbFEKISQuAfXJFfQ1HOGV1rf5nHdw/6WE/BU6JL4\n08GL7Dl0iVWZsQs+nz/XstsFkfMts9wLPAag0+nKAKNer7/xf/Zd4Ht6vf4N1xd0Ot37dTrdl2b+\nngakAiFtMWOzO6htMRKuVrBmnsYnk22tGH7wPWcg94lPE1Vc4mGVc2eD5Kg0JzYVu4xQPHe7F6/J\npGr7ckyT0+x+oYmhgfn35C0mTvQ2YrZbqNKskR7i3eBC1yiG/knKViYTGxUYGQB1ZiYZTz6FPDqG\nvuf/l5G3983peJVcRUVqGaPTY7QNnvOSytBBEAS2lGlxiCK1i2QAbyDjMiBzjQsKRYwdI7z6Ugt2\nm4O7HspnWe7851oeajYict2QLBQoSMwlTh3LsZ5TmG2Sw/VspMZHkJ8Tz/nOEQwh+qw0r2BOr9fX\nAyd1Ol09TifLT+p0ug/rdLqHdTpdBPAh4CM3jCH4KLAH2KTT6WqB3cDHZyuxDHaaLw4wOjHN+nnW\naU+2nsb43LMgimg+9Rmiioq9oHLu3LiQWOwh/SP0CCmuhaRr1KMLSdHqDDbsWM7UpJU9LzQx1B+a\ni5SncM5lOoJMkLFe458y5WDjoB+NT+6EWqsl48mvII+Joe+F3zC8/605HV818/OXnHndY21eKmqV\n3DlixyEZofiLIfMwbYN6smMyyYgOneDkRrrah/nLSy047CJ3PZTPUl3yvM9lszs43OI0PqlYQEAY\naMhlcqo0azDbLZzoXdgMzsXCNf+CEB1TMO+eOb1e/9RNX7rREeN2W7iLqkHh4IzpxXyMTyZPt2D8\n0Q9AENB86rNE5hd4Wt68cRmhvN6+j5O9TdKDsRtsKtHS1j5MTZOBJ7av9Nh5C8szEASB2r0X2P1i\nEw+8r5jEZKmH7la0j3XSNWGkJLmAOPXCSy1CnXHTNCf0faQnRqDLivO3nHeh1mjJfPIpOr/zNP0v\nPg+iSPx293roMqKdLoBtg3qGzMMkhMV7WW1wE65WsG5VKjVNRlqvDPnNCGex02A8jogYsi68Xe1D\nvPaHVkRRZOcj+eQsT1rQ+VzGJ9vLM+bcrx7orNes4fX2/Rw2HKFKszbkjXAWSvHyJGKjVNS19vDo\n5mVBb4RzM/MeGi5xZ/qGTbRdGWJFRizaOT5cT7Q0OQM5mQzNpz8XUIGciyrNGgQEqdTSTUpWJBET\nqaL+dA/TVrtHz11QpmXjzpWYTVb2vNDMYN+ER88fKrjcv1z29BJ3pr61B5tdZHNJ4M5lUqVryHzy\nKeSxcfT/9gWG977p9rFVmrWIiDQYj3tRYeiw0VUu3hSaO9uBjt1hp777OGFyNeWpgVGl40k6rzgD\nOUSRXY8ULDiQg+su0qFUYukiTh1LUdIqOieMtI91+ltOwKOQy9hYpGHKYuPY2V5/y/E4UjDnJWpm\negs2z7E8aaKpEeOPngOZDO2nP0fkqnxvyFsw8WFx5CfmcnW8k85x6cN9NhRyGdVF6ZgsNk7o+zx+\n/vxSDZt2rcQ8ZWXPi00M9EoB3Y2YrCZO9TWTHJ7Iyvhl/pYT8IiiSE2TEYVcRmXB3K3AfYkqLd0Z\n0MXF0f/SiwzvfWP2g4Dy1GLC5Grqu49jd3h2gyUUyUmLJis1iuaLg4xMSH06vubMkJ4RyygVaWWo\n5Sp/y/EonVeGeP2PM4HcowVkeyDz2z8yRduVIZbPY0M9WNigdfkXHPGzkuBgY7EGQYCDjaHX+ysF\nc17AVacdFa5k9RzqvSeaGjH+5IcIcjnaz3yeiLxVXlS5cDZonQ3Y0kLiHhuLNQhcL7/1NKtKNGy+\nW4d5yjYT0AW/a5enONbbiNVho0qzFpkgLXuzcaFrlJ4hE6tzk4kKV/pbzqyo0tLIfPIpFPHx9L/0\nW4befH3WY9RyFRVpZYxYRjkzpPeByuBGEAQ2lTiNUA63dM9+gIRHcVXBhJrxyTsDuUKylnqmhPea\n8Ulx6GXlXOjil5McnsjJviZM1sB0fg8kEmPDKFqayJXuMa72hNbzkfRU4wVOne9n3GSlqjANpZsz\n8t4RyH32C0Tk5nlZ5cLJT8wlXh3H8d5GzDazv+UEPMlx4eQvSeBi1yiGfu9kzvKK09lyjw6L2cae\nF5ulgA5nlqnOcBS5IGdd+mp/ywkKXKV0wfQgpEpNI+NLT6GIT2Dg979j6PXXZj3G9WAsGaG4x7pV\nqaiUMg41G3GIkhGKrxg2j9A2eI7s6EwyQ8j45N2BXIJHzhuqxic3IxNkVGnWYnXYOCYZobiFq1ru\nQIgZoUjBnBdwOcBtKnGvxPJdgZwu15vyPIbTFbACi32aE71N/pYTFLjuCW9l5wByi9LZcm+uFNDN\n0D7WiXGyh6KkVUSrQrPcxpNMTFk5fq6f1IQIVmYGnvHJnVClppLx5FMoEhIY+ONLDL1x54AuM1pD\ndnQmrQPnGDaP+Ehl8BKuVrA2L5WBUTNn2of8LWfRUN89Y3yiDZ2sXOeVIV7/w2mPB3Jw3fhkfUFa\nyBmf3Mza9HJkgox64zFEaYNlVgqXJpIYo+bomV6mLDZ/y/EYUjDnYboHJznXMUJedjxpCRGzvj5Y\nAzkX6zVrkAkyDks7225RvDyR2EgVDa09WDxshHIjuYVpN2XoFm8PXf0imMvkSRraerDZHWwq1gSs\n8cmdUKWkXM/Q/eGlWUsuq7RrnEYo3ZIRijtsLHEZoYRe30kg4hAd1BuPESZXU5YSGsYn1wI54O7H\nPBvIwfXN0lA0PrmZGFU0RUn5GCa6uTouGaHMhkwmUF2kwWK1h5QRihTMeZi5uCdNNDe9s0cuyAI5\ncDoqFSbm0Tlu4KrkqDQrCrmM6uIZI5RznjdCuZHcohtLLhenKcqUzcyJvmYSw+LRJSz3t5yARxRF\nDjUbkcsE1hcGtvHJnVClpDgzdK6SyzsEdOUpJajlKuqNx3GIDh+qDE6WpseQkRxF0wVn9kPCu5wZ\ndBqfrE4rJUxxu6lPwcO1QE4QuPuxQjKXeDaQ6xmcDHnjk5u5NjdTchd3iw1F6QiCs68yVJCCOQ9i\ntTmob+0hOkJJ2co7G59MNDdh/PFz1wO5IOiRux1VkqPSnLhuhOL9mu0bA7pXfrv4AroTvU1M26ev\nZZAl7swl4xiG/knKViYTExHcjnnuBnRhCjUVqaUMW0Y4MygZocyG0whFg90hUndaMkLxNoeNzs/V\nDSFQWfCOQO7RAo8HcgB7j14F5jffN1jJTVhBQlg8J/qaJf8CN0iICaNwaSJXusfpCJE2FOnpxoM0\nXuhnYspKVUE6Cvnt39pQCuQA8hJWkCgtJG6TFBtOwdJELhnG6PLBTLjcovRrLpeLLaCrNx5FJsgk\n4xM3ORRi5UnOksuvoIiPZ+D3v7vt2IIqrcsI5Zgv5QUtlfmpqBQyDjVJRijeZNg8QuuAy/hkbmOO\nAo1rZideDORsdgdvHesgMkzBal3oGp/cjEyQsT59DdOSf4HbuMy9QiU7JwVzHsRVYlldnH7b14Ra\nIAfOhaRSWkjmhGvX0BfZOXC6XN4Y0C2GweId4110jBsoSMwjTh3rbzkBj8nsHKaaHBdGbna8v+V4\nDFVq6kwPnXNswa0Gi2dFZ5AVraV18CwjllE/qAwuIsKUVOSm0Dcyxbmrw/6WE7I0zBifVGnX+FvK\nguhqH77mWumtQA6cxicj4xYqF4Hxyc1UalYjIEgbUm5StDyR2CgVDW29XvUv8BVSMOch+oZNnL06\nzMqMWNITI2/5momWZrpv7JELgUDOxbr0cmkhmQNFyxOJm1lIpn20kNwY0O15MfQDOte96OonkLgz\nR8/0MG1zsLFYgywIjU/uhDOg+8oNg8XfHdBVadbiEB00GCUjFHdwOfNKRijewWl8chy1XEV5Som/\n5cybrvbhd7hWeiuQgxuNT4I7izkf4tSxFCTl0jHeRed4aNnuewO5TMaGwnSmfOBf4AukYM5D1M4M\nUd14m/KkydYWun/8HMhkaD/9uZAK5ADiw+LIT3QtJNKH+2zIZTI2FM0sJHrfLSTvDOiaGfTSvDt/\nY7FPc6KnkTh1LKsSdf6WE/CIokhNk9P4ZEPh7SsLghlVqnOw+LWAbt/ed3x/dWoJKrmKOuMxyQjF\nDZZpY9AmRXLqfD9jJskIxdOcGdQzbBmhIjV4jU8MV52BnEMU2flIgcddK2+kf2SKtitDrFqSgDbp\n1hvqoc71uZnSpro7bAyhUkspmPMANw6ovFWd9mTraYw//AEIgjOQy1vlB5Xex5UBqZcWErfYUDSz\nkPh4ZzuvOJ1Nu1ZinrKy58VmhvonfXp9X3Cqtxmz3cL69ArJ+MQN2nvG6eiboGR5ErFRwfng6A6q\n1DQyv/QU8tg4+n/7AsP737r2vTBFGBWpJQxbRjg7dN6PKoMDQRDYOGOEUn+6x99yQg7XuJ+qIJ0t\nZ+wY4bWZQG7XIwVkL0v06vVqW5yfozvXZXv1OoHMqgQdsaoYjvc0YrFLGyyzkRwXTn5OPBe6RjEM\nBPdzkPSU4wFOXxpkdHKayvx312lPtrVi/OH3AdB86rMhG8gB5CfmOheS3lNMSwvJrKTEhbMqJ57z\nXaN0D/p2IVlVomHjzpWYTVb2vNjEUJAvZDdTZzyKgEClpsLfUoICV6nc7SoLQglVWhqZT34FeWws\n/S8+z8iB/de+d21nW7L4dovK/DQUchk1TQZpYLEHGbGM0jpwdqaXM8PfcuaMsWOEv/y+BYddZOfD\n+V4P5OwO54Z6uFpBVfHiK7F0IZfJWa+pwGw3c6qvxd9ygoKNMyW5tUGenZOCOQ9Q03zrByHT2TPv\nCOQi8wt8rs2XyGVyKtNXM2Uz09h32t9yggJXmr+22fcW3/mlGqrvWsHUTEA37OOA0lsYJ3q4MtZB\nXuJKEsJCx8jDW5jMVo6e7SUxJoz8HO+VQQUSqrR0Mr/0FeQxMfQ9/7+MHHwbcBqhZEZpOC0ZobhF\nVLiSitxkeoen0HeM+FtOyHCk+4TT+CQI+32Nne8M5HKWJ3n9mi2XBhmZmKYyPxX1IjM+uZnK9DUI\nCNQbpQ0pdyhdkUR0hJL61h6stuAtr5eCuQUyNGbm9OVBlqTHkJlyfUCl6ewZDM89C6KI5lOfIbKg\n0I8qfYeOJXrBAAAgAElEQVQrE1InLSRuUboimahwJXWt3djsvl9ICsq0bNixnKlJZ8nlyJDJ5xo8\njeveC4W5TL6gtsmAZdpOdXE6MlloGZ/cCVW6xmmKEh1D329+zUjNQQRBYP2MEcqR7pP+lhgUXDNC\nCfKd7UDBZXyikqtYnRpcxifdXaO89vvTOOwidz2UT84K7wdycH0z1LU5uphJDI8nN2EFl0evYpyQ\nyp9nQyGXUVWQzsSUlVPn+/0tZ95IwdwCOdzSjSi+cy6T6dxZZyDncJD+iU8TWVDkR4W+JSk8kdz4\nFVwabadnstffcgIepULG+oI0xk1WGi8M+EVDYXkGVduXY5qYZs8LTUEd0E3brRztOUWMKpqCxNAy\nGfIWbx65iiBAddHiexBSa7RkfOnLyKOi6fvfXzJaW0NFWgkqmZIGyQjFLVZkxJKeGMFJfR/jkhHK\ngtEPX2TQPER5SjFhijB/y3GbHsMof3mpBbvNwY4HV7FkpW8CueFxC82XBshJiyYrNdon1wx0XOXi\nkn+Be7jGiQWzEYoUzC0Ah0OktsWIWiVnTZ7T+MR0Xo/hB99DtNtJ//iniCoq9rNK37P+mhGKZPHt\nDtcclXw0c+5WFK3OYP3WZUxOTLPnxSZGh4MzoGvqP82UbYp16auRyxZ3uY07dPSOc6FzhOJlScRH\nh67xyZ1QazPI+NKXkUVF0fvrXzJ95ARlKcUMmIc4P3zJ3/ICHkEQ2FiswWYXaWiTNvAWSn0QjlTp\nNY7x6u9asFntbH9gFUt1yT679uEWI6K4OPp93aUwKY9oZRTHek5htVv9LSfgSU+MZGVmHGevDtMX\npM8+UjC3ANrahxgcs7A2L5UwlYKpCxcwfP8/EO12NB/7JFHFwVUi4SmKkvOJUkZytOckVofN33IC\nHk1SJCsyYmlrH6Z/ZMpvOorXZFK5ZRmT49PsfqGZMT9qmS+uEsv16cHzIORPbtfvu9hQZ2SS+cUv\nI4uIoPdXP2edQQVIO9vusr4gDblM4FCzUTJCWQDj0xM097eRHplKTkyWv+W4RV/3GK/+rvlaILcs\n13eBnEMUqW3pRq2UszYv1WfXDXQUMgXr0lczaTPR1N/qbzlBwaZrYwp871/gCaRgbgG4LOU3FmuY\nunSRrme/i2izkf4PnyCqtMzP6vyHUqZgbVo5E9ZJWvrb/C0nKLhmhNLi34WkZG0m6zYvZXLcwu4X\nmoIqoOuZ7OPiyBVy41eQHOFd97RQwGK1c6Sth8TYMAq9OP8pWFBnZpHxxS8jC49AfGk3aw0qmvtb\nmZgODWMgbxIdoaJsZTLGgUkuGcf8LSdoOdpzErtop0qzFkEI/P7V/p5xXvltC9ZpO9vuz2N53rtH\nM3mTs+3DDIyaqchLIVyt8Om1A531kn/BnCjXJROhVnD4tH/8CxaKFMzNk+FxM00XB8hIjiJ9qg/D\n976DaJ0m/aMfI7qs3N/y/I5rIZF2tt1jda7zw+hwixG7w78LSem6LNZuWsLEmIU9LzYzPmr2qx53\ncd1r64OoPMmfnDjXx5TFzvY1Wchl0kcBQFhWNhlffBJZeDhrDxlYemWSYz2SEYo7uLK7vp6bGSqI\noki98TgKQU5FWqm/5cyKM5BrZtpiY+t9eaxY5fvMmKuyYJNkfPIuUiKSWRm3jAsjl+k1Ba+xh69Q\nKeVUFqQxNjlN88VBf8uZM9In+DzZf7wTu0Nkh8aB4dnv4pieJv3vP0Z0uTTXCiAtMpVl/5+98w6P\nqzzz9n2maNR7GY2qLVtHlmQ1y5JlyZYbPRBIJ6SRkGSTXTaNAMlmv939vs0mISG97mY3mwRsCIGA\nAQO2wb0XVZex3NR7H82Mpp3vD0m2wUVtNOeMdO7r8oXQzDnvz/KZV+/zPs/7eyLSOdvfQI/N/z4Y\nvsag17IqJ4EBi4O6C31yy6GoLI2Va9IZHrSzdUs1liFlB3ROj4sjHScI1YeQF5cjtxy/YE9NGwJw\nW8nCbbJ7IwLT0kn+2mNoAgO549AQTft3qKWDU2BZWhSxEYEcPduJbVQtr58uFwYv02ntoiB+OaH6\nELnl3JKeTguvPlfDqN3FhnuyyMzxfSA3ZHVQda6bpLgQFpvCfT6+P1B+xb9A3VSfCldLLf1vQ2rG\nwZwoij8RRfGQKIoHRVFc+Z7XNomieHT89X+eyjX+hCRJbD/SSJKrH+O2P+Gx2zF+7guErVQzAtdy\n1VFJNUKZCkqbSIrL0ykuT2NowM4rm6uxDI/KLemm1PWcxuIcodS4Ar1GLbeZjLaeEc63DJK9KJqE\n6GC55SiOwEWLSf7aY3j0OlbtaubC/jfklqR4NILAmnwTDqeHI6dVI5Tp4i/GJ71dFl59rppRu4v1\nd4uIy42y6DhY14HbI7E2z+QXJalykB+XS4gumCPtJ3Cp/gWTkhwfymJTOPUXe+n1k4qkCWYUzImi\nWAksNZvNZcDngJ+/5y0/Bz4IlAO3i6KYPYVr/AZz0wCe1iY+1roDyW7D+NnPE166Sm5ZiqMwfjlB\nukAOtx/D7XHLLUfxpCaEkWYMo/ZCL/0KCZyKK9IpWp3K0ICdrZurGbEoQ9d7OdA6bnxi8ts9Ip8y\nsWGg9mW6OUGLM9B8/iFcWgHXn1/AUqWWW05GxfJEBEHtOTddrE4bJ7tqiQ2KYUnkYrnl3JS+7hG2\nPleD3eZi3V0iWXmJsuiQJIm9NW3otBrKcuUJJv0BvVZPSWIRw04LdT1n5JbjF6zNNyGB3/Wcm2lm\nbiPwMoDZbD4DRImiGA4giuJioM9sNjebzWYPsG38/Te9xt84sbeaj7XuQO8cxfjwI4SXrZZbkiIJ\n0AawMqGQQccwp3rPyi3HL6jMN+GRJPbXKcNRSRAEStYsonBVCoP9NrZuqcGqsICux9bH2f4GMiLS\nMYaojmaT4XR5OFjfQViwnkIfNfX1VzILKtl1ewpuDbT99tdYqqvklqRoosIM5GfE0tgxTGPHsNxy\n/IbjnVU4PU7KE0vQCMo8/dLfM8LWLdXYrU7W3pHJsnx5AjmAhpZBOvqsrBDjCA3Sy6bDH5hwdlaN\nUKZGWY6R+8rTycvwLxO1mc4aRuDasLV7/Hs3eq0LSJzkGr/BbrWTvf95gjwO4j/zMOGry+WWpGhW\nj5daHlBrtqdEaXYCAXoN+2ra8HiUcU5HEARKKxeTX5LCQK+Vrc/VYB1RTnPgQ+1jZbwTZb0qt6aq\noRuLzUl5biI6rTIXjkpBI2hYWrSOl9dFIGkE2n/7Kyy1NXLLUjRX+mbWqtm5qXKw7SgaQUNpojLN\n0/p7rWzdUoPN6mTN7UvJKZQ3o69WFkwdU6iRReFpnO1roNcm/3l8paPXabh/zWK/O37grcMltypY\nvtlrUypyjooKRqdTTvNfpyOQk8lLCC9dwdIH7pFbjuKJixPJOJ/Gqb6zaEJcxARHyS1J8VQWJrPj\naBO157spyPSt1fOtuO8j+QQG6jiy9xLbXqjjU18qIyRU3kbTbo+bI4eOE6wP4rbs1Rh0AbLq8QcO\nv1gHwH3rlhAXFwZw5b8q13NPaCWvX9rOoXviqdh2mfZf/4Jl//QkUUXKdxyUgw3RITyz4xxHT3fy\n5Q8XEBhwdZmhPmfXc7GvkWZLGyuT8lmSnCS3nOvo7bbw2vNjG3h33p9LyZpFsuqx2JwcN3eTGBvC\nmhUpNzwvpz5n7+ZOcS2/OfZnaodq+UjqvXLLmRco7RmbaTDXxruzaiag/SavJY1/z3GLa25KvwK7\nsa/+58eIiwuju1stI5kKJfEruNDfyOv1u7lr0Sa55SielWIcO4428dbhRpKiguSW8y4Ky1Kxjjio\nO9HKH355gPsezCcoWL4Aqq7nNP22QdYmlTHUPwooqwRUaXQP2Khu6CYzOYJADXR3D6tz2aRoyYnJ\n4ph0mnWf+wTO3/+ZM9/9PqZHv0pITq7c4hTJ6twEXjvYyJv7L1K+fKwcT33ObsxrZ3cBUBxTpLif\nz2C/jVc2VzEy7GD1xgwWZcXKrvGdky04nG5W5yTQ02O57nX1ObuepcEigVoDO88foDJ+rWJLef0F\nOZ+xmwWRM/0X3Q58CEAUxSKgzWw2DwOYzebLQLgoiumiKOqA942//6bXqMxvihMKCNAGcLD9GB7J\n/5ox+poMUzhJsSEcrm9nyKqcckYYK7ks37SEnCITfd0jvPZcLXabUzY9B670llNLLKfCvvHStzVq\nedK0mHAYPBLai+kfvgJA2y9/hvXMaTllKZY1ecpy5lUqo24HxzuriTREkB0jyi3nXQwN2Ni6pZqR\nYQdl6zPIX5kit6Qx45PqNrQagYrl8p3Z8zcM2gCKEwoYGB3kdK9Zbjkqc8CMgjmz2XwQOCGK4kHG\nXCn/XhTFz4ii+MD4W74EbAH2Ac+bzeZzN7pm9vJV/IFAXSDF8fn02fs529cgtxzFIwgCa/NNuNwS\nB+s65JZzHYIgsOa2pWQXJNLTNdFvyPcB3cDoIKd6z5IalkRKmBqcTIbb42F/bTtBBh3FWcop3/UH\nsqNFIgLCOdZZhT4rcyygkyRaf/FTrGdVl7j3EhcZRHZ6FA0tg7T3jsgtR7Gc7KzB7h6lLHGlorIl\nQwM2tm6uxjI0yqp1iykolT+QA7jcMUxTl4X8JbFEyFzi729cbRWl+hfcCsnjYfjEMdyW67O+SmbG\nZ+bMZvOT7/lWzTWv7QXKpnCNygJhtamUg+3HONB2VHE7kEqkLNfIX/dcYG9NG3eU3PhcgJwIgsDa\nOzKRJDhT086rz9Vy78fyMAT6zlnscPtxPJJHzcpNkboLfQxYHKwvSsKgV845ZH9Aq9FSZlrJm5ff\npqqrjtLcFSR++VHaf/0LWn/+E5K++g2CM9V57VrW5ps4fbmfvTVtfHTDUrnlKJKD7UcREChLVE5L\nFcuQna1bahgeGqVk7SIKV6XKLekK+64Yn6hZuemSGp5MSqiJut4zDI4OEWHwSzP5OUXyeOj4w+8Z\nPnSQuAcfImrjbXJLmjLK2QpSmdekh6dgCjFS23OKIYdaXTsZoUF6Vi830dFnpaFlUG45N0QQBCrv\nzCRruZHujmFee76WUbtvGpN6JA8H244RoNFTnFDgkzH9nYmSt0q1xHJGTCy4Jyy+Q/PySfy7v0dy\nu2n92Y+xNZyTU57iKFw6Zht/oK4Dl1str38vbZYOLg42khW9lJggZRiDWYbsvLK5muFBOysr0lmx\nOk1uSVewO1wcPt1JVJiB3EX+ZRuvFFabSvFIHg63H5dbiuKQPB46//d/GD50kMDFiwlfXSG3pGmh\nBnMqPkEQBMrHJ5Ij7Wrz3alwx6qxX6R7qpV77kQQBCrvEsnMTaCrfZjX/1KLY3TuA7pz/RfotfdR\nlJBPkC5wzsfzd/qHR6m50EO6MYzUBGW5cPkLsUHRZEUt5cLgZTpGugAILSjE9Hd/j+Ry0fLTH2M7\nr5aRT6DXaVida8Ric1LV0CO3HMVxsH3ivG+JzErGsAyP8srmaoYG7KwoT6O4Il1uSe/i6Jku7A43\na/IS0WiUVaniL6w0FqDX6FX/gvcgeTx0/ukPDB3cjyF9EUlffQxtkLLM5yZDDeZUfEaJsRC9RsfB\ntqNIkjJ6qCmZ3IwYEqKCOG7uYkSGM2lTRaMRWH93Fktz4ulsG+L1F+Y+oJvIjqi95abG/rp2JEnt\nyzRbJhbe1547CS0sIvELX0JyOmj96dPYLpyXS57iuNJzTjVCeRdOt5Oj7ScJ1YeQF5sttxxGhkfZ\nOh7IFa1OZaXCAjkYe4YErprrqEyfIF0QRfF59Nh6aei/KLccRSB5PHQ980eG9u/DkJZO8tcfQxvs\nXz3mQA3mVHxIsD6Ygrg8umw9NAyoE8lkCILA2gITTpeHQ/XKM0K5Fo1GYMM9WSxZFk9HyxDbXqjD\n6ZibgG7YYaGm+xTGkAQWhSvnPIdS8UgS+2raCNBrKM1OkFuOX5MXl0OoPoQjHSdweq4+32Erikn8\nwt/hcYwHdBfV+Q3AFBvCkuQITl/qo7NPeW2G5KKmu54Rl5VVicXoNN5q9zszRiyjbN1SzWC/jcJV\nqZSsWaS4M9rNXRYutg2xPCOGmAi1EmM2TGyATmyILmQkSaLr2T8zuHcPhtQ0kr/2GNrgELllzQg1\nmFPxKRMW3+pEMjXKcxPRagT21rQpPpup0WjYeG8WGVlxtLcM8voLdTgdbq+Pc7TjJG7JTbmpRHGL\nDiVy5nI/PYN2SrISCDLIu3D0d/QaHaXGFVicI9R2n3rXa2HFJSQ+8kU8djutP/kh9suXZFKpLCrz\nTUjAjiONcktRDPvHf//JXWJpHXGwdUsNA302CkpTKK1UXiAHsLd6wvhEzcrNlsURaRiD46nprsfi\nXLhOs5Ik0bX5zwzu2YUhJYXkr38TbWio3LJmjBrMqfiUJZGLSAiOo7q7nhGnulM7GeEhARQujaWl\ne4SL7UNyy5kUjUbDpvuWsViMo715kG1/9W5AJ0kSB9qOohO0lBiLvHbf+cxEidvaAnUh5A1utSEV\nVlKK8ZEv4LHbafnxD7FfvuxjdcqjOCueIIOOHUebcHvUczqd1m4aBi6yNHIxCcFxsukYC+SqGei1\nkr8ymVXrFisykHM43Rw61UFESAB5GarxyWwRBIHVphJckpujHSflliMLkiTRveVZBne9Q0BSMslf\nf9yvAzlQgzkVH3NlIvG4FuxEMl0mFuF7FWyEci0TAd2izFjamgbGAjqndwK6C4OX6bR2kR+XS6je\nP8shfMmQ1cHJc92YYkPIMKlW1N4gISSepZGLMfefp9vae93r4aVlGD/7eTw221hA13jZ9yIVhEGv\npSwngb4hO3UX+uSWIzsT5y0rZDzva7M6ePW5Gvp7rCwvTqJsQ4YiAzmA4+YurKMuKvIS0WnVJas3\nKDWuQCtoObAA/QskSaL7+S0MvLNzLJB77HG0Yf5vCqZ+MlR8ztWJ5MiCm0hmQnZ6NLERgRw904XN\nB06R3kCr1XDb+7OvBHRv/LUOlxcCuomFkGp8MjUO1nXg9kiszTcpdrHmj1xpwNt+4wa84WWrMX72\nETw2Ky1P/xB708IuMZwoj9tT3SqzEnlxeVwcbj9OiD6Y/LhcWTTYrA5e3VJDX/cIy1ckUb5xiaLn\nhgk35zVqiaXXCA0IIT8uh46RTi4NNcktx2dIkkT3X55jYOd2Akwmkr/xOLqw+bHJqQZzKj4nLCCU\n/Lgc2hfYRDJTNILAmrxERp1ujpzplFvOlJkI6NKXxtDaOMAbL9bPKqCzuWyc7KolNjCapVGLvah0\nfiJJEntq2tBpxyziVbxHQVwuIbpgDrUdw+W58QZLeFk5CZ/53HhA9xSjzQt3rktNCENMjaL2Yi99\nQ3a55chGbc9pLM4RSo0r0Gv1Ph9/IpDr7R4hp8hE+SZlB3JtPSM0tAySkx5FfKR/WcUrnYVmhCJJ\nEj1/eY6BHW8RkGgi+RtPoAufH4EcqMGcikwstIlktlTkmdAIgt+UWk6g1Wq4/f4c0pbE0HK5nzdf\nqsflmllAd7SjCqfHyWpTCRpBnbom41zzAJ19Voqzxpo3q3gPvVZPSWIRw04LdT1nbvq+iPIKEj7z\nWTxWK81PP8Voc7MPVSqL21elIUmwr7ZdbimycaB1oqWK741P7DYnrz43HsgVmlhz21JFB3Jw7Xnf\nJJmVzD8yozKICYzmZGcNNtf83mCRJImevz5P/463CDAmkvzY4+giIuSW5VXUFZGKLCykicQbRIUZ\nyMuI4XLHMI0dw3LLmRZarYY77s8hLSOa5kv9vPnSqWkHdJIksb/1MBpBw6rElXOkdH6xezzwr1TL\nk+aEqW5IRZSvIeHTD+OxWMYydK0tvpCnONYUJBEYoGVvTRsez8Irr++x9XK2v4GMiHSMIb5tEWK3\nOccycl0jZBeaWHO78gM5p8vDwfoOwoL1FC6NlVvOvEMjaFhtKsHhcXKso0puOXPGWCD3F/rfehO9\n0UjyY0+gi4iUW5bXUYM5FVm4diI53jl/JxJvcsUIxQ8b8Gp1Gu54IJfUjGiaL/bx1jQDustDTbSN\ndJAXm0OEwf8PK881w1YHJ8xdJMYEk5ky/35xKYHEkAQWR6Rztq+BHtutjT0iKtaS8KmHcVuGafnR\nDxZkQBdk0LEqx0j/8Ch1F683jpnvHJDpvO9ERq6ny0J2QSJr/SCQAzh5rhuLzUn5ctX4ZK4oSyxG\nI2jmrX+BJEn0vPgC/W+9gd5oJOWxJ9FFzs/fh+onREU2rk4kNzYRUHk3yxdHExVm4PDpDkbnoH/b\nXDMW0OWQujiapmkGdBN9mSqSVOOTqXCwvgOXW6JSNT6ZUypMpUhIHJrCHBaxtpL4T34G9/DCDegq\nrxih+N+G1Gxwe9wcbj9OkC6Iwvg8n417JZDrHA/k7sj0m/lgwixH7S03d0QYwlkem02LpY3G4flV\nAi5JEj0v/ZX+N7ehT5jfgRyowZyKjEQYwsmNWUbzcCtNwwtvYTNdtBoNFcsTsY26OXa2S245M0Kn\n03LHB6YX0FmdNk501hAbGI0YtcRHSv0XSZLYUz1ufLI8UW4585rC+DyCdEEcaj+G2zP5xkRk5boF\nHdClGcNIN4ZRc6GH/uFRueX4jLreMww5hikxFhHgI+OTawO5Zfn+Fch19ls52zSAmBKJMTpYbjnz\nmokWGftb549/gSRJ9P7tRfrfeB19QgIp33xiXgdyoAZzKjJztQGvmp2bCmvyExHwz1LLCaYb0B3r\nHDM+KTeVqsYnU+Bc8wAdqvGJTwjQ6ikxFjHoGKa+9+yUrlnoAV1lgWncCMV/57DpMnGu0lfGJ6P2\ndwdylXf6TyAHV3+/VRaoWbm5Jit6KTGB0ZzorMbmssktZ9ZIkkTvyy/Rt+019PEJJD/2JLrIKLll\nzTnqykhFVrJjRCINERzvqGLU7ZBbjuKJjQgiZ3E051sHae22yC1nxkw1oLvW+KQ0sVgGpf7HHtX4\nxKdc3ZCa+s72Qg7oSpYlYAjQsm+BGKH02vo503uOReGpJIXOfab82kAuK8/od4Gcy+3hQG07IYE6\nVohxcsuZ92gEDeXzxAhlIiPX9/qr6OPiSX7sCfRR8z+QAzWYU5EZjaBhdeJK7O5RTnRWyy3HL7hy\n7sSPs3MwtYDu0rjxSb5qfDIlLDYnx81dGKNV4xNfkRSayKLwVE73mumz90/5uncFdD9cOAFdkEHH\nquwEeodGqb90a+OY+cCh9qNISD4xPrHbnGzdUkN3x1ggt+4u0a8COYDqhh6GrE7Kco3odVq55SwI\nViWuRCNo2O/HRigTZ+T6tr2GPiGB5Me/hT46Wm5ZPkMN5lRkZ7WpBAFhXtVszyX5S2IJDwngUH0H\nzhn2bFMKkwV0V/oyqcYnU+JAXfuY8UmBanziS8rHjVAOth2b1nVXAjrLwgroJsrnJkwu5ituj5tD\n7ccJ1AZSlJA/p2O994ycPwZycHWTUq0s8B0RhjDyYnNotbRzeahJbjnT5opr5fgZueTHnlwwGbkJ\n1GBORXaiAiPJjV1G43AzTUMLYzEzG3TaMSOUEbvLb41QruW9Ad1EHzqr08aJLtX4ZKpcNT4RKFeN\nT3xKUUI+gdrAKRuhXMtCDOjSjeGkJYRRc753XhuhnO4zMzA6yEpjIQZtwJyNM9FHzl/PyE3QM2Dj\n9KU+liRFkBQXKrecBcWEU/T+aZSLK4ErgdyEa+U3F14gB2owp6IQ1iStAmB/22GZlfgHlQUmBGB3\nlX+XWk5wbUDXfLGPN1+s53DriTHjkyTV+GQqXDE+EeNV4xMfY9AGsNJYyMDoIKf7zNO+/rqArnl+\n2YTfiMoCEx5JYn9du9xS5gxfGJ/YrA62bqm+0kfOXwM5gL217UioxidyIEYtITYwmhOdNVid/mGE\nMtYQ/PmxQM5oHHetXHiBHKjBnIpCWBadSUxgFMfmiaPSXBMXGUTu4hjOtw7S3OW/RijXotNpufMD\nuaRlRNN8qZ/6t/rQenSsUo1PpsQV4xN1ISQLE2eipmOEci2RleuI/9RYQNf89A+wNzV6U57iKM1O\nwKDXsre6DY+fntO5Ff32Aep7zpIalkxKWNKcjGGzOnh1Sw29XSPkFJr8qv3Ae3G5PeyraSPIoKM4\nK15uOQsOjaChPKkUp8fJ0c6TcsuZFEmS6HnhefrfepMAY+J4H7mFGciBGsypKIQxR6VSHG6H3zsq\n+Yp1hWOL9t1V8+fcyVhj8Vzi0oLR94eTfWktQah9hiZDNT6Rn5QwE2lhKdT3nKXfPjCje0SuXUfC\nZz6LZ2SElh89hb3xsndFKoggg47S7Hh6h+ycnodGKIfaj40bn8xNVm4sI1dDb/cIOUUm1ty+1G8D\nOYCqhh4GRxyU5xox6FXjEzlYlViMRtBwoFXZRiiSJNHzl+fo3z4WyCUvgD5ykzGjYE4URb0ois+K\norhfFMU9oiguvsF7PiqK4lFRFA+Lovjd8e99RhTFZlEUd4//+afZ/gVU5g9lpjFHpX2thxU9kSiF\nvIwYosIMHDzVgW3UJbccr6HVaRhZfomhqA6k3kC2/bUOp8O/jV7mmoOq8YkiKDeVICFxqH16RijX\nElGxloTPfA6PzUrL009hv3zJiwqVxdr8sYzVRFZ5vuD2uDnQdpRArYHihAKv3986MhbI9XWPkFuU\nxJrb/DuQg6ubkusK5yaLqTI54QFh5Mfl0jbSwaUhZVYGSJJE9/Ob6d/xFgGJJpK/+SS6iIUdyMHM\nM3MfBwbMZnMF8F3ge9e+KIpiMPADYCNQBmwSRTF7/OXnzWbzuvE/353h+CrzkPCAMAoUPpEoCa1G\nQ2WBiVGHmyNnOuWW4zWsTitVPTVYc5tYlBlLW9MAr/+lFqdj/gSs3kSSJPbUjBmfrM41yi1nQbMi\noYBArYEDbUenbYRyLRHlFRg/+3k8NhstTz+F7eIFL6pUDosSw0iJD6X6fA8DlvljhFLXe4aB0UFK\njEUE6gK9em+rZZStW6rp6x5h+YokKm5b4veBXHvvCGca+8lKjcQUGyK3nAVNxXi5uBLdxSWPh67N\nfzONTh8AACAASURBVGZg5w4CTEkkP/YEuogIuWUpgpkGcxuBv41/vRMov/ZFs9lsBZabzeZhs9ks\nAb1AzIxVqiwYKkxjRij7WlUjlKmwJs+ERhDYfbJ13mQzj3ZU4fS4KE9eyW3vzyYjK472lkFe+0st\njnmUgfQW55oHaO+1skKMJyx47hzzVCYnUGegxFjEwOgg9b1nZnWv8LLVGB/5Ih67ndaf/AjbhfNe\nUqkcBEGgssCE2yOxv3b+GKHsazkEwJqkMq/e1zI8ysubq+nvsZJXnEz5Jv8P5OCqkZealZOfzKgM\n4oJiONlVg9VplVvOFSSPh65n/sjgrncISE4ZK61UA7krzDSYMwLdAGaz2QNIoii+axVhNpuHAURR\nXA6kAxOr80pRFN8URfFtURQLZzi+yjwlMyqD+OBYTnbVYnGOyC1H8USFGShcGktTl4WL7UNyy5k1\nkiSxv+0wWkFLWeJKtFoNm+5bxpJl8XS0DKkB3Q2Y6Mu0TjU+UQQTC/i94wv62RBeuorEL3wJz+go\nLT/+EbaGhlnfU2msyh47I7WnuhWPx/83pLqs3ZztbyAjIh1TqPcy5ZYhO688W8Vgn42C0hRWb8yY\nF4HcqNPNgbp2wkMCKMqMk1vOgmfCv8DpcXGkQxlGKJLHQ+cf/8Dg3j0YUtNIeewJdGHhcstSFLrJ\n3iCK4iPAI+/59ns7+N5wRhFFcSmwGfi42Wx2iqJ4GOg2m82vi6JYBvwJWH6r8aOigtHplHkYNi4u\nTG4J85I7Myv5U/WLnBo+xfvEjXLLkZWpPGP3r1/CiXPdHD7Txar8ZB+omjvMPRdoH+lkVUoRi5Ou\n9kr72MMreXlLNfVVrbzxYj0Pfb6UIDULxdCIgxPmbpLiQigvSpnx4k6dy7xHXFwYyy4t4Ux3A65A\nG4lhs3Pmi7t7I+GRwZz70U9o/dnTZP+fbxORk+Mltb7lZs/Z+uIU3jx0mcYeKyU5/l0q/Eb1dgDu\nWbbBa5+rgT4rrz5Xy9CAnTWblrLuTv9sCH4jdh5twjrq4sMVS0k0eifTos5ns+OesEpevfQWhzuP\n8eHCO2V91iS3m4af/4qhA/sIXbqEnH/9Z3Sh8vcgVNozNmkwZzabfw/8/trviaL4v4xl52pEUdQD\ngtlsdrznPcnAy8AnzWZz9fi9zgJnx78+JIpinCiKWrPZfNPDBf39yknzXktcXBjd3cNyy5iX5ITl\notO8wpvndlMStXLe/NKaLlN9xhIjA4mPCmJvVSv3l6cTEui/PcZeO70LgJUxK677u5fftgSn04W5\nvpM//PIA934sn8AF3k/tjSONOF0e1ixPpKdnZi0q1LnM+6yKL+FM93leqdvJB5feO/sbLs0l8e++\nTNtvf82pf/13kh79KsHLsie/TkHc6jkrWxbPm4cu87ddDSyK998zUw63k10XDhKqD2Fx4BKvfK4G\n+21s3VKNZWiUlRXp5BYnzfizrkS27r2AAKzMjPXKz0udz7yBQEFsLie6ajh8vo4lkYtkUSG53XT8\n938yfPQIgYszSHj06/TbJLDJ++8r5zN2syBypmWW24EPj399L7DrBu/5b+BLZrP5Sp5WFMXHRVF8\ncPzrXMaydKpNncq7CNWHUBSfR5e1h4aB+Xnw35toBIF1BUk4XR4O1HXILWfGWJwjnOyqIS4ohsyo\njOte12gE1t+TxbL8RHo6LbyyuRrriOMGd1oYeDwSu062EqDTUJ6XOPkFKj6jIC6XMH0oh9uP43A7\nvXLP0MIVmL78KHg8tP7sx4zU1XrlvkogJT6UpckR1F/qo1OhG7hT4WRXDSMuK6tNJeg1k+6VT8pA\nn5VXNldhGRqltHIRxRXpsxepIBo7hrnUPkReRgyxEUFyy1G5hoqk2fXNnC2Sy0X7f/5mLJBbspSk\nrz2GNlhtU3QzZhrMPQ9oRVHcD/w98C0AURSfFEWxTBTFTGAN8H+vaUNwH2Mll18QRXEP8Dvgc7P/\nK6jMR9YkqUYo06F8uRGdVsPuKv81Qjncfhynx8WapDI0wo2nJkEQqLwzk9yiJPq6R9i6uZqReeSC\nNx3qLvbSM2inNDvBr7Ox8xGdRsdqUwlWl40TXTVeu29ofgGmR78KgkDbr36OpUoZZ1q8wfqiMfML\nf+6bua/1MALCFUfA2dDfO8Irm6sZGXZQtn4xRWVpXlCoLHZVtQCq8YkSWRqZQXyQPP4FHqeTtt/9\nGsuJ4wRliiR/9Rtog9Rg/1bMaOtoPJv28A2+//1r/vdmIfT6mYypsrBYFJ6GKcRIdXc9Q45hwgOU\nVZ+sNMKCA1iZFcehU52YmwbISouSW9K08Ege9rYcQq/RU5ZYfMv3CoJAxW1L0OoEao628Mqz1dz3\nYD6h4d61AFc6u8YXvRuK/Puc5Hyl3FTK9sZd7Gs9NOkzPR1CcnJJ+srXaf3FT2n77a9I/PwXCSue\nm8bUvmRFZjzhwQ3sr23ngTWLCfCzxtHNw61cHmoiNyaLmKDoWd2rt9vCq8/VYBtxUr5pCXnF8+8z\nbrW7OHy6k5jwQJYvVs3OlYYgCKxJWsWL51/jUNsxbktb55NxPQ4Hbb/+Jdb6WoKylpH06FfRGAw+\nGdufmWlmTkVlThEEgYqkVXgkD4faZt6AdyExsbu5yw93tk/3mum197EyoYBg/eSlFIIgULY+g6Ky\nVAb7bbz8bDVDAzYfKFUGXQM26i70kpEUTppR3ehQIjFBUeTGZtE41EzjULNX7x2ctYzkrz6GRq+n\n/Xe/YejwQa/eXw70Og1r8k2M2F0cPdMlt5xps6/VO+0IujuGeeXZamwjTtbcvnReBnIAh0514HB6\nWFdoQqNZmOfilc6qxGICNHr2tR7CI3nmfDyP3U7rz3+Ctb6W4Nw8kv7xa2ogN0XUYE5FsZQYCwnQ\n6DnQdsQnE4m/syQpgqS4EE6e62bQz86S7WkdW4yuTV495WsEQaC0cjEr16QzPGjnlc3VDPrxeZvp\nsPtkKxKwoXB+LvTmC2uSxp7nuSgXD1q6lKSvP44mKIiO//4vBvfv9foYvqaywIQgXC2/8xdsLhvH\nOqqIDowiO0ac8X06WgbZuqUax6iL9XeL5BbNz/JDSZLYVdWKViNQkae2VFEqwfpgVhoL6bX3c6r3\n7JyO5bZaafnp09jOnhk7H/z3j6IJUB2rp4oazKkoliBdEMUJYxPJmb5zcstRPIIgsL4wabwBb5vc\ncqZMt7WXM73nWBSeRkrY9BcvxeXprFq3GMvQKC8/W01/z/zuT+hwutlX20ZYsJ7irNnZ3qvMLcui\nlxIbGM3xzuo5acAbtHgxyd94HE1ICJ3/+z8M7H7H62P4ktiIIPIzYrnUPmaM4S8caT+Jw+OkwlR6\n0/O+k9Ha2M+rz9fgdLjZeO8ysuaxqdG55gHaekZYIcYREaIu2JXM2vENqT0tc5f9d1sstPz4h9jP\nNxBWUkriF7+ERq+eA58OajCnomiuGqHMvgHvQqAsZ6IBb5vfNODd13oICYnKaWTl3kvhqlTKNy7B\nanHw8rPVdHfMX2vqo2e6GLG7WJtvQq9Tp3AloxE0VCStwulxcrjjxJyMEZiWTspjT6ANC6frmT/R\nv/3NORnHV2wYz0btOukf5eKSJLGv9RBaQctq08zOLjZd7OX1F+rweCTueCCXpdkJXlapLHZXj202\nrleNTxRPcpiJjIhFnOk7R6e12+v3dw0N0fyjHzB6+RLhqyswPvJFBN3snWAXGupKQEXRpIYnkxqW\nTH3PWXpsfXLLUTxBBh2l2Qn0DNqpv9Qrt5xJcbgdHGw/Rpg+lIL45bO6V97KZCrvzMRuc7J1SzUd\nrYNeUqks3jnZgiCMlaSpKJ+yxJXoNLqxTYs5cpo1JKeQ8viTaCMj6f7Lc/RufdlvXW2zF0UTHxnE\nkTOdWGzeaeswl5wfuEiHtYvC+OWEBUy/mfFFczdv/LUegLs+uJxFmbHelqgohkYcHD/bhSk2hMyU\nSLnlqEyByuSxc6D7Wry7qe4a6Kflqe/haGkmYt0GEj7zWQSNGpbMBPWnpqJ41iWXIyGxt9X/D/n7\ngondznf8YGf7eGc1NpeNci/1ZcouMLHx3mU4HW5efa6Glsv9XlCpHC61D3G5Y5j8jFi1L5OfEBpw\ntW+muf/8nI0TkGgi5Ylvo4+No3fry/S88LxfBnQaQWBd4UTfzHa55UzKxHnImRifnDvVyfaXT6HR\nCtzz4eWkLp6dC6Y/sK+2DbdHYl2BCUFQjU/8gfy4XCICwjjccRy7yzutgJy9PTT/4Hs4OtqJuu0O\n4h/6pBrIzQL1J6eieIoS8gnTh3Kw7Rijbv8y9pCDNGMYGaZw6i70KroBryRJ7G05ONaXabyc1htk\n5iRwxwM5eDwS216opfG88jOUU+WdE2PGEBtWqOVJ/sTa8YX+XPfNDIiLJ/mJbxNgTKR/+5t0/fmP\nSB7/M4+qyEtEr9Owq6oVj4ID0sHRYaq76zGFGMmISJ/WtWdq2nn71TPoA7Tc+7F8kvysncxM8Hgk\n9lS3EaDXsDrXKLcclSmi0+goN5Vic9k51lk16/s5Ojpo/sF/4OzuIvp99xL7kY+pgf0sUYM5FcWj\n1+ioSCoddwybP01y55JNxSlIwNsnlOsKd2moiWZLG3lxOUQFerfcZlFmHHd/aDmCIPDmS/VcOOt/\nVufvZdjq4MiZLuKjgshOn/87+POJ9PBUkkNN1PacYmB0bst/9VFRJD/xLQwpqQzu3U3Hf/8nkss1\np2N6m9AgPSXL4unqt3H6snLL6w+1H8UtuVmTtGpai9GaY83sfsOMIVDHfQ8WYEyKmEOVyqH+Ui89\ng3ZKlyUQHKgaXPgT5Ulj5j57Ww7OKuNvb2qk+Qf/gauvj9gPfIjY+z+oBnJeQA3mVPyCiqRVaAQN\nu1sO+GXpkK9ZIcYRGRrA/tp2bKPKXMjtHXfHqkyaufHJrUhZFM09H81Dq9Ow45XTnK1VfsnWrdhf\n147L7WFDYRIa9ZefXyEIAmuTyvBIHg60HZ3z8XRh4SR/8wkCM5YwfOQwbb/9FR6n8s+fXcuGorG2\nG0o1QvFIHva3HiFAG8BKY9GUrpEkiWP7LnHw7QsEhwZw/0OFxC2gPpE7jo9XFhSpLVX8jUhDBIVx\ny2kb6eD8wKUZ3cN2voGWH34ft2WY+Ic+RfTd7/OyyoWLGsyp+AUTE0n7SCcNAxfklqN4dFoN6wuT\nsDvcHKzvkFvOdQw5hjnZVYsxOJ7MqIw5G8eUEsl9D+YTYNCxa5uZOgVnKm+FxyOx62QrAToN5fPY\nsnw+U2wsJFAbyIHWI7g97jkfTxscQvLXHiMoaxkj1VW0/eKneEa9c97FFyxKDCfdGEb1+R56B+1y\ny7mOup7T9I8OUJJQSJAucNL3S5LEgbfPc/xAI+GRgTzwiUKi40J8oFQZtPaMcOpSH5kpkaQtoAB2\nPjHRB3bPDPwLRk6fouXHP8QzOorxc58ncv0Gb8tb0KjBnIrfsC6lAoDdzQdkVuIfVBYmodNq2Hm8\nWXHnTg62jZcnJZfNeYlFfGI47/94AUEhevbvOM/xA5f9Lrs7UZ60KieBELU8yS8xaAMoTVzBoGOI\nmp5TPhlTExhI0le+Rkh+AdbTp2j5yY9wW/2nD+P6oiQkCfbUKC87t6t5P3D199Kt8Hg8Y5tJx1uJ\nig3m/ocKCY9cWAZGbx9vBuC2YjUr569kRKSTFJpITXf9tMrFLVUnaPv5T8DjwfTlRwlfNTfVOAsZ\nNZhT8RsWhaeSGpZMbc9petU2BZMSHhxAaXY8nf026i8q5+fl9rjZ13p4bHFrXOGTMWPiQ7n/oULC\nwg0c23eZA2+f96uAbsKZdH2huhDyZyrHjVB2Ne/z2ZgafQCmL/0DYSWl2MfLnFyDAz4bfzaULEsg\nJFDH3uo2nC7lGLk0D7fSMHCRZdGZJIbcuiec2+VhxyunMdd1EGcM4/6HCgkJM/hIqTKw2JwcrO8g\nJjyQwqVxcstRmSGCIFCZtPpKifFUGDp0gLbf/Aq0WpK+8nVCCwrnWOXCRA3mVPwGQRCuaVOgNhGf\nCptWpACw80SzzEquUtd7hoHRQUqMK6ZUnuQtIqODuf+TRUTFBlN3vJV3Xj+Lxw+c/roGbNRd6CUj\nKVwtT/JzEkLiyY3J4uJgI5eHmnw2rqDTYXzki0RUrmO0uZnm7/8Hzm7vNwD2Nga9loq8RIasTo6d\n7ZRbzhUmsnLrJ8nKOR1u3nixjovmHkwpEdz3YD6BQQsvs763pg2Hy8PGFcloNOp5X3+m2DhWVry/\n7TAuz63P4w+8s5OO//4vNIGBJH/9mwQvy/aRyoWHGsyp+BUTbQoOtB1V2xRMgTRjGJnJEdRf7KO9\nVxnlVRPGJ2tn0JdptoSGGbj/oULiE8M4V9/JWy+dwuWa+/NLs2H3yVYkVNOA+cL6lDXA1YDAVwga\nDfGf+DTR99yLs7uLpu9/l9FW5Z8h3ViUjCDA9qPNisimD44Oc6KzmoTgOJZFZ970faN2J6/9pYbm\nS/2kZcRwz0fyCDDMvpemv+H2eHjnZAsGvZa1+ep5X3/HoA2gLHElww4L1V11N3yPJEn0vvoKXZuf\nQRseTso3v0VQxhIfK11YqMGcil+htimYPpuKx7JzSmhT0DHSibn/PEsjF2MKlafPUGCQnvsezCcp\nLZLL53t5/S91OBTq+Gl3uNhb00Z4sJ5iMV5uOSpeQIxaginEyMmuWvrtvi13FASB2Ac+SNxHHsQ9\nOEDzD76H7cLcNTL3BrGRQawQ42nqsmBukr88dH/rIVySm3XJFWiEGy+hRiyjvPJsNR0tQyzJjueO\nD+Sg02t9rFQZnDzXQ9/QKKuXG9V2BPOENeMbsTcyQpE8Hrq3PEPvK39DFxNDyhPfxpCS4muJCw41\nmFPxOybaFOyZZb+ThUJhZizR4QYO1HVgtctrT76nZaw8dsIVSy70ATru+XAeizJjaWsaYOuWamxW\n5WV699e2Yx11saEoGb1Ona7nA4IgsD5lDR7Jw56W6bvCeYOo2+8g4eFH8NhttDz9FCOn6mXRMVVu\nXzm2GNx+TN5ycafbyb7WwwTpgihNvPF534E+K3/7cxW93SPkFJnY+L5laLUL97O7Y9z4ZNMKtbJg\nvhAfHEt2tMjFwUaah6+aE0kuFx2//x0D77xNQFIyqd/6DgEJanN4X7BwZxgVv+Xafidqm4LJ0Wo0\nbChKZtTpZr+MvdZGnFYOtx8jyhBJfmyObDom0Oo03H5/Nll5Rro7LLz8TBWWIeVYoHs8EjuON6PX\naVhXlCS3HBUvsjKhgFB9CAfajshWLh5RXoHpy4+Cx0Prz3/C8LG57383U5YkRZBhCqfmfA8dfVbZ\ndBzvqmHYaaHCVIpBG3Dd690dw/ztmSqGB+2srEhnzW1LF/QZsUvtQ5xvGWT54hgSYxZOG4aFQOVE\nm4LxDSmP3U7rL37K8NEjBC5ZSsrj30IXGSWnxAWFGsyp+CVqm4LpsTbfhF6n4e2TLXg88mQz97Ue\nxuFxsj6lAq1GGSVHGo2GdXeJFJSmMNBn46U/n6SvWxlnC6sauukesLM610h48PULRxX/Ra/Vsyap\nDKvLxpH2E7LpCC0oJOlrj6HR62n/z98wsPsd2bRMxm0rU5C4munxNZIksat5HxpBw9rk68/7tlzu\n55XN1ditTtbesZTiivQ5b7uidHaq7QjmLdkxIrFBMRzrOEl/bzstTz+F9VQ9IXn5JH/tMbQhavDu\nS9RgTsUvUdsUTI/QID1lOQl0D9ipudDj8/GdHhd7Wg4QqA1ktanE5+PfCkEQKFufQdn6xYwMO/jb\nM1W0N8t/Nuet8ZKyiRIzlfnFmqQydIKWXS378EjyuaoGi1kkf/NJtKGhdD3zJ3pefkmR5esrxDhi\nwg0cqGvHYvN9uXjDwEVaLe0UxOUSHfjujMOFs128/kItbreH2+/PJqdQzaQPWEY5eqaLxJhgchZF\nyy1HxctoBA0bU9YQZHHQ8tT3sF+6SPjqckxffhSNYWG13lACajCn4peobQqmz5U2Bcd9b4RyvKOK\nIccw5UklPm1HMB0KSlPZ+L4sXE43rz5fy6Vz8lm3X2gb5HzLIHkZannSfCXCEMaKhAK6rD2c7jXL\nqiUwLZ2UJ7+DPi6Ovte20vnHPyC5leXyqtVo2LgiBYfTw55q3zcRn3Af3TDuRjpB/clWtr98Gq1W\nwz0fziMjSzUqAthd1YrbI7GpOGXBZyjnK0Uk8ZEdAxh6hwi/7XYSHn4EQbfwHFuVgBrMqfgtE20K\nDrYdxaG2KZiU5PhQslIjOdPYT2u3xWfjSpLE28170Qga1iffui+T3GTmGrnrQ8sRBHjrb6c4VeX7\nRSOM2bAD3KFm5eY1crUpuBEBCQmkPPkdDKlpDO3fS9uvfo5ndFRuWe9ibb4JQ4CWt0+04HL7LpvZ\nbe2lruc06eGpLIpIA8bmtWP7L7NvewNBwXre//ECktPVM0IATpebXVWtBBt0rM5RDTDmI7bzDXT8\n8ClCrW72FYRgLlfLiuVEDeZU/JaJNgVWl42japuCKTHRpmCnD9sUnO47R/tIJyvi84kKjPTZuDMl\ndXE07/94AYYgPXvfauDovks+LTvrGbBx3NxFanwoWWnq4nA+kxJmYmnkYs72N9Bm6ZBbDrqICFIe\nf5LgnFxGamtoefop3MPDcsu6QnCgjjV5iQxYHBw72+Wzcfe0HEBCutIk3OPxsOfNcxzff5mwiEAe\n+GQhccYwn+lROkdOdzFsdbK2YCz4VplfDJ84TsvTT+Gx2Yj45CeozY3gnaa9spaLL3RmFMyJoqgX\nRfFZURT3i6K4RxTFxTd4j1MUxd3X/NFO5ToVlelQkbQKnaDl7WZ1IpkKBUtiiY0I5FB9h8/OnbzT\ntBeAjalrfTKeN4hPDOcDnywkPDKQEwca2fPmOTwe3zxfO463IElwe4lanrQQ2HAlO7dPZiVjaAKD\nSHr0q4StKsN+8QJN3/8uzh75So7fy6biFAR810Tc5rJzqP3YFRdlp8PFGy/Wc6amndiEUB74ZCER\nUcFzrsNfkCSJnceb0QgCG4tU45P5Rv/bO2j/7a9AoyXpH79GQuUmSo0r6LH3UdN9Sm55C5aZZuY+\nDgyYzeYK4LvA927wnkGz2bzumj/uKV6nojJlIg0RlBiL6LL2qBPJFNBoBDYUJeNwedjtgxLC5uE2\nzvY3kBmZQUqYf5kCREQF88AnColNCOVMTTtvvXQKp3NuzxFZ7S721rYRGRpAybKEOR1LRRnkxi4j\nNiiGo51VDDt8V/58KwSdDuNnP0/UnXfj7Oyg6Xv/jr2pUW5ZAMRHBlGUGUdj5zDnfGBUdKj9GHb3\nKJVJqxm1unhlczVNF/pIGc/gh4SqZg/Xcq55gKYuC0WZscREKPN8tMr0kTweul94nu4tz6INDyfl\n8ScJyV0OwMbxDamdTXsUaZ60EJhpMLcR+Nv41zuB8jm+TkXlpmxKrURAYEfjbnUimQJr800EGbTs\nPN6MY46Dk3ea/S8rdy3BoQbe//ECktIiuXy+l62bq7GOzN35zL01bYw63GwqTkG3gBsNLyQmzpK6\nPC72tx6WW84VBI2GuA99hLiPfRz30BAtT32PkdPK2DC7zUdNxD2Sh93NB9Br9OQELeelP1fR3WEh\nK8/IXR/MJcCgmj28lx3jBlsTJf0q/o/H6aTj9/9J/1tvoDcaSf3WdwhMS7/yekJIPMtjs7k81MSF\nwcuy6VzIzHS1YAS6AcxmsweQRFF8byOkQFEUN4uieEAUxa9P4zoVlWmREBJPflwOjcPNnOtXm4hP\nRnCgjg1FyQxZneyvm7sm4v32AY53VmMMjic7RpyzceaaAIOOez6SR2ZOAl3tw7z0p5P093q/F53L\n7WHniWYMei2VBSav319FuaxKLCZIF8ie1oM4PS655byLqE23k/j5v0NyuWj92Y8Z3L9XbkksTY5g\nUWIY1Q09dPbPXRPxup7T9Nr7WKEv4a0tZ680A193l4hW3Wy5js4+K1UN3aQlhLE0OUJuOSpewG0d\nofWnTzN89DCBGUtIffI76GPjrnvfptRKAN5ukn9+WIhMuq0kiuIjwCPv+Xbpe/7/Rgc7HgOeASRg\nryiKN/oXnvRASFRUMDqdMg/QxsWpB56Vwkfy76F6Zz17OvZRIRbKLcdrzNUz9tE7sthxrJntx1v4\n0Ka5WZi8VbMTj+Th/pzbSYj3/1/sH314JXu2n2Pv9nO8/Ew1H314JWkZMV67/56TLfQNjfK+ikWk\np/i2L5M6l8lNGBszKnjNvJNz1rOsW3R9U2o5ibtnEzHpJs5+7wd0/u//oB8ZJPWhB6d9ptObz9kH\nN2Tyo2dPcOBUJ198IM9r972WfXUHCe8zYrsUiSS5ue+j+RSUpM7JWPOB53ZdQJLgY3eIxMeHy6ZD\nnc+8w2hPL6ef/gG2xiaiV5WS+fWvoL1JD7nY2OUsuZxOXc9pnIYRTOHz28VUac/YpMGc2Wz+PfD7\na78niuL/MpZlqxFFUQ8IZrPZ8Z7rfnvN+98GlgNtk133XvrncNdtNsTFhdHdrRyXr4VOBDFkRmZQ\n03GGkxfP+t35rBsx189YeV4iu0628vq+C5R52T7a7rKz4/xewvShZAUvmzeflZwiE1q9hj1vmPnz\n7w6x/u4sMnNmf7ZNkiT++vY5BKAiJ8GnPy91LlMGJdEreZ23eeX0DrJDcpRnfhOfQvIT36H1Zz+m\n5YUXGWxsJeHhz6HR66d0ubefs0xTGFFhBnYcaeLO4mSCA6emY6qc779EV52LlKZCNAEabr8/h6RF\nUepn5Sb0Ddl5+1gTCdHBLDXKN6eo85l3sDdepvUXP8U9MEDkhk3EfOzj9A05gJsv2StNFZzvu8xf\na97kwawP+k6sj5HzGbtZEDnT7fjtwIfHv74X2HXti+IYm0VRFERR1DF2Nu7UZNepqMyG29LWAbCj\ncbesOvyFO0tS0QgC2w43ev2s4cH2Y9hcdiqTy9FrvbvIkpus5Ubu+UgeOp2Gt189w4mDs//5ewCo\nOwAAIABJREFUnWse4HLHMEWZccSrzngLkpigKIri82i1tHO6T94m4jcjwGgk5dvfITBjCcNHD9P6\n4x/itshj2qLTati0IplRp5s9NW1evbfH42H7G7UkNmVjCNbx/o8XkLrYt9lyf+Oto824PRJ3r0pF\no1HYRoTKtLBUnaT5B/+Be3CQ2A9/lLgHH0LQTB4uFMTlEhMYzZGOE4oxc1oozDSYex7QiqK4H/h7\n4FsAoig+KYpimdlsNgPNwFHgALDNbDYfvdl1KireYFl0JsmhJk521dJt7ZVbjuKJiwyiJDue1u4R\nai547+fl9rjZ1bwfvUbPmuRVXruvkkhOj+KBTxQRGm7g6N5L7H7DjHsWTYwnjBxuL1FNAxYyd6Rv\nAOCNS28r1sxJFxZO8jceJ7R4JbaGczR9799xdPmu59u1rC0wYdBr2Xm8BafLO61DRu0uXtxyDKEp\nAk+onY98eqXaQ24ShqwO9tS0EhVm8HqVh4rvkCSJ/u1v0vbrXwBg+vI/EH3HXVOuEtAIGjakrsHp\ncbG35eBcSlV5DzOyYhpvM/DwDb7//Wu+fmKq16moeANBELgtbR1/OLWZnc17eFD8gNySFM/dpWkc\nPtXJtsONFCyJ9co9q7vr6LP3szapjFB9iFfuqUSi40L4wKeK2PZCHWdrO7AMjXL7/dkYplnu1dln\npbqhh8WmcJYk+f/ZQpWZkxSayPLYbOp6TtMwcIHMqCVyS7ohmoAAEr/wJXpi4+h/cxvN//H/MD36\nFYIyfKs3JFDPukITbx1tZn9dO+sLZ1dePzRgY9tf6+jvsTEU0cXdD+QRptrrT8rO4y04nB4+WJmq\nuvD6KZLLRdeWZxjcsxttZCRJj371XY6VU6UscSWvX9zO3tZD3Ja2jgCt6nHoC9RPncq8ojBuOTGB\n0RxuP86QQ62bn4zk+FDyM2I43zLolZ5NkiSxs2kvAgLrx3vPzGdCQg3c/1ABaUtiaLncz0t/OslA\n3/TO+W473IgE3L5SbRKuAndOZOcuvyOzklsz0bog/lOfwW0doeWH32foyCGf67izJBW9TsO2Q424\nZpEd72gd5MU/naS/x0pPwiWEFZ3kJvivC6+vsI26ePtEC6FBetbmqy68/ojbaqX1Fz9lcM9uDCkp\npH77/8wokAMwaANYm1SGxTnCkY4T3hWqclPUYE5lXqHVaNmUWonL42J38wG55fgFd5elAWNBxWxp\nGLhI03ALebHZxAd7J9OndPQBOu78QC4FpSkM9Nl46U8nabncP6VrewZsHKzvIDEmmGIxfo6VqvgD\n6eGpLIvO5Fz/eS76Qc+myLXrSPrHryHo9XT81+/oeemvSB7vlDxOhYhQA5X5JnqH7Byq75jRPRpO\nd7J1czWjNif65YN0pJ3hjkUb1M2VKbCrqhXbqIvbV6Zg0CvTeVzl5jh7umn+/r9jPVVPSF4+KU/8\nE/ro2Z0PXZtcjk7Q8k7TPjyS7+aChYwazKnMO1YlFhOmD2Vv60FsLrvcchTP0uRIMpMjqL3QS1Pn\n7LKZ2y7tAK6a0SwUNBqBsvUZrL9bxOlw8/pfajlVNbkpw2uHGnF7JN63Ol01DVC5wp3pGwF4U+HZ\nuQlCcpeT8q1/Rh+fQN+212j7zS/x2H039961Kg2dVuC1Q5dxTyOQlCSJ4/svs3PrGbQ6DRXvT6cm\n+BCmECPLY5fNneB5gsPpZvvRJoIMWjYU+b+D9ELDdr6Bpu/+PxxtbURuuh3TP3wFTeDsy4ojDGGU\nGIvosvVQ033KC0pVJkMN5lTmHQFaPetSKrC57BxoOyK3HL9gIjv3xpGmGd/jXP8FGgYukh0tsigi\nzVvS/IqsvETuezCfAIOOvW+dY/+OBjw3WVz2DNo4UNdOQnQwpctm395AZf6wJHIRSyIXcar3LE3D\nLXLLmRIGk4nUb/8zQVnLGKk6SdP3v4uzt8cnY0eFGajIM9E9YOfI6c4pXeN0utm59QzH9l8mLCKQ\nBz5RRC0n8Ege7khbj0ZQl0eTsa+2nSGrkw1F3m8NoTK3DO7bQ/MPv497xEL8Q58k/mMfn5Jj5VTZ\nlFqJgMAbl3eq2TkfoM5WKvOStUmrMGgDeKdpL06PS245imf54hiS40I5eqaTrhn2dpzIyt296DZv\nSvM7ElMi+eCni4iOC6HuRCvbXqhj1O687n3bxrNy965OU7NyKtdxV/omAN7yk+wcgDY0lOSvfoOI\ndRtwtDTT9O//F9v5Bp+MffeqVLQagVcPNuLx3NoJdHjQzsvPVHH+TBfGpHA+8KkiNOEujrQfJz4o\nlqKEfJ9o9mdcbg9vHmlEr9NwW7HqwusvSG43XZufofOPf0BjCCT5a48RuX6j18dJCImnOKGQVku7\nmp3zAWowN0usVisf+tC9APzLv3yL0dGZlZZcvHiedetW0d7u3X45C5VgfTAVSasYdAxzrOOk3HIU\njyAI3F2WiiTBmzPIzl3JysWILIpInQOF/kV4ZBAPfKKQtIwYmi9db4zSO2hnX2078VFBlGarWTmV\n6xGjlpAenkp1dz1tlpmdBZMDQacj4ROfIv6hT+IesdDyox8weGD/nI8bGxHE6lwjnX1Wjp29eauE\ntqYB/vrHE/R0WliWn8h9DxYQHBLAzqY9uCQ3t6lZuSlx5HQnvUOjrM03ER6iOhb6A26LhZaf/IiB\nd3YSYEoi9Tv/QvCy7Dkb765FGxEQ2HZph5qdm2PUGcuL/Nu/fQ+DYfr1xpIk8ctf/ozkZHV3y5ts\nSFmDVtCys2mPOpFMgZVZ8cRFBrK/rp0By+iUr5MkidcvbQfgngWelbuWAIOOOz941RjlxT+eoPH8\nWD+/bYcnsnLpaL1Y2qIyfxAE4Yqz5VuN/pOdmyBy/UaSvvoNhIAAOv/wey794Y9IbvecjnlPWRoa\nQeC1g5fx3KBP36mqVl59rgaH3cWa25dSeWcmWp2GYYeF/a1HiDJEUmIsnFON8wGPR+L1Q41oNQJ3\nlqibd/7AaGsLTd/9N2xnzxBSUEjqt79DQPzcmm4lBMdRYiyibaSDqq66OR1roaOuImbAyIiFhx9+\nmC9/+RH+9Kf/ufL9D33oXqxWK9/97r/y61//jK985Ut8+tMfY/v2N/jKV77MZz7zcSwWy3X3e/31\nrRQXryQyMsqXf415T6QhglJjEZ3Wbk521sgtR/FoNRruLE3D5ZbYMd7Eeio0DFzg/MAlcmKySA9X\nf7Ffy4Qxyob3Zf1/9s47LK7r2tvv9KEz9N5haAIB6r1axSoukuxILrFjp7cvNzeJc53Yvk67uTex\nZStxrFiucWxLsmWrS1axJKuLJkAwIHqvQxkGpp7vDyxZBTUEzADzPg8Pw5k95/xmZrPPXnutvRYW\ni8DurXkcO3SRY7l1+Hk6MSXJ4ZVzcGOSvRMIdg0kszGXJn2zreXcMS6JSYT912+RBQRQ9+l2al76\nP8xdnUN2PT+VM1OS/Klt6SZL8/XnZbFYObJXw9F9JcgVUpY/nEpyevDlbJWHqo9hsppYED4bqXhA\n5XfHFFnFzTS06ZmaFIC3ow6f3aPLzqTqD7/D1NyM1/KVBH3/R4iVTsNy7SURCxCLxA7v3BAz4ket\nzYcu3jSkYiBMjPdjzbwbFz/dt28PsbGxPP30jzh4cD8HDuy7ro1EImX9+td44YVnycs7z/r1f+fF\nF39DVtY5Zs2ac7ldR0c7e/fu4uWX/86JE0MfijLWWBQxj1MNmews30+aXwoSsSN18s2YMS6Az74s\n53B2LUumhOPqdPNN7X1euUt75RYMh8QRiTo5AC8fF/Z+kk/+mRoigGkTQxxeOQc3pc87N59N+f9i\nf+UXPJKw2taS7hi5fwBhv/4t2vffou30WapefIGg7/8QZUTkkFzv3qnhnMxvYOeJCjLUvvToTezb\nVkBDTQc+fq4sfjD5qkLgelMPR2tO4iZ3ZVrgpCHRNJoQhD6vnAhYMsWxeGfPCFYrbbt20PrZNkRy\nOYHf/QFuEyYOqwZfZ28mB2Rwsv4sWY25THB4vocEx0xiAFRUlJGW1tch09Iy+m2TkJAEgLe3D3Fx\nfYVHVSpvuruv9sy99tqrPP3095BKR7xdbZf4OHkzI2gyzT2tnKo/Z2s5do9MKmHJ5DB6jZbbqjtX\nrO3zyiU7vHK3xDfAjQWrxtGFgAoRdZl1aFu7bS3LgZ0z3jeZAGc/Tjdk0tpze/UL7Q2JszPxv/oF\n3vc9gFnbRvWffk/Hl8eG5FqB3i5MTPCjqknHl2eq+fidTBpqOoiO9+W+R9KuMuQAjtScoNfSy/zQ\nWcgljoyMt6KgvI3Kxi4y4v0I9HaxtRwHN8DS3U3d316h9bNtSL29CXvm2WE35C6xOGJ+n3fOkdly\nyBjxFsSaeTE39aINBYIA4q9W1G+UNUsikfT7WLgmjj8z8yxlZaUAVFSU8+tf/5z161/D3d1jsGWP\nWRZHzOdk/Tl2VxxgYkC644Z9C+alB7P/bDUHM2tYkBGCl3v/YTRXe+Uce+Vuh0O5dRQhsDjSi9Zy\nLR+/k8X8ZfFExvnaWpoDO0UsErMoYh7vXPiQA1Vf8JD6fltLGhAisRjvZStQhkdQ/89/0Pj2Jnor\nyvtSog/yYuayqeGUFzaRd7gMETB5diRpU8KuKwLeazZwuOYYzlInZgZPGVQNoxFBEPjseDkA904Z\nm+VnRgK9VZXUv7YBU3MzzolJBD79XSRubjbT4+PkxdTACRyvO8O5xhwmBaTbTMtoxeGZGwBhYeHk\n5+cDkJV1d96eLVu2s3Hj22zc+DZxcWr+8If/cxhyg4yHwp05IdNpN3RwtPaEreXYPTKphPtmRGIy\nW9n+1Y27PzTai5R2lJPsnUC4uyN5z63Qdhk4klOHj4eSB1aNY8GKBASrwN5PCjh9tOyW6dQdjF0y\n/FLxcfLmRP1Z2g0dtpZzV7iMSyHs2eeRh4TS8cUhqv/3T5jbB8/jaDJaKDpdTQRizAgkzYogfWr4\ndYYcwJd1p+g26ZkTOgOl1LH361ZkFbdQWttJepwv4QG2Mw4c3JiO48eo/uNX++OWLSf4p/9hU0Pu\nEovC5yMRSdhd/jkW69AmQhqLOIy5AbB48b3k5OTwk598j+rqyn5vEg7si4Xhc3CSKtlfeZge88DK\nR4wlpo0LINDbmWPn66nvJxRQEIQr6so59srdDntOV2K2WLl3ajhSiZjYRH/ufzQdNw8lWSeq2LX5\nPPpuo61lOrBDJGIJi8LnYraaOVB5xNZy7hq5n19f2NekKfSWXqTyxefRF2vu+rzaVj0fv5tJSUET\nnr4uXEDgWGnLdRExAHqTnn0Vh3CSOjEnZPpdX3u0Y7Fa+fhIKWKRiAdnR9lajoNrsJqMNL77Fo1v\nbUIkkxH0o5/ic9+Dg1oI/G7wdlIxNWgizT2tnG3MtrWcUYfk+eeft7WGm6LXG5+3tYZrUSgUPPLI\nw8yatZD09AmsWfMNANasWYtMJmPWrDmEhfWFIEyaNIXYWPV1j/tj6dLluNnBCspoRC6RYRWs5LcW\nIRVJiFNF21rSLXFxUaDX22ZyLxaJ8HRVcKawic5uIxMTrs66qNFeZG/lIcb5JLAgbLZNNI4k2nUG\n3thZiMpVzhNLEy4XCXd2lROX7I+2RU91eRslFxrxC3C7bl/PUGLLfubg9gl08edsQzYa7UUmBqTh\nLHO2taQ74tp+JpJKcU3PQOLsjC47i87jX4JYjFNM7IAWSEuLmti9NY/uLiPjMoJZfH8SFU06Ciu1\nxIV64ut5dfa+HeX7KNaWsjxqEfFesXf9/kY7R3PrOJ7XwOzxQcxICbK1nBsyFsczU0sztS//le7z\nuShCwwj5+S9wirK/OU6IayBHa05Q01XHrOCpI7aeoy37mIuL4oX+jo/MT9KBgwEwJ2QGbjJXDlYf\npct4fYkIB1eTHudDZKA75zTNlNd/nU78qr1yEY69crfD3tNVmMxW7p0agVRy9bCrdJKxZFUyU+ZG\n0dNtZPsHOWSeqOzXm+Bg7CIVS1kZvQSLYOGz0j22ljMoiEQiVAsXEfqfzyD1VNH66SfUrv8r5s7b\nL19gsVg5fvAi+z+9gCAILFiRwIyFsUgkYpZPjwBg6xelV9Wda+1p40j1cbyUKmYHTxvstzXqMBgt\nfHasHLlMzIoZQ5OF1MHA6M47T+WLz2OorMB9+kxCn3kWue/Q1o8bKCqlJ9OCJtPS28bphixbyxlV\nOIw5B2MGpVTB4oj5GCxG9lcetrUcu0ckErFqTt/q3tYvSi8f12gvUtZRwTifBMLcQ2wlb8Sg7TLw\nRXYtXu4KZqQE9ttGJBKRNjmMlevScHaVc+ZoObu25NEzxlaYHdycdL8UItzDyGo6T1nHrbPNjhSc\nYmMJf+6/cRmXgr4gn8oXfoteU3TL13V19LL9gxzOn63B09uZBx/PIDbx6yiCyEB3Jif6U9HQxemC\nxsvHd5TtwyxYWB61CJkjIdYt2X+umo5uI/dMDMPTVWFrOQ4AwWymecuH1K7/K4LBgN9j38T/m08i\nlsttLe2mLIqYi1QsZW/FAcxWs63ljBocxpyDMcX04Ml4KVUcrT2Jtrfd1nLsnoRwFcmRXhRWaimo\naPvKK7cfcGSwvF0+PlKK0WxlxfTI67xy1xIY4sHqJyYQGuVFdVkbW97KpL5mZCe8cDB4iEQiHohZ\nBsC2iztHlfdW4urat89n1RosXZ3U/N//0LpzO4K1/1TmpUXNbH7zHA01ncQk+PLgY+l4+VyfKv/B\n2VFIJWK2HinFYLJQ1VnD2cZsQt2CmeA/fqjf1oinU29kz6lKXJ1kLJnsKD9jDxibm6j60+/R7tuL\nzN+f0GeexXPWnBGRv8FT4cGMoMm09mo5XZ9pazmjBocx52BMIRNLWRq5ELPVzO7yA7aWMyJ4cPbX\n3rlLHoFUnyTC3BxeuVtRVtfJifwGQv1cmTGuf6/ctTg5y7l39Tgmz45ErzPw2fvZZJ+qGlUTdwcD\nJ9ozgvG+4yjrqCSnOd/WcgYVkViM1+KlV4ddvvyXq8IuzSYLR/Zq2P9pAVaLlTlL1CxYkYhc0X95\nAx8PJxZNCkXbZWDf6Uq2XdwFwP3R947YPTvDyc4TFfQaLSyfHoHTDT5jB8NH15nTVP33cxgqynGb\nOo3w3zyPMjzC1rLuiHvC5yITS9lTcRCTxWRrOaMCx0jmYMwxOSCdAGc/TjWco1HfbGs5dk94gBuT\nEvyobGrno6LtSEUS7ou519ay7B5BEPjgYDEAaxfEXk56cjuIRCLSp4az4hvjcXKRc+qLMnZ8mIuu\nyzBUch2MIFZGL0EsEvNp6e5RGap0VdjlhQIqX/gt3QX5tDbr2PpOJhdy6vH2dWHVNzNISA28pUdi\n6ZRw3J1l7CnMpLi9lERvNWqv4a1POxJpau/hcFYtvp5K5qYF21rOmMZqMNDwzpvUb3wNwWol4Mmn\nCfzWtxErnW79YjvDQ+HOrJBpaA3tHKw+Zms5owKHMedgzCEWiVkWtQirYGVX2X5byxkR3D8rCllg\nOd2WLmaHzMDP2cfWkuye04WNlNZ2kqH2RR2mGtA5gsI8WfPkBMJjvKmtbGfzprOUaRwLEGMdP2cf\nZgdPo6WnlWO1p2wtZ0i4MuzSrOvi9KYdbH3zLNoWPcnpwTzweDqqfsIq+8NJIWXlzAgILARBxP3R\njsWo2+HTo2VYrAIPzIq+ZYi4g6HDUFNN1e9eoPPYURRh4YT/5gXcp43schpLIubjJnNlX8VBx5aX\nQcDx33mX6PV6Vq1aDsBzzz2DwXBnNcw2bXqdhx++nx/+8Nv88IffZufOT4dCpoNrGO+bTJhbMJlN\nuVR31dpajt0jczIgDypHMCpw0yXaWo7dYzBZ2HK4FKlExOq5d+cBcHKWs+TBZGYtisVitrJvWwGH\ndxdhMo4+j4yD22dx5HycpEr2lB9Ab9LbWs6QIBKLcZ69kNKZ30HjNxWx2UC6IZdJCUqkUskdnUvm\nV4fYWYe5JQhz9+0ZgWOZyoYuTl1oJDzAjYkJ9pkdcbQjWK1oDx2g6vf/jbG+Ds8FC/uyVQYE2Fra\nXeMkdWJl9BKMVtPl0GcHA8dhzA0iL7zwRxSKO68PtXr1w2zYsJENGzaybNl9Q6DMwbWIRCJWRC0B\nYHvpXsd+pFuw7eIurCILQp2aPcdrMZgstpZk1+w9XYW2y8A9E8Pw87z7MBiRSERSWjCrnsjAx9+V\novMNbHkrk8a620/h7mB04SpzYXHEfLrNevZWHrK1nCGhqqyVzZvOUlmjJzDEnQXBraiqs6n6/Qto\n9++9YXKUazFYjOwu349UJMVUE8tHhy46xvxbsOWLiwCsnhONeAQk1hhtmNpaqX3pLzT/+1+IFAqC\nfvgT/B5eh1g2erKvTg7MIMI9jMymXEq0pbd+gYMbMqDdrGq1Wga8DYQDFuAJjUZTdsXzGcBfrnhJ\nInAfcA+wDrjkCnlPo9FsGogGW9LdreMXv/gxOp2elJSvs2GtWrWcd9/9iJde+jMqlQqNpoj2di3r\n1j3Orl076OhoZ8OGjbi6utpQvYNLxHvFolbFcKFNw/mWAlJ9k20tyS4p0fYlPolwDyMyagq7T1Vx\nMLOGpVPCbS3NLmnr7GXPqUo8XOTcO3VwPyOVtwsPPJbOmaPl5JyuZtt7WUycGUnalLA72pPnYHQw\nO3gaR2tOcKT6OLOCp+Hj5GVrSYOCyWjmxKFSLuTUIxaLmDTrUh9PR5eeRONbb9K8+UO6887j/+TT\nyFQ3D2M+VHWMDmMXi8PncbE5kLyyVnIvtjI+1hEu3h8F5W1cqNCSFOlFYsTo6FMjBUEQ6Dp1kqZ/\nv4e1pweXlFT8H38CqYenraUNOmKRmDVxK/nzuVfZXPwZv5r4EyTiO/O4O+hjoKmJ1gLtGo1mnVqt\nvgf4I/DQpSc1Gk0mMAdArVZ7Ap8Bp+gz5tZrNJoNdyP6Sj65uJPsprzBOh0AaX7jLqd/7o99+/YQ\nGxvL00//iIMH93PgwL7r2kgkUtavf40XXniWvLzzrF//d1588TdkZZ1j1qw5V7U9fPggx44dQS6X\n89Of/idBQY6NxsOBSCRiTdx9/OHMS2wu/gy1Kgal9M49q6MZq2BlS8l2AFbHrcBPHsiRnDp2nKhg\nSqI/Xu6Oz+tatn5VimDdPVFDkv1NIhEzdW40YVFeHNxZyJmj5VSWtjJ3aTwqb+dBv54D+0UmkbEi\neglvFfyb7aV7eDJ5na0l3TV11e0c2llEV0cvXr4uzF+WgI//1wugrinjUb7wOxrf3kT3+Vwqn3sW\n/0cfx23ipH7P12ns4vOqw7jKXFgQPod0DwsF5W18dPgiyVFejr1g12AVBLZ8cRERfV45B8OHuauT\npn+9iy7zHCKFEv/Hn8B9xqwRUXJgoIS7hzI1cCIn689yrO4Uc0JG9l5AWzHQUWw+sO2rxweAm336\nPwde1mg0txcPMQKoqCgjLS0NgLS0jH7bJCQkAeDt7UNcnBoAlcqb7m7dVe2mTp3OU099l5df/jv3\n3LOEl1/+3yFU7uBaAlz8uCd8Du2GDnaVf25rOXbH8brT1OrqmRIwgQj3MJyVMtbMjcFgtPCv/cWO\nUKVrKK3t4FRBI+H+bky/zVIEAyU4XMWaJycSk+BLY20nW946R/bpKqxWx3cylsjwSyXcPZTMplzK\nR3AhcbPZwolDF/ns/Rx0nb2kTQlj1eMZVxlyl5C6uxP0o5/i9+jjCGYT9a//nbrXNmDuuL4m457y\nAxgsRpZGLsRJqiTYx4XZ44NobNPzRbZjv/S1HM+rp6pRx5Qkf8L83WwtZ8ygy8mm8rln0WWewyk2\njojnX8Rj5uxRbchdYmX0EpykSnaW7afLqLv1Cxxcx0CXjQOAZgCNRmNVq9WCWq2WazQa45WN1Gq1\nE7AI+O0Vh1er1eqVgAH4kUajKR+gBgAeiFl2Uy/aUCAIIBb32cE3mjhJJJJ+H187+U1M/Dq0b8aM\n2bz22quDKdXBbbAofB7nGnM4XP0lkwLSCXVzeEYB9CY9O8r2oZQoWBG95PLxGSmBnCxoIOdiC5ma\nZibEOzbHQ9+K9gcHSwD4xoLYYdlnonSSsXBlElHqZo7tL+bU4TLKNM3MWxp/25n+HIxsLhUSfynr\nNT65uIufpX9vxE0Amxu6OLizEG2LHg+VE/PujScgxOOmrxGJRHjOnotzfAKNb7+JLvMc+qJC/B5e\nh9uUqYhEIup0DXxZdxo/Jx9mBE2+/NqVMyM5daGBz74sZ2pyAC7K0bMP6W7o6Day+dBFFHIJD8xy\neOWGA4u+m+bNH9L55TFEUik+qx9CtXARIvHY8Ri7yV25N/IetpZsZ0fZPtbGP2hrSSOOWxpzarX6\nKeCpaw5PvubvG9057gN2XeGV2w0c0mg0R9Vq9cPAq8BNLTGVyvmOs1YNNYmJceTn57No0SJOnfoC\niUSMr68bEokYHx9XlEoZHh5O+Pq64ewsx81Ned3jS/zud79j8eLFTJgwgSNHsoiPV1/1vIPh4TuT\n1vG7I6+wpfRTfj//F5eNdVtjy77wZtZuuk16Hkm9n5iQoKue++naDH70f4f58FAJsyaE4eLkmAwd\nzqymrK6TGalBTE8PHdZr+/q6kZIWzN5PC8jPrmXL25nMWaRm6uwoxLcRRuYYc0Y2vr4pHG8cz5na\nHIp7NMwIn2hrSf1ybT8zmSwc/byYk4dLsVoFJk6PYP69CTcsAN7/Sd0I+vPvadizl4p336dh00YM\n5zOJ/M7TfHTxE6yClScmrCHA/+s9R77AwwvVvLXzAgez6/jWCsd+aYBNe87S3Wvmu/ePIz7G19Zy\nBsxIGc9aT56i/PU3MGm1uERGEvv/foxLeJitZdmEB7zv4VTjWU7UnWF50lyivOx7T7699bFbjpga\njeYN4I0rj6nV6rfp887lfpUMRXStV+4rlgGvXXGuM1c8tx34n1tdX6u1v5TLM2Ys4LnnfsnatY+Q\nkjIeq1WgubkLi8VKS4uO3l4THR09NDd3odcb6erqve7xJRYsuJc//vEPSKVSRCIRv/w59cBaAAAg\nAElEQVTls1c972B4CJSEMMF/POcac/gk93Nmh0yztSR8fd1s1hfqdA3sv3gUPycfJqomXqdDDiyb\nGs62Y+W8/nEujy5S20SnvWAwWnhrRwFSiZgVU8Nt9r3NXBRLSKSKo/uKObirkLysGuYujcfL98Ze\nOlv2MweDx9LQRWTXF7Ap80MCpcG4y+1rsnFtP6up0HJkr4bO9l7c3BXMWaomJMKLjs6eAZ1fOmkm\n4ZHxNLzzJtqzmbSez0M+XsGEqVMIk0Vc18enxPuy45iSHcfKmBDrQ9AY92RnFTdzPLeOmBAPJsT5\njNgxYSSMZ+Z2LU3v/wtddiYiqRTv+x7Aa/FS9FIpejvXPpQ8ELWcV3I28vrpD/hZxvcQi+xjUf1a\nbNnHbmREigay50WtVq8F5mk0mqfUavUDwAMajeaRftoVAxkajabrq7/XA1s1Gs0xtVq9FPiBRqO5\nafXO5uYuu9wAMhIGDAd3Rqexi/8+9X8IgsBvpvwHnoqbh/kMNbbqY4Ig8GrOP9FoL/K9lCdI9kno\nt53ZYuX5t85S19LNM4+kExsy+rJt3S7bjpax40QFy6aF20V4Um+PieMHLlJc0IhYIiJtShjpU8P6\njXJwjGWjh8PVX7K1ZDupvsk8nfyoXYVbXupnvT0mThy8iCa/EZEIUiaGMHFGJDL54ETgCIJAzYGd\ndHzyCXKTgCI+nqBvfguZz/WepuziZl79JI/IQDd+/WgGEjuJyBhu9L0m/uuN03T3mHjhyUkEeo9c\nw9aexzPBaqXj2BFatm7G2tODU5wa/8e+iTxgaPdXjyTeyHuP7OY8Hkt4iMmB/eeksDU2Nub6HdQH\nOnJ9BEjUavWXwA+AZwDUavWv1Gr11CvaeV4y5L7iDeB/1Gr1EeA/gZ8M8PoOHAw67nI3VkYvodfS\ny8clO2wtx2bkthSg0V4k0Vt9Q0MOQCoR8/jiPo/cO3s1mC2jJsfRHVHZ0MXuU5V4usrtplyD0knG\n/OUJLFmVjJOzjMzjlXz0xlmqytpsLc3BEDI7ZBoxnpHkNueT2ZhjazlXIQgCxQWNfLDxDJr8Rnz8\nXXnw8QymzYsZNEMOQEBgs1c17y31whwXgaGoiIrf/hetO7djNZmuapsW58vUJH/K67vYfXLkJo+5\nWzYfvkiHzsjy6ZEj2pCzZ4wN9dT8759oeu8dAPwe/SYhP/+lw5C7hvtjliETy/i0dDc95l5byxkx\nDCgBikajsQBP9HP8T9f87XfN33mA7ePXHDi4AdODJnG6PpOspvNMadWQ5D22wge7jDo+0mxDIpKw\nKmb5LdvHhngyJy2YL7Jr2XOqkuXTI4dBpf1gNFnYuKMAi1XgyaUJKOWDX4rgboiI8SEo1JNzX1Zw\n/lwNuzafJzrel2nzY3B1U9hanoNBRiwS80j8Gv5w5q9sLv6MWFUMHgrbh1t2tvewf1sBpZpmpLK+\n0hopE4OHZG/ykZoTlHVUMD5iPAnLHqHr9EmaN39I66ef0HnyBH7rHsUlMely+7UL4yiqamf78QpS\non0ID7D95zWcFFa0cTS3nhBfV5ZMHpv7tYYSq8mIdu8e2nbtQDCbcU3PwG/tI0g9b14bcazi7aTi\nnvA57Cr/nN3ln/Ng7K3nIQ5A8vzzz9taw03R643P21pDf7i4KNDr+9sm6GAkIxKJCHcP5XjdGUrb\nK5geNMlmRSyHu48JgsBbFz6guquWldFLSPFNuvWLgNgQD47nN5BfrmVigh+uYygZykeHLnK+tJX5\n6SEsmDC8SU9uF4lUTGiUF5FxPrQ06agu11KYW49EKsYv0A1XV6VjLBtFuMicUUqU5LTk09LTSrpf\nqs3CLU0mC5knKjmwo5DW5m5CI1XcuzqF8BjvIdHU0tPKG3nvoZQq+X7qkyilChQhoXjMnIVgMKIv\nyKPr5HGM9XU4xcQgVjohl0oI9nHhRH4DF2s7mJkShERsP+GpQ4nBZOGlLbn0GMz8ZFUK3qOgbqi9\nzM0EQaA7J4u6V19Bl52JxM2dgCefwmfl/YiVTraWZ9eEu4eR2ZTLhVYNMZ6ReDvZV+F6W/YxFxfF\nC/0ddxhzA8ReBgwHg4+73A2DxUBBaxECAvFesTbRMdx97Mu6UxyqPkacKoaH1fff9mRLJpXg467k\ndGEjtc06piUH2NVenaGioLyN9z8vJtDbme/dn2z3xYedXeTEpwTg6q6ktlJLRUkrFSUtBIZ4IB3E\nMDcHtifMPYSL7WVcaCvGz9mXYNfhDeUSBIHSomb2fpxP5cVWnJxlLFudQtrUMJRDtNhjFay8kfcv\nmnpaWBe/iiiPr0OexTI5LuNScEkdj6G6Cn1BPh1HjyCSylBGROLv7UJHt5G80lYsVitJEfY1eRwq\nPjlSxvnSVhZPDmPGENfFHC7sYW5mrK+j4Y2NtO3aidVgQHXPIgK/+wOUYfYRhm/vSMQSItzDONVw\njsLWYiYHZCCXyG0t6zIOY24AOIw5B7YgyiOCs43ZXGjTMN43GTf59YVrh5rh7GMN3Y38M+89lBIF\nP0p7Gifpna0cBno7U9WoI7+8DW8PJeGjvNisrsfEXz/KwWS28v/WjMfbY2SsaItEInwD3IhPCaC3\nx0R1mZacM9V0tOnxDXBDobSvMFEHA0MkEhHjGcmJujMUtZUwKSADpXR4wmpbGnUc+OwCuWdqMJut\njJ8cxj0rE4lR+w/pePZl3SmO1J5gnE8CK6IW97ugJPX0xH3GTKQqFfqiQrpzstBlZyEPCCA5PZYz\nhY2cL20lKcILr1HgpboZ5fWdvLWnED9PJ757n/0vRt0utpybWXp6aN32MQ1vvYGpsRHnpGSCf/gT\n3CdPRSwbOxErg4FK6YFUJCG3pYBGfRMZfuPtZpHYYcwNAIcx58AWSMUSfJ28OduYTVlHBZMDMoY9\n3HK4+pjJauZvuZtoN3TwzcRvEOFx5/smRCIRsSEeHMmto7BCy4xxgShGqbdHEAQ27SqkrK6T+2dG\nMSnB39aS7hiZTEJkrA8hESo62nqoLG2jIKcOk8mCX6AbEunomNiNZZxlziilSnKa82nuaSVjiMMt\ne3tMnDhUypG9Gro6DETEeLNkVTIxCX5IpOIhHc/aerVszHsHuUTG91O/hZP0xoaYSCRCGR6Bx4xZ\nWLq70Rfk0XnyOOaqCpKnjuN4qQ5NdTszU4JGjYFzLWaLlfVbcunsNvH9+8cR4OVsa0mDhi3mZoLV\nSueJ49RtWI/+QgEybx8CnngK7/seQOrmPqxaRhORHuGUdVRwoa0YV7krEe72sZXBYcwNAHs35vR6\nPd/4xgOsWbOW5557hmnTZiCV3tnq9iuv/IXXX/87e/bsJD19Am6Of367wN/Zl05jFwWtRWgN7aT6\nJA3rytBwDRifXdxNbksB0wIncU/E3AGfx0khRSGTkFXcQnVTF5MT/RHbyUraYHKqoJGdJyqICfHg\niSUJdrNaOBDc3JXMmBuLVCGmsbaDqtI2is7XI5NL8fF3GdHvzQGEufWFWxa2FePr7DMk4ZYWs5X8\nzFr2bSugvqYDT29n5i9PYML0iKtCKodqPBMEgTcL3qdB38TDcfcTq4q6rdeJFQpcx6fhkjoeY1Mj\n+gsFkHmCeC8xmTonukwiUqK9B12vPbDrZAVnCpuYlRrEQjvd6ztQhnui3V2QT/3rf6fjyGEQBLyX\nrSDg6e+gCAlxjJ93iUgkQu0Vw5mGLPJaC0nxSbSL+pkOY24A2LsxZzKZ2LZtK2vWrGXu3AV3bMid\nPPkl586d4dVXX0el8qKkREN8fOIQqXZwp8R7xVLUVkJBaxFuclfCh3FlaDgGjMK2Yj4q3oafsw/f\nTnkc6V16HyMD3alq1JFX1obJbCUpcnTtPWnp6OGVj88jkYj5+UPjR0WyFxdXBU6uchLTgpBKxdRW\ntVNe3EKZphl3Tyc8VI7N+iOVy+GW9WcpauvbezJY4ZZWq4Amr4F92wooLWpGIhExeXYUc++NR+V9\nvadnqMazU/XnOFR9jASvOB6IWXbHE2ippyfuU6ejjIjEUFmJoqqE9M4Syus6UEZE4Os9/CH2Q8nF\nmg7e2lOIm4ucHz84Dlk/tSdHMsM10e6tKKdh0z9p2/EZlo4O3CZNIegHP8Z1fBoiyej6TG2JUqok\nwMWPMw1ZlGjLmBo4wWZJ6S5hj8acY4PEAOju1vGLX/wYnU5PSsr4y8dXrVrOu+9+xEsv/RmVSoVG\nU0R7u5Z16x5n164ddHS0s2HDRlxdv745HD9+jIULlwAwffrMYX8vDm6OTCzlqeRH+NPZ9Wwt2UGI\nW/BVG+tHMjpjN+9d+AiJSMITiWtRDMIGY7FIxFPLEvndu+fYe7qKcH83JieOvDDE/rBaBTbtLKTH\nYOGJpfH4eo4uI0cmk5AxPYL41EDOHC2n6HwDuzafJzjck0kzIwkI8bC1RAcDwMfJm/uil7K5+FP+\nXfQx30355l15DARBoLy4hTNHy9G26pFIRKROCiFtShhOzsObpKBWV8+Wks9QShSsjX9wwO9LJBLh\nmjoel+RxdBw7QtO2T5jVloPuryU0r16Fz6yZiO5wodYeae3oZcMn57Fa4elliTgrR/5i1HBjbGig\n5dOP0Z07C4BzUjI+D652JDcZQsb5JDInZDpf1Bzn45IdfCP+QVtLsjtG/OjUvOVDur76pxos3CZM\nxHf1wzd8ft++PcTGxvL00z/i4MH9HDiw77o2EomU9etf44UXniUv7zzr1/+dF1/8DVlZ55g1a87l\ndvX19Tg7F7J9+ycoFAp+9rNfEuAoImlXqJSePJm0jldz/skbee/xq0k/sQtX/90gCALvF22lw9jF\nfdFLCXMPGbRzOyul/OjBcbz4zjne2l1IoLczYaMgIcq+s1VoqttJj/MdNZnf+sPFVcHcpfGMywjh\n1BelVJdr2VaZTWikiokzI/EPcoSBjzRmBk8hpzmf/NZCdpbvZ3nUogGdp6ZCy+kjZTTVdyESQUJq\nIBOmh+Nqg2QhHYYuXst9C4PFyJNJ6/BS3n3dLpFEguecebhNnsqpje/jmX8S7b/fQff5HryWLcd9\nyrQR63XpNZp55ePzdOpNrFsYR+IYydg5WJjb22nd8Skdx46C1YoyMgqfB1fjHJ9ga2ljgvuil1LS\nXsaXdaeJ94ojzW+crSXZFaNzd+8QU1FRRlpaGgBpaRn9tklI6KvR5e3tQ1xcX+Fplcqb7m7dVe0E\nQcDNzZ31619j/vx72LDh5SFU7mCgqL1iWBm9hA5jJ2/mv4/FarG1pLviy7rTnG8pIM4zmvlhswb9\n/IHeLjy9PBGj2cqGT/LQ9ZgG/RrDSWVDF58cKcPDRc7ji9VjYi+Ej78ryx5KZeW68QSFeVJdruWT\nd7PYveU8zQ1dtpbn4A4Qi8Q8mbQWH6UXeysOcqLuzhZA66vb2fFhLjs+zKWpvovoeF8eemoSc5ao\nbWLIGS1GXs97G62hneVRi8jwTx3U80ucnJj8wyfZPeERMj3UGFpbaXxrExXPPkPH8WMIlpE1/luF\nvqiC6iYdc9KCmZcebGtJIwZTWxtNH7xP+a9/QceRL5D5+hH4vR8Q+uvfOAy5YUQmkfFk0lpkYhnv\nF22lrVdra0l2xYj3zPmufvimXrShQBBALO6zg61Wod82kitW7658LAhXt/fy8iItLR2ASZOm8t57\nbw22XAeDxIKw2VR0VpHTnM9nZXt4IGaZrSUNiFpdPR+X7MBZ6sRjiQ8hFg3Nmk5arC8rpkew/XgF\nr32az88eSkUiHnnrR01aPS9vycViFXhiaQJuwxxKZmuCQj1ZuXY8tZVazn5ZQWVpG5WlbUTEeDNx\nZgQ+o8DrOhZwk7vy/fHf4i/n/sYHmo9RKT1I8Iq7YXtBEKgqayP7ZBX1NR0AhEaqmDw7Ct8A233n\nVsHKuxc+orKzmskBGSwKnzck15FKxDy1dhp/eE/BqbZknnCvxeXCORrf2kTbzh0jylO3/ctyMoub\niQ/zZO2C2DGxGHW3GBsbadu7i84Tx8FiQerlhde9K/CYMXNEfOejkQAXf1bHreDfRR/zdsGH/CTt\n2zbfP2cvjLyZlR0QFhZOfn4+AFlZ5+7qXFOmTOP06ZMAaDSFhIY64q7tFZFIxCMJa/B39uVg1VGy\nms7bWtId09jdxKvZ/8RkNbEuYTUqpeeQXm/FjEjGx/hQWKll6xelQ3qtoaBdZ+AvH+XQ0W3kGwti\nR212u9shOFzFyrXjWf5wKgHB7lRcbGXLW5ns2nKemgrtdQtVDuwPf2dfvp3yOGJEvJH3L2p19de1\nsVoFLhY2sfWtTHZvyaO+poPwaC/ueySNZQ+l2tSQA9hZtp/s5jyiPSL5xl3sk7sdVG4KfvZQKlZX\nDzb0qOn5zq/wmDsPs7btsqeu/cgXWI32W6boTGEj249X4Oup5Pv3jxu15RYGC0NtDfX//AcVz/6K\nzmNHkfn44P/NbxH5hz/jOXuOw5CzMdMCJ5Hml0JpRzlbS7Y77jtfMeI9c7Zg8eJ7ee65X5KZ+T1S\nUu6ukOHcuQv4y1/+xPe+9yQSiZRf/OK/BlGpg8HGSark6XGP8edzr/Kvws0EufgT4DIyEny09LTx\nSs4/6TLpeCjuPsb7Jg/5NcUiEU8vT+TFd86x70w14f5uTEkKGPLrDgbdvX2FwZvbe1kxPWLUpfAe\nCCKRiJAIFcHhntRUaDl3vJKq0jaqStvw8XMldVII0Ql+SBwTRrslxjOSRxPW8NaFD3gt9y1+PuEH\neCo8sJitaAoayDlVTYe2B5EIYhL8SJsSho+/fWR0PFl/jn2Vh/B18ubbKY8hEw/9FCbQ24WfrErl\nfz/M5u+Havjl2vuIWHIvbXt20XnsKE3vvU3rto/xmDMXz7nzkHoM7QLZnVBe38mmXYUo5RJ+/GDK\nqMi+O1T0lJXStnsn3TnZAMhDQvFeugzXCRMRjcCIktGKSCRirfpBmvTNHK09iUQs4cGY5WPe2yyy\nd6u2ubnLLgX6+rrR3OzYNzJWyWzM5c2C9/F39uPnGd/HWTb4RVcHs4+1Gzp4KfM1WnrbuD/mXhaE\nzR6U894u9a3dvPjOOaxWgWceySDcxqv7t8JgtPCXj3K4WNvB/PQQ1i4cvaFJd9vPGus6yT1TTZmm\nGUEAFzc54yaEkJgaiMKRLc9u2VdxiO1lewmThTHLupjivCb0OiNiiYj4cQGMnxyKh2rwxrW77Wcl\n2lJezXkDuUTOf2b8AH8Xv0HTdjtkFTfzt215uDrJ+PWjGfirnDG3a2k/dJD2I4exdncjkkpxmzwV\n1cJ7UITYdvFH22XgxXfO0qEz8uNVKaTG+NhUz3BxJ/3MajKiO3sW7aEDGCrKAVBGReG1dDkuqXe3\nUO9gaOky6ng5+3UauhtZGDaHldFLhu37suX839fXrd836agzN0BsWWfCge0Jcg3AYDaQ13qBgtYi\nUn2TBq1+0yUGq491GXWsz95Ic08LSyMWsDhi/iCouzPcnOUE+bhwsqCR/LJW0mJ9cbHTib7ZYmXD\ntjyKqtqZkujP40viR2Xx80vcbT9zdVMQHe9HXHKfh7qhtpOq0jbys+rQdxtxdVcOe8p6BzdHEASU\nnSp0eQqkFwJorOmbmIzLCOaelUnEJvlfVfB7MLibftakb2ZDzhuYBDPfT31iWOt9XiLQ2wV3Fzln\ni5rIK21lUoI/Tu6uOCck4jlvAVKVF8aGenoKL9DxxWF6SkoQu7oi8/UbdqPAaLLw0uZcGtr0rJkb\nw4yU0Zt991pup5+Z2lrR7tlNw6aNdJ0+iaWjA5fxafivewzv+x9EERjoMOTsHIVEznjfceS3FnK+\n5QJWBNSqmGG5tj3WmXMYcwPEYcw5UHvF0G3Sk99aSE5zHkne8bjIXAbt/IPRx/QmPa/kbKS+u5H5\nobNYEb3YZjepQG8XpBIRmcUtnLnQiDpMhcptcA3gu8VqFfjnzgtkl7QwLsqb765MGpFJW+6EwRrL\nFEoZYVHeJKcHoVDKaG3UUVPRTkFWHTXlbYhE4KFydoRg2hCjwUxhbj2HdxeRe6YGa5cU3IzUBRUS\nPE3KkknTkCuGJnRxoP2s26TnleyNtBs7WBu/ivE2TEkeGeiOxSqQXdJCUZWWyYn+SCViRFIpyshI\nPOfOQxkRibmzg56iC3SdPkXn8S+x6PXIfHyQOA/e/eFGGIwWXt9eQFGVlmnJAayeGz2mDJMb9TNB\nEOjRFNG85SOa/vUuPcUaRDIZnvMXEvjUt/GcMw+Zr++Y+qxGOkqpglTfZPJaLnC+pQAxImJVUUN+\nXYcxNwAcxpwDe0UkEpHk3ZemPrelgKzG88R5ReOhGJw6XHfbx3rNvWzI3UR1Vy0zgqewOnaFzW9U\ncaGeuDrJyNQ0c+pCI2H+rvh7DX6I6kAQBIH3Py/meF4DMSEe/GRVCnLp6N/sPthjmVQqITDEg+SM\nYLz9XDAazNRVdVBR0kpeZi1dHb04u8hxdpXbvD+OBQRBoL66g8wTlRzeraG8pAVjr5noBD9mLYpj\nyuwoMg1nKNAWAhDrGTUk38tA+lmzvpUNuW/QqG9mYdgcFobPGXRdd0p8mCetnb3klbVR1ahjYrwf\nYnHf5yUSiZAHBOAxbQYu49MQrFZ6KyrouVBA+8ED9JSUgFSCzM9vSBJpaLv6EjZpqttJCFf1LUaN\nscWTa/uZsamJ9oOf0/ju27R/vg9jfR2KkFB8HniQgCeewnVcyrAY2Q6GBqVUSapvMrnNBeS25CMT\nSYn2jBzSa9qjMefYMzdAHHvmHFzJ0ZqTbC7+FIVEzndSHiduENz9d9PHjBYjf899k5L2MiYFpPNo\nwpohK0EwELKKm3l9ewEWi8Cji+KYPd62dY8EQWDbsXJ2nqggxNeVX61Lw9lOw0AHm+EYyzrbeyjK\na6DofAPdXQYAfPxciRvnT7Ta1ya1ykY7rc06SgoaKbnQhK6z7zN3dVeQOD6IhNRAnF2+Dn3V9rbz\nl8y/ozW0k+qbzKMJa3CSDu53cqf97EKrhjcL/k2PuYc5IdN5MHa53YxhZouVVz/OI6+slXFR3jy1\n7MYlS6y9vXSdO0vn8WP0lBQDIHZ2wX3KFNynzUQRHj4oxnNFQyevbD1Pu87IzJRAHl2kHpOZK319\n3Wgor6fr3Bk6T56gt/QiACK5HNe0jD7vaXSMYyFplNHa08ZLWf9Aa2jngZhlQ1I/9xL2uGfOYcwN\nEIcx5+BaMhtzeefCh4iAJ5LW3nU40ED7WJdRx9sFH1CkLSHNdxxPJK21y1osF2s7eGXreXQ9JpZP\ni+C+mZE2ucHqeky8u0/DuaIm/DydeOaRdDxc7Sv8cygZzrHMahWoLm+jMKeeiostXLr9BAS7ExXv\n6zDs7hJdZy8lhU2UFDTS2tQNgFwhIUrtS2yiP0Fhnpe9SNfSaezizfz3KWkvw8/Zh6eTHyPIdfAy\nz95uPxMEgQNVR/isdA8SsYSH1Q8wNXDCoOkYLHqNZv62LZ+C8jZUbgq+syKJuNCbZ7I0NtTT8eUx\nOk98iaWzEwCZjy+uGRm4pk9AGRk1oMyJ54qaeGPnBUxmK6vnxrBoUuiYM1asBgPdBfkYss6gPZeJ\nYDaDSIRzfAJuU6bhlpGBWOlka5kOhpBmfSsvZ/+DdkMHq2JXMDd0xpBcx2HMDQCHMedgJFHUVsLr\nee9gspj4hvoBpgdPHvC5BtLHspvy+FDzCTpTN8neCTw97lGkw5C+e6A0tOl5aXNf+v/pyQE8viR+\nWFeTCyra2LTzAu06IzEhHnxneRLeHmPLmLDVWKbvNlKmaaa0qJn66vbLhp1/sDvRal+i4x2G3a0Q\nBIGWRh2Vpa1UlrbSVNf3PYrFIsKivYhL8ic82hup7PYWcyxWC5+V7eFg1VHkYhnrElYzwX/8oGi9\nnX5mtBh5v2gr5xpz8JC78+2Ux4hwDxuU6w8FVkFg98lKth0rQ4SI+2dFsmRK+C0TJglmM935eXSd\nOU33+Rysvb0ASFVeuKZn4JoxAaeY2FsadoIgsPNkJduOlqGQS/jO8iTGx46NrJUA5vZ2dOdz6M7J\nRl94AcFkAkAeHIL7lGm4TZ6CzMvLxiodDCeN+mZezvoHncYuMvxSWR23Ejf54JZWcRhzA8DejTm9\nXs9jjz3E1q07eO65Z/j1r59Dobj9CchLL/2Z0q/CAAyGXlxd3Xjppb8NlWwHw0BlZzV/z30Tnamb\nJRHzWRQ+D5nkzkP27mTA0Bm72Vz8KZlNucjEUpZHLWZu6Ay7CUu6GZ3dRtZvzaW8voukCBXfv38c\nTkOUhOESJrOFj4+Usf9sNRKxiJUzIlk6JfyGXovRjD0sTN3IsPP2dSEk0ovQSBWBIR63bZSMZkwm\nC7UV2ssGXHdX394NkQgCQz2JSfAlOt7vrrJRZjWd51+FmzFYjMwNncH90ffetXf/Vv2stUfLxrx3\nqNHVEeURzlPJj+GhsO8SJpfQVGl5fXsB7TojyZFePLUsEXeX28vgajUZ0RcUoMs6hy4nG6teD4DE\n3R2Xcak4JybiHJ9wXf06k9nC23uKOFnQiLe7gh+vSiXUzz7qAQ4VgiBgrKlBl5uNLif7cjkBAHlQ\nMC6p4wm7Zy56V+8x55l08DVN+mbeufARFZ1VuEideTB2OZMC0getTziMuQEwkoy5u+XNNzcSERHF\nvHkLBkGhA1vS0N3Ehpw30BraUSk8uTdyIZMC0u9oQnS7A0ZOUx4farbRZdIR6R7Oowmrh70G091i\nMFr4x2f55Ja24uOhZNm0CKYlBwyJl66mScfGHQXUNHfj7+XMt5cnEhk4OElrRiL2YMxdib7bSHlx\nM2WaFuqr27FY+m4BEomIwFBPQiJUhEaq8PZzHRMTNrPZQlN9F/XVHdRXt1NX3YHFbAVAoZQSHu1N\neIw3oZGqQa3r19DdxD/z3qVB30S0RwTfSn7krpI73ayfFWsvsin/fXSmbqYHTWZN3Eq7jijoj069\nkTd2XiC/rA0PVznfXZGEOkx1R+cQzGb0RYV9hl1WFhbd15+XPDgE54REnBMT6TUi0lQAABIYSURB\nVPQJ5e2DFVys7SA6yJ0fPjBuVIaGC4KAsb6OnmINPcXF9JRoMGu1fU+KxTjFqXFNHY9Lahpyv757\nnr2NZw5sg1WwcqTmBNvL9mK0GEnwiuMb6gfwdrp7T63DmBsA9mjMdXfreOGFX6PT6UlJGc+BA/vY\nunUHq1Yt5913P+Kll/6MSqVCoymivV3LunWPs2vXDjo62tmwYSOurtevnnV2dvLMM//Bhg0bx8QE\nZSygM3Wzv/IwR2pOYLaaCXD2Y3nUIlJ9k2/rO77VgKEzdbNZ0+eNk4qlLI9axLzQmSPCG9cfFquV\nT46U8fm5aswWAW93JfdODWdGSuCgGHVWQeDA2Wq2HinFbBGYmxbMmrkxKORj29tjz5Mfs8lCfU0H\n1eVaairaLu8DA1A6SfELcscvwK3vd6DbqKhn19tjoqGmg/qaDhpqOmhq6MJq+fo26OXrctmA8w9y\nH1Jvcq/ZwPtFW8hqOo+73I17wucyKSAdF9mdZ6C9tp8JgoBGe5FjtSfJbS5AJBKxJu4+ZgZPGcy3\nMKxYBYF9p6v4+EgZAgLLp0WwaFLYgCINBKsVQ001+gsF6Asv0FNSjGDs88JaEFGv9MEaFE7G3HRc\noqL66tmN8DIqgtmMobbmCuOt+CqDVuLmhnNCIi6pabgkj0Picn0WSnsezxwMP609bXyg+YTCtmLk\nYhnLoxczJ2T6Xc2TRpUxp1arZwNbgCc1Gs3Ofp5fB/wUsAIbNRrNJrVaLQPeBsIBC/CERqMpu9l1\nbmXMnThUSllR04Dew42Iivdj2rzoGz7/ySdbaG1t4Omnf8TBg/t57bVXrzPmfHx8+c53fsALLzyL\nq6sb//Efv+TFF3/D7NnzmTVrznXnfP/9d/D0VHHvvSsG9b04sD3a3nZ2lx/gZP1ZBATC3UNZGbUE\ntdfNM172N2AIgkBTTwsXWjXsqzxEl1FHpHsYjySsIWCEeeNuhLbLwJ5TlRzJrcNktuLlrmDplHBm\npgQhk97ZACwIArUt3WSXtJBZ1ERVkw53ZxlPLE0gNWbs7C25GSNp8qPvNlJToaW6vI36qna6vsrS\neAk3DyV+gW74Bbrh4++Kh8oZV3eFXS6QWSxWOrQ9tDV3f/3T0k2HtudyG5EIfPzdCAzxICDEg8BQ\nj6uyUA4HgiBwuPoYn5buwSJYkIqlpPokMS1oEnGq6NueFF2OZjH1cLohk2O1J2nUNwMQ4hrE6riV\nxAxxSvHh4mJNB//Ynk9bpwGZVExqjA9TE/1JjvK+4zHMYrVyrqiZz0+VY64oI7ynnnhzE166JkRX\nzN/ETk4owiNQhkegjIhEERqKzMcXkdT+PJyCIGDp7MBQU4OhphpDTTXGmmqM9fV9iUu+QurljVNc\nHE5xapzj1Mj8A275vzySxjMHw4MgCJxtzGZryXa6TXrC3UO5P3opkR7hA4oAsEdjbkD/5Wq1Ohr4\nGXD8Bs+7AL8FJgFG4Kxard4GLAfaNRrNOrVafQ/wR+ChgWiwJRUVZcyaNR2AtLSMftskJCQB4O3t\nQ3h4BAAqlTfd3bp+23/++T5ef/3NwRfrwOaolJ6sS1jFgrBZ7CjfT3bTeV7J2Ui8KpYM//H4OKnw\nVnqjUnr0OzHqNunRaC9S2FpMkbaEtt6+MBOpWMp90UuZHzZrxHrj+kPlpmDtwjiWTg1n7+kqvsiu\n5V/7i9l1spLFk8OID1Ph4SLH1VnWb6IBi9XKxZoOsktayC5pprm9L7mAWCRiQrwfjyyMu+39LA7s\nC2cXOXFJ/sQl+QN9xl1zfRdN9Z001XfRVN9FaVHf3rtLSCQi3FVOeKqc8fBywsOr77GzqxwnZxly\nhXRIjD1BEOjRm+juMqDrNPT91hnoau+lraWb9lY9VuvVa5UKpZTgcE8CvzLc/IPckcltOxkXiUTM\nC5vFxIB0TjdkcrLuLJlNuWQ25eKtVDE1cCJTAiegUt48k2OFtprPig5wtiEbo9WEVCRhon86s0Km\nEukeZpcG90CJCfHg+ScmcSirhlMFjZwrauJcURPOCikT4n2ZkhhAXJjnTROl9BjMHMut4/NzNbR2\n9iIC0saPY8akpcQEeyAYjRiqquitLKe3ohxDRQU9miJ6igq/PolIhNTbG7mvPzI/P2R+fsj9/JD5\n+iH1VCF2dh4yb57VYMDc1oqpre2K322YWlsw1tZg6bp6MiySy1GEhqEIDcMpNg6nuDhk3o4FNwd3\nj0gkYlJAOglecWwt2c65xhxezn4duVhGpEc4sZ5RxHhGEeEeOqD8BvbAQO8S9cADwKYbPD8ZOKvR\naDoA1Gr1cWA6MB9496s2B4C7tl6mzYu+qRdtKBAEEH81AF57M76E5IqCoFc+7s8TWl1dhYeH5x0l\nTnEw8vB38eOp5Eeo6qxhe9leCtv6jLNLiEVivBSeeDt54a30wqvWnbx6DVWdNQj09RtnqRNpfikk\neMWS5B2Pp8LDVm9nyPF0VfDw/FiWTAln3+kqDmXX8MGBKz8vEe4uMjxcFHi4ynF3kWO2WMkva0PX\n05fVTCGXMCHej7RYH1KivXEZI7XjxgrOLnLCY/pCDqFvfO3q6KWpvuuyl6ujTU+Htgdti77fc4jF\nIpTOMpycZTg5y1E6y1AqpYglYsRiEWKJCLFI9PXfYhGCIGAyWjCZrJhMFsxGCyaTpe+Y0YK+20h3\nl+GG9wepTIyPvytevi54+brg7euCl48LTi72W0jdTe7KgrDZzA+dRXlnJSe+Mup2lu9nV/nnhLoF\nIxGJsQoCVsGCFQGLYEUQrJis5suLUF5KFTODpjA1aOKgZ5mzJ1ydZKyYHsnyaRFUNeo4daGB0xca\nOZpbz9HcelRuCqIC3TGYLfQaLfQaLBhMZnqNFgxGC8av9kXKpWLmpQezcGIo/qqvw1tFCgVOsbE4\nxcZePmbp6cFQVUlvRTnGujpMTY0Ym5rQFxZAYcH1IkUixC4uSFxdkbi4InFz6/vt4gISSZ+hJxZf\n9xurFWtv7xU/PV8/7unB3NmBVdf/wjWAzNcPZUwsipBQFCEhKEJCR0WYqAP7xk3uyhNJa5kWOImc\n5jwutpej0V5Eo+1LQigVS4l0DyPGM4q5oTMGFE5uKwZkzGk0Gj2AWq2+UZMAoPmKv5uAwCuPazQa\nq1qtFtRqtVyj0dywlLpK5YxUal97WhIT4/j/7d1/bNx1Hcfx5931x91Kf6xrywZjMlp8M9g6fkiA\nbOAGmDFHQgIjKuBPCIRgAhJ/YBQGaoLRGFQ0RoNCUKMmmihG4wgoiuDI1EQywTcMxhDWrb/W7Xrt\n9cfd+cf3u9F2Ldj12uv3+nokl37vc5e7zyWvfu7e3+/n+/nu2rWLTZs2sWPHUyQScZqba0kk4jQ1\nnUAyWUl9fYrm5loWLaqitjZ5zPZYO3a8Snv7Wce0S3lqbl7Fea2reKV3L3v73qAz001nfw+dmR46\nM91HBxY6IBGLc0ZzG2uXrqL9xFWctnjF0R0JC0VzM7SduoQbtpzJkzv/y4HeDAfTQ/SlhziYztLR\nO8DeA2/t5W2sS7L57JO5YPVS2tuaqJxn48d8VE5jT0tLHa2nj59yXCgUyPQP09vVT09Xht6eDJnD\nQ2Qywwz0DzGQGSZ9aGjcOXnHKx6PUVNbzbLl9dQ1pKirT1Jbn6KuIUldfZK6hhT1DSliEV45taVl\nDRe0rWFwJMuzr/+dP+55lld69xKLxUjE4sRj8aN/j9zOWXYW72u9hHOXrV5wY1hLSx3vWXMSt+YL\n/PvVbv78zzd55vl9/OOl4GdSLAap6gqSVRXU1VSTWlLBouoK2k9vYvNFK6cxk6AWVrTA+vPHteay\nWbL7D5Dt2M9gRwfZ/fsZ6etjNN3PyOHDjKbTZDs7IZ+f0eeMJRIkUimqG+qpbmulqqmJ6ubwFm5X\nLVlConp2F2spp/FMiq+5+RzW2zkAHB7q5z9du3mh8yVe6HqZ3X17eLnvVZYubmTzuze+zWvMr4y9\nYzFnZjcBN01o3ubu26fxPlN9a73jt9nBg5PvTS2l9esvZ9u2z3HddTfQ3n42+XyBrq40uVye7u5+\nstkRDh0aDFe7HCadzh6zPdaePW9QXV2jed4LTB2NrKlthAljwnBumJ7sQSoWFajNNZCsCI/Y5qGn\nZ+Y/NqPs4tUnHtNWKBTIDuc4lBkml8uzrKnm6PSlvnk4fsw3C+kck1RtFctrq1h+2uSrDI6O5sgO\njDCUHSWfLwS3XP6t7XyBfK5ALAaVVQkqqxJUVCaorAy2KysTxBOxtz26NpLL0d0z9VGLqGmvW0v7\n2rXv+LwjOVvoY9iy+iQf3NjK1ktWksmOkqxKUFURnzIzQwNDdA0MTfrYtNQ0Qlsj1W1nMlkpVcjn\nyQ8OkuvvJz+QoZDLUcjngwKvUBh/Px4jnkwRTybf+ptKEquonPJzjIQ3Dg8TnH0zOxbSeCbFsbK6\nlZWntLLllM0MjAywL3OAFbXLp8xRic+Zm7R9RqtZmtkjwC8nLoBiZhuAW9z9Q+H9h4FfAVuBn7n7\n9nAxlNfc/eS3e4/5uJolaMCQ2aeMyVxQzmQuKGcyF5QzmW1lswDK/+E54CEzawBGCc6XuwOoA64F\nthMshvKnWXp/ERERERGRsnZcE9fNbIuZPQVcAdxvZo+H7XeZ2UXuPgjcRVC0PQHcFy6G8gsgYWZ/\nBW4DPl+EzyAiIiIiIrLg6KLhx0mH8mW2KWMyF5QzmQvKmcwF5Uxm23ycZrmwlpQSEREREREpEyrm\nREREREREIkjFnIiIiIiISASpmBMREREREYkgFXMiIiIiIiIRpGJOREREREQkglTMiYiIiIiIRNC8\nv86ciIiIiIiIHEtH5kRERERERCJIxZyIiIiIiEgEqZgTERERERGJIBVzIiIiIiIiEaRiTkRERERE\nJIJUzImIiIiIiERQRak7EDVm9gBwIVAAbnf3nSXukpQJM/sacDHB/+X9wE7gx0AC6AA+7O5Dpeuh\nlAszSwG7gC8DT6KcSZGZ2fXAZ4FR4B7geZQzKRIzOwF4FFgMVAP3AfuB7xH8Pnve3W8tXQ8l6sxs\nNfAb4AF3/46ZncIkY1g41t0B5IEfuPsP57qvOjI3DWb2XuB0d78IuBH4dom7JGXCzDYCq8NsXQF8\nE/gS8F13vxjYDXyihF2U8vJFoDfcVs6kqMxsCbANWA9cCVyFcibF9THA3X0jsBX4FsH35u3uvg6o\nN7PNJeyfRJiZ1QAPEuzsPOKYMSx83j3A5cAG4FNm1jjH3VUxN02XAb8GcPcXgcVmVlfaLkmZ+Atw\nbbjdB9QQDAyPhW2/JRgsRGbEzM4AzgR+FzZtQDmT4roceMLd0+7e4e43o5xJcXUDS8LtxQQ7p1aO\nmS2ljMlMDAHvB/aNadvAsWPYBcBOdz/k7oPAM8C6OewnoGJuupYCXWPud4VtIjPi7jl3z4R3bwR+\nD9SMmYbUCSwrSeek3HwDuHPMfeVMiu1UYJGZPWZmT5vZZShnUkTu/nNghZntJtgZ+mng4JinKGNy\n3Nx9NCzOxppsDJtYF5QkdyrmZiZW6g5IeTGzqwiKuU9OeEhZkxkzs48Af3P3PVM8RTmTYogRHDW5\nmmA63MOMz5ZyJjNiZjcAr7t7G3Ap8JMJT1HGZDZNla+S5E7F3PTsY/yRuJMIToIUmTEz2wR8Adjs\n7oeA/nChCoCTGX+4X+R4bAGuMrMdwE3A3ShnUnwHgGfDvduvAGkgrZxJEa0DtgO4+7+AFNA05nFl\nTIptsu/KiXVBSXKnYm56Hic40RYzOxfY5+7p0nZJyoGZ1QNfB6509yMLUzwBXBNuXwP8oRR9k/Lh\n7h9w9/Pd/ULgIYLVLJUzKbbHgUvNLB4uhnICypkU126C85Uws3cR7DB40czWh49fjTImxTXZGPYc\ncL6ZNYQrrK4Dnp7rjsUKhcJcv2ekmdlXgUsIliC9LdwjJDIjZnYzcC/w0pjmjxL84E4Ce4GPu/vI\n3PdOypGZ3Qu8RrB3+1GUMykiM7uFYMo4wFcILrWinElRhD+cfwScSHA5n7sJLk3wfYIDFc+5+51T\nv4LI1MzsPILzy08FRoA3geuBR5gwhpnZVuAzBJfEeNDdfzrX/VUxJyIiIiIiEkGaZikiIiIiIhJB\nKuZEREREREQiSMWciIiIiIhIBKmYExERERERiSAVcyIiIiIiIhGkYk5ERERERCSCVMyJiIiIiIhE\nkIo5ERERERGRCPofbqGZzpTDMCsAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f181c2412e8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6HjtmgH1UgQ5"
      },
      "source": [
        "We also experimented with using learned positional embeddings [(cite)](JonasFaceNet2017) instead, and found that the two versions produced nearly identical results.  We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training.    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZtnFnHH9UgQ6"
      },
      "source": [
        "## Generation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "heaKRIaZUgQ6"
      },
      "source": [
        "class Generator(nn.Module):\n",
        "    \"Standard generation step. (Not described in the paper.)\"\n",
        "    def __init__(self, d_model, vocab):\n",
        "        super(Generator, self).__init__()\n",
        "        self.proj = nn.Linear(d_model, vocab)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return F.log_softmax(self.proj(x), dim=-1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0i97-Y7AUgQ8"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dd3lP9fTUgQ9"
      },
      "source": [
        "## Full Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b-LDhRoaUgQ-"
      },
      "source": [
        "def make_model(src_vocab, tgt_vocab, N=6, d_model=512, d_ff=2048, h=8, dropout=0.1):\n",
        "    \"Construct a model object based on hyperparameters.\"\n",
        "    c = copy.deepcopy\n",
        "    attn = MultiHeadedAttention(h, d_model, dropout)\n",
        "    ff = PositionwiseFeedForward(d_model, d_ff, dropout)\n",
        "    position = PositionalEncoding(d_model, dropout)\n",
        "    model = EncoderDecoder(\n",
        "        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),\n",
        "        Decoder(DecoderLayer(d_model, c(attn), c(attn), c(ff), dropout), N),\n",
        "        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),\n",
        "        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),\n",
        "        Generator(d_model, tgt_vocab))\n",
        "    \n",
        "    # This was important from their code. Initialize parameters with Glorot or fan_avg.\n",
        "    for p in model.parameters():\n",
        "        if p.dim() > 1:\n",
        "            nn.init.xavier_uniform(p)\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qP-g4KfhUgQ_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3023
        },
        "outputId": "e5d671a2-a9d4-461b-f08c-bf5b7a4ddb73"
      },
      "source": [
        "# Small example model.\n",
        "tmp_model = make_model(10, 10, 2)\n",
        "tmp_model"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "EncoderDecoder(\n",
              "  (encoder): Encoder(\n",
              "    (layers): ModuleList(\n",
              "      (0): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (1): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (norm): LayerNorm(\n",
              "    )\n",
              "  )\n",
              "  (decoder): Decoder(\n",
              "    (layers): ModuleList(\n",
              "      (0): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (1): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (norm): LayerNorm(\n",
              "    )\n",
              "  )\n",
              "  (src_embed): Sequential(\n",
              "    (0): Embeddings(\n",
              "      (lut): Embedding(10, 512)\n",
              "    )\n",
              "    (1): PositionalEncoding(\n",
              "      (dropout): Dropout(p=0.1)\n",
              "    )\n",
              "  )\n",
              "  (tgt_embed): Sequential(\n",
              "    (0): Embeddings(\n",
              "      (lut): Embedding(10, 512)\n",
              "    )\n",
              "    (1): PositionalEncoding(\n",
              "      (dropout): Dropout(p=0.1)\n",
              "    )\n",
              "  )\n",
              "  (generator): Generator(\n",
              "    (proj): Linear(in_features=512, out_features=10)\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MuV1e8nEUgRB"
      },
      "source": [
        "# Training\n",
        "\n",
        "This section describes the training regime for our models.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OhVxtlZaUgRC"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "naFpt_GUUgRD"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X60DPvyaUgRF"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lz6n3REAUgRH"
      },
      "source": [
        "## Training Data and Batching\n",
        "\n",
        "We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs.  Sentences were encoded using byte-pair encoding \\citep{DBLP:journals/corr/BritzGLL17}, which has a shared source-target vocabulary of about 37000 tokens. For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary [(cite)](wu2016google). \n",
        "\n",
        "\n",
        "Sentence pairs were batched together by approximate sequence length.  Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens.     "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PRoJURieUgRH"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "52xwbbb4UgRK"
      },
      "source": [
        "## Hardware and Schedule                                                                                                                                                                                                   \n",
        "We trained our models on one machine with 8 NVIDIA P100 GPUs.  For our base models using the hyperparameters described throughout the paper, each training step took about 0.4 seconds.  We trained the base models for a total of 100,000 steps or 12 hours.  For our big models, step time was 1.0 seconds.  The big models were trained for 300,000 steps (3.5 days)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPp__T_uUgRK"
      },
      "source": [
        "## Optimizer\n",
        "\n",
        "We used the Adam optimizer [(cite)](kingma2014adam) with $\\beta_1=0.9$, $\\beta_2=0.98$ and $\\epsilon=10^{-9}$.  We varied the learning rate over the course of training, according to the formula:                                                                                            \n",
        "$$                                                                                                                                                                                                                                                                                         \n",
        "lrate = d_{\\text{model}}^{-0.5} \\cdot                                                                                                                                                                                                                                                                                                \n",
        "  \\min({step\\_num}^{-0.5},                                                                                                                                                                                                                                                                                                  \n",
        "    {step\\_num} \\cdot {warmup\\_steps}^{-1.5})                                                                                                                                                                                                                                                                               \n",
        "$$                                                                                                                                                                                             \n",
        "This corresponds to increasing the learning rate linearly for the first $warmup\\_steps$ training steps, and decreasing it thereafter proportionally to the inverse square root of the step number.  We used $warmup\\_steps=4000$.                            "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q5yV0f2QUgRL"
      },
      "source": [
        "# Note: This part is incredibly important. \n",
        "# Need to train with this setup of the model is very unstable.\n",
        "class NoamOpt:\n",
        "    \"Optim wrapper that implements rate.\"\n",
        "    def __init__(self, model_size, factor, warmup, optimizer):\n",
        "        self.optimizer = optimizer\n",
        "        self._step = 0\n",
        "        self.warmup = warmup\n",
        "        self.factor = factor\n",
        "        self.model_size = model_size\n",
        "        self._rate = 0\n",
        "        \n",
        "    def step(self):\n",
        "        \"Update parameters and rate\"\n",
        "        self._step += 1\n",
        "        rate = self.rate()\n",
        "        for p in self.optimizer.param_groups:\n",
        "            p['lr'] = rate\n",
        "        self._rate = rate\n",
        "        self.optimizer.step()\n",
        "        \n",
        "    def rate(self, step = None):\n",
        "        \"Implement `lrate` above\"\n",
        "        if step is None:\n",
        "            step = self._step\n",
        "        return self.factor * \\\n",
        "            (self.model_size ** (-0.5) *\n",
        "            min(step ** (-0.5), step * self.warmup**(-1.5)))\n",
        "        \n",
        "def get_std_opt(model):\n",
        "    return NoamOpt(model.src_embed[0].d_model, 2, 4000,\n",
        "            torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FM5n1PJxUgRN"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PC2wCVEFUgRO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "outputId": "b3fc5e15-3be4-4e1b-fb3b-fd722207a9a9"
      },
      "source": [
        "# Three settings of the lrate hyperparameters.\n",
        "opts = [NoamOpt(512, 1, 4000, None), \n",
        "        NoamOpt(512, 1, 8000, None),\n",
        "        NoamOpt(256, 1, 4000, None)]\n",
        "plt.plot(np.arange(1, 20000), [[opt.rate(i) for opt in opts] for i in range(1, 20000)])\n",
        "plt.legend([\"512:4000\", \"512:8000\", \"256:4000\"])\n",
        "None"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl4VOXZ+PHvrEkmM9kn+0ZCcoCE\nPewIIuBKra1drL7ua6utVbF1raKivvpatVr9QWtVsOJSt7qggAjITiALEHJIIPu+TjKZLLP9/pgQ\nQQhkmSST5Plcl1fkzHPO3Cch3HOe5X4UTqcTQRAEYXRTDnUAgiAIwtATyUAQBEEQyUAQBEEQyUAQ\nBEFAJANBEAQBUA91AH1RU9Pc5ylQgYE6Ghos7gzHLURcvSPi6j1PjU3E1Tv9ictoNCi6e23UPRmo\n1aqhDuGMRFy9I+LqPU+NTcTVOwMV16hLBoIgCMLpRDIQBEEQRDIQBEEQejiALEnSi8BswAncLcvy\nvpNeWwI8DdiBr2RZfvJs50iS9AfgBSBQlmVz57FrgD8CDmC1LMtvuOf2BEEQhJ4455OBJEkLgSRZ\nlucANwN/+1GTvwFXAvOACyVJmtDdOZIkXQeEAeUnXd8X+AuwBDgfuEeSpKB+3pcgCILQCz3pJloM\nfAogy/IRIFCSJD8ASZISgHpZlktkWXYAX3W27+6cT2RZfhjX08IJs4B9siybZFluBXbgSiyCIAjC\nIOlJN1E4sP+kP9d0Hmvq/Fpz0mvVQCIQcqZzZFk+2s31f3yNiLMFFBio69f0KqPR0OdzB5KIq3dE\nXL3nqbGJuHpnIOLqy6KzbhctnOW1s53T67b9WQhiNBqoqWnu8/kDZaDishzJoa24iMALL0ah6M2P\nYWDj6i8RV+95amwirt7pT1xnSyI9SQbluD69nxAJVHTzWlTnsY6znHOu60cBu3sQl9ADpS88B4Da\n3x+/2XOHOBpBGH4OHEjnL395gPj4BAASE8dyzz1/4sMP3+PVV19k/frv0Ol0AHz77Qbee+8dFAol\n06fP4Pbb7zzjNffs2cV99/2e7dvTAdiwYT0ffLAOhULBT3/6M5YtuwKbzcbKlY9TWVmBSqXiwQf/\nQlRUNLm5uTz88KMoFJCYmMTy5Q+65T57MmawAfgFgCRJ04ByWZabAWRZLgT8JEmKlyRJDSzrbN/t\nOWewB5ghSVKAJEl6XOMF3/f9loQTOip/yL/V772LrblpCKMRhOFrypRpvPrqal59dTX33PMn1q//\ngvr6OkJCjF1t2traeP31V3j55ddZtepN0tP3UlBw/LRrtbe3s3btmwQHhwDQ2trKm2/+g5deeo1X\nX13F+++/S1OTiY0bv0avN/D6629w3XU3sWrV3wFYuXIld999H6+//i/MZjO7du1wyz2eMxnIsrwT\n2C9J0k5cs4LulCTpBkmSftbZ5LfAOlz/gL8vy/LRM50DIEnSw5IkbcH1JLBekqTnOgeNHwC+ATYB\nK2RZNrnl7kY5c2YGAF4xMTjMZmreWzfEEQnCyLBw4SJuv/3OU7pevb29WbPmPXQ6XxQKBf7+/jQ1\nmcjLk3njjVVd7daufZOf//xXaDQaAHJyDjF+fAp6vR4vL28mTpxMdnYW6el7WbDgfADS0mZy8GAW\nVquVsrIyxo9PAWDevPNIT9/rlnvq0ZiBLMsP/OhQ1kmvbQPm9OAcZFleCaw8w/H/AP/pSSxCz5kz\nM0ChIOqPyyl/9WWa9+zCb/YcfCdOGurQBKFPPticz77cardec8a4UH51wdiztiksLODPf76HpqYm\nbrrpVmbMmH3GdjqdLwDHjuVTWVlBSspE1Go1SUkSAMXFReTnH+WWW+7gtddeBqCuro6AgICuawQG\nBlFXV0t9fR0BAYEAKJVKFAoFdXV1+Pn5ndbWHcQK5BHK1tRE27F8fMYmofb3J+z6G0Glomrt2zja\nWoc6PEEYNmJiYrnxxlt59tm/8sgjK3jmmSexWq3dti8pKWbFiod57LGnUKtP/bz9yit/5fe/v/es\n79fdvvRnOu7OPeyHZQlr4dxasjPB6cR3ylQAvKJjCLrkUuq/+JyaD98n7NobhjZAQeiDX10w9pyf\n4t3NaAxl8eILAYiKiiY4OJiammoiI6NOa1tdXcWDDy7n0Uef6HoaOKGmppqiokJWrHgEgLq6Wu66\n6zZuuuk26urqutrV1taQkjKRkBAj9fWu4zabDafTSUhICI2Njae0PXncoj/Ek8EIdWK8QN+ZDACC\nLrscbVQ0pq1bMGdndXeqIAgn2bBhPe++uxags/umHqMx9Ixtn332SZYvfwBJGnfaa0ZjKB988Bmr\nV7/F6tVvERwcwquvriYlJZXc3Byam5uxWCxkZ2cxefJUZsyYzXffbQJgx45tTJuWhlqtJiEhgays\nTAC2bt3MrFmn9dL3iXgyGIEc7e1Ycg6jjYhEG/bDrF2lRkPELbdTvHIFVW+9gc+KlagMnrmoRhA8\nxfz5C3j88UfYvn0rVquV5csf4N1317Bv3x7q6+tYvvwPpKZOZNmyK8jKyuCf//x/XededdU1hIWF\ns23bFm6++fYzXt/Ly5s77riLe++9C4VCwU033Yper2fx4qWkp+/ht7+9Ga1Wy0MPPQbAQw89xIMP\nPozT6WDChFRmzJjllvtUuLPPabD0Z6ezkbiQ5MfMmRmUv/oygZdchvHKX572ev0366n98H30U6cT\n8bu7zroYbTR8v9zJU+MCz41NxNU7/Vx0JnY6G03MmQeAU7uITha49CJ8kiXMGftp2umeOcqCIAxv\nIhmMME6Hg5asTFR+fniPSThjG4VSSfjNt6L09qb63XdOWZwmCMLoJJLBCNN27Bj25mZ8J09Boez+\nx6sJDiH0uhtwtrdR/v9ew9HRMYhRCoLgaUQyGGF+6CKads62fjNn47/wfDpKS6h5792BDk0QBA8m\nksEIY87KQKHVohs/oUftjVddjVdMDKZtW2jas2uAoxMEwVOJZDCCdFRWYK2sxDdlIkqttkfnKDVa\nIu64E4WXN1Vr3qajovzcJwmCMOKIdQYjiDnDtdDMt5tZRN3RhoUTfv2NVKx+nfK/v0LMQ4+i6izJ\nKwijnTtLWGdmHmDVqr+jVqvx8fHhkUeewM/Pj3ffXdO5wMy1zmDOnPmYzWZWrHgYs9mMj4+Oxx9/\nCj8/f3bu3Mlzzz2PUqlizpx53HDDLW65T5EMRhBz5gFQKNBPmtzrcw0zZ9FWWEDDhq+p/OcqIu+6\n+6wD0IIwmkyZMo2nnnqu689nK2G9Zs17+PjouO22G7jwwksYc9KsvldeeZHHHnuS2Nh41qz5F599\n9jGLFy9l06YNrFr1JmazmTvvvIWZM+fwwQfvMnXqdK6++jo+++xj3nnnbX73uz/w1FNP8dxzL2M0\nhnLXXbexcOEFp7xHX4nf9hHCZjLRdvwYPmOT+ryqOOTKX6JLSaUlO4u6Tz92c4SCMHL0tYS1v38A\nJpOrQn9zczMBAQEcOJDO7Nlz0Wg0BAYGEh4eQWFhAfv372PBgkUAzJu3gPT0vZSVleLv709YWDhK\npZI5c+axf/8glrAWPN+PC9P1hUKlIuK231K88gnqv/oCr+gYjJctcWOUgtA/H+d/QUb1Qbdec2ro\nRH4+dtlZ27irhPUf/nAvd911GwaDAYPBj9tvv5N3313TVaoaIDAwkLq62s7S1oGnHKuvryMoKOiU\ntmVlZf26/xPEk8EIYe4sXNWTKaVno/L1JfKuP6Dw8qbyrTdozst3R3iCMGy5s4T1iy8+z9NPP8+6\ndR8zadIUPvnk9G1czlQhqPuy1r27l7MRTwYjQFdhushItGFh/b6eV2QUEbfeTvnf/8aRJ58m+oFH\n0BjdUyZXEPrj52OXnfNTvLu5q4Q1wLFjeUyaNAWAGTNmsWHDeqZPn0FxcVFXm5qaakJCQggJCaG+\nvha9Xt9VqjokxEhtbe1pbd1BPBmMAJacwzg7Ovr9VHAy/ZSpGH9zDVaTibKX/4rdbHbbtQVhOHFX\nCWuA4ODgrn2Rjxw5TExMLNOmzWDXru1YrVZqa2uoqakhPj6BmTNns3mzq4T1li3fMmvWHCIiIjGb\nzVRUlGOz2di5c3u3XVa9JaqWeoj+xFX55hs07fiemAcfwSfRvRt/mL/4mPJP/4tPskTUPctRdu7b\nOtRG4s9xoHlqbJ4el8XSwuOPP4LZ3IzVauWmm27l6FGZffv2kJNziHHjJnSVsL7xxqu79ieG00tY\nHzyYxWuvvYxKpcbPz58HH/wLBoOB//znPTZs+BqFQsGtt/6WtLSZWCwWnnzyUUwmE3q9gb/85Un0\nej2Fhbk888z/ArBw4QVcffW1vbmnbquWimTgIfoal9Ph4Ph9d4NSScLzL7p9OmhIsC/ZK5/DnL4P\nw4yZhN96h0dMOR1pP8fB4Kmxibh6R5SwFs6o7Vg+9uZm9OcoTNdXJyqc+iQl07xvL9Xr3nHrvquC\nIHgGkQyGuRPbW/ZnSum5KDVaIu+6G210DKbvNlP78ekzIARBGN5EMhjmzJkZKLy8elyYrq9Uvr5E\n37McTVg4Deu/pP6rLwb0/QRBGFwiGQxjHRXlWKsq8U1JRanpWWG6/lD7+xN93/2og4Kp/fg/NHTO\ndBAEYfgTyWAYO9FF5M4ppeeiCQom+r77Ufn5UfPuOzRu2Txo7y0IwsARyWAYM2dmgEKB78RJg/q+\n2rBwou/7EyqDgep31ognBEEYAcQK5GGqqzBdUnKfC9P1h1dUNNH3P0Dp//0vNe++AzY7gRdeNOhx\nCMJgeO21l8nKysRut3PttTewffs2ZPkIfn7+AFx99XXMnTufvLyjPPvskwCcd97CbstL79mzi/vu\n+z3bt6cDroVtH3ywDoVCwU9/+jOWLbsCm83GypWPU1lZgUql4sEH/0JUVDS5ubk8/PCjKBSQmJjE\n8uUPuuUeRTIYplqyXIXp9AM4i+hcvCKjiPnTg5Q8/7/UfLAOp91O0CWXDlk8gjAQDhxI5/jxY6xa\n9SYmUyM33nhN514FdzFv3nmntH3uuZX86U8Pk5SUzIoVj9DW1oa3t/cpbdrb21m79k2Cg11lJFpb\nW3nzzX/wj3+sQaNRc8st17FgwSJ27Pgevd7A668/xd69u1m16u888cQzrFy5krvvvo/x41N4/PGH\n2bVrB3PmzOv3fYpuomHKnHViSungjReciTY8gpg/PYA6MIjajz6g9rNPxDoEYUSZPHkqTz7pWvGr\n1xtoa2vD4bCf1q6+vo7W1lYkaRxKpZIVK57G29v7lBLWAGvXvsnPf/4rNJ2r+XNyDjF+fAp6vR4v\nL28mTpxMdnYW6el7WbDgfADS0mZy8GAWVquVsrKyrlXO8+adR3q6KGE9av1QmC4KbeiZa6QMJm1Y\nONF/eoCyF56n/vPPsDc3E3r1/3jESmVhZKn58D2a0/e59ZqGtBkYf3lVt6+rVCp8fHwA+OKLz5gz\nZy5KpYqPPvqA99//N4GBgdxzz5+pqKjAz8+PlSsfp7S0mEWLlvCrX11NUpLUVbSuuLiI/Pyj3HLL\nHbz22ssAnaWqA7reLzAwqKtc9YkS1kqlEoVCQV1dHX5+fqe1dQeRDIYhS84hnFbrkHYR/ZjWGErM\nAw9T9vILmLZsxt7cRPgtt3tMLSNB6K/vv9/CF198xosv/p3c3Bz8/f1JSpJYu/Yt/vWvVVx44aVU\nVJTzzDP/h5eXN7fffiNpabNISEjsusYrr/yVP/7x/rO+T/flqk8/7s6n8B4lA0mSXgRmA07gblmW\n95302hLgacAOfCXL8pPdnSNJUgywFlABFcC1siy3S5K0EjgfV7fVJ7Is/7C/nHCavu51PNDUAQFE\n3/8A5a/+DfP+dMpaWoi88w+oOj9VCUJ/GX951Vk/xQ+UPXt2sWbNv3jhhVfQ6/Wkpc3sem3+/AW8\n8MKzBAUFMWZMAv7+rk/5kyZNoaDgeFcyqKmppqiokBUrHgFcFVDvuus2brrpNurq6rquV1tbQ0rK\nREJCjNTXu47bbDacTichISE0Njae0vbkrTf745zP8ZIkLQSSZFmeA9wM/O1HTf4GXAnMAy6UJGnC\nWc55Avi7LMvnAfnATZIkpQKLZFme13mNGyVJCnfDvY1IToeDluwsVP4BeMePGepwTqPS+RL1x/vw\nnTqN1twjlD73DNb6+qEOSxD6zGw289prL/Pccy91zR56+OH7KSsrBSAjYz9jxiQSGRmFxWKhqcmE\nw+EgP18mNjau6zpGYygffPAZq1e/xerVbxEcHMKrr64mJSWV3NwcmpubsVgsZGdnMXnyVGbMmM13\n37mmbe/YsY1p09JQq9UkJCSQ1bmZ1datm5k1a45b7rMnTwaLgU8BZFk+IklSoCRJfrIsN0mSlADU\ny7JcAiBJ0led7Y1nOgfXp/87Oq/7ObAc+ALwliTJC9cTgwOwuOXuRqDW/Dzs5mb8F5zvsX3ySq2W\nyDvupPrfazFt20Lx008Qddcf8Y6PH+rQBKHXvv12A42NjTz66ANdxy677HIee+whvL298fHx4aGH\nHgPg97+/l/vu+wMKhYJZs+aQlJRMXp7cVcL6TLy8vLnjjru49967UCgU3HTTrej1ehYvXkp6+h5+\n+9ub0Wq1Xe/x0EMP8eCDD+N0OpgwIZUZM2a55T7PWcJakqTVwJeyLH/W+efvgZtlWT4qSdJc4H5Z\nln/W+drNQCIQcqZzgO2yLId2HksE1sqyPFeSpAeBu3ElgydkWX7lbDHZbHanWq3q800PZwVvvk35\np/9l/KMPEZQ2fajDOSun00n5fz+n8M01KDUaku+9m+A57tmIQxCEPum2hHVfBpC7vdhZXjvTcQVA\n59PFz4AEQAPslCTpfVmWq7t7k4aGvj84DOca5U6nk5pde1B4eWGNjB+U++jv90s7dxGRugAq/vH/\nyH32eUKu/CWBF1+KQnG2v0YDH9dA8dS4wHNjE3H1Tj/3M+j2tZ70M5QDJ/fhR+Ia/D3Ta1Gdx7o7\nxyxJks+P2s4A9siybJFl2QRkA6k9iGvU6aiowFpVhW/qxEEpTOcu+ilTifnzQ51rET6k8p+rcbS3\nD3VYgiCcpCfJYAPwCwBJkqYB5bIsNwPIslwI+EmSFC9JkhpY1tm+u3M24RpspvPr17gGktMkSVJK\nkqQBJgLH3XN7I0tL5gEA9JM9axZRT3jHxhH78F/wTkigec8uip95io7qbh/+BEEYZOdMBrIs7wT2\nS5K0E9esoDslSbpBkqSfdTb5LbAO+B54X5blo2c6p7PtY8D1nWMIQcDbsizvx5U8tgNbgX92Jhnh\nR8yZGaBU4jtp8lCH0ieuqacP4r9wER2lJRQ/9Tjm7KyhDksQBMQeyB7jXHHZTI0cX34PPknJxPzJ\nPYWp3BFXX5l2fE/12rdx2u0E/+SnBC27vFezo4brz3EoeWpsIq7eEXsgj3ItWVmdhemGthaRu/jP\nO4+YBx5BHRRE3X8/pezFF7CdtJhGEITBJZLBMGHuHC/wnTr8xgu64x0fT9yjK/CdNBnLkcMUrXiU\nlkPZQx2WIIxKIhkMA472dixHctBGRaM1Dn1hOndS6fVE/v6PGK+6GkdrK2Uv/ZWaD97DabMNdWiC\nMKqIZDAMtBzuLEw3ecpQhzIgFAoFgUsuJOahR9GEhdGw4WvXbKPKyqEOTRBGDZEMhoETU0qHeu+C\ngeYdG0fcoyvwmzuP9qJCip74Cw2bNuB0OIY6NEEY8UQy8HBOux1zV2G6+KEOZ8Apvb0Jv+lWIu74\nHQqtlpr33qX0heew1tQMdWiCMKKJZODhWo/l4zCb0U+Z4rGF6QaCIW0m8StW4jtlKq1yLoWPP0rj\nti1iFzVBGCCj51+XYaol07V3wUiZUtoban9/Iu/8A+E334pCqaB6zVuUvfQCHTVi5bIguJtIBh7M\n6XRizsxA4eWNz7jxQx3OkFAoFPjNmUfcipXoUidiOXyIosceofTjT8WMI0FwI5EMPFhHRTnW6ip8\nU1NH/faRmqAgou6+l/Bb70Dp5U3R22spemoFrcdFGStBcAeRDDzYD11EI2ehWX8oFAr8Zs0m/smn\nCV2ymI7SEkqeeZLqd9/BbhH7IQlCf4hk4MHMmQdchekmDs/CdANFpdeT9PvfEf2nB9GEhdG4eROF\nDz+Aafv3YhqqIPSRSAYeymZqpO34cXySklHp9UMdjkfSJUvEPfYkwVf8HEd7G1VvvUHJM0/RevzY\nUIcmCMOOSAYeyty54XV/u4gy8mr48Lt8bPaR+YlZqdEQvOxy4p96FsPM2bQVHKfk6Sep/Nc/sZlE\n4TtB6Km+bHspDIIT4wW+/UgGHVY7r3x0EIC6pjZuuzwFZT+3m/RUmqAgIm67A//zF1Gz7h2adm7H\nfCCdwEsuI3DJhSi9vIY6REHwaOLJwAM52tqw5Bzud2G63TlVXf+/90g16zbljfhFW7pkidhHVxB6\nzXUo1BrqPvmIgof/jGnbVpx2+1CHJwgeSyQDD9Ry+BBOm61fXUROp5ON+0pQKRU8cfNMooy+fLu/\nlC92FbkxUs+kUCoJWHQB8c88R9BlP8FhsVC15k2KVjyKOTNjxCdEQegLkQw8kDumlB4paqCstoW0\ncaFEG/Xc+6spBPt588m243yXUeauUD2ayseHkJ9dSfzK/8XvvAV0VFRQ/urLlD73DK15eUMdniB4\nFJEMPIzTbsd8MAtVQABecfF9vs7GfSUALEmLBiDQ4MW9v56MQadh7Tcy27LK3RHusKAJDCT8+puI\nW/GUq9ZR3lFK/nclpX99ntZj+UMdniB4BJEMPExXYbrJU/tcmK6q3kL2sToSI/1IjPTvOh4R7Mv9\nV01F76Ph7fW5fD+KEgKAV2QUUXfdTcwDD6Mbn4Il5zAlzzxF6Yv/J6ajCqOemE3kYVoyXHsX6Pux\nveWm/aU4gSVpMae9Fh2qZ/lVU3h+XQZvrc9FqVQwb2JEn99rOPIZm0T0ffdjOSpT999PsRw+hOXw\nIXSpkwj56RV4j0kY6hAFYdCJZOBBTilMJ/WtMJ2lzcb2gxUEGryYLhnP2CY2zMD9v5nK8+sy+NeX\nR7A7nCyYHNmf0IclXbKEbvmfsci51H32CZZD2RQfykY3PoWgSy/DZ9x4FCN0Kq4g/JhIBh6ko7wc\na001+ulpfS5Mtz27nPYOO8vmxKFWdd/NFBtmYPlVU3nh/UzeWp9LS6uVS2bH9TX0YU0njcPn/gdo\nzT1C/VdfYDlyGMuRw3jFjyHokkvRT50+qvaSEEYnkQw8iLlze8u+7l3gcDjZtL8UjVrJwilR52wf\nF27ggWum8cL7mXy45RjmNiu/WJg4Kj8NKxQKdOMnoBs/gbaC49Sv/xJzxgEqXv87mrBwgi6+BMPs\nuaO+eqwwcomPOx6kJSujszDdpD6dn5lfS62pjTkp4eh9evaPVmSILw/+zzTCgnSs313M21/LOByj\nex6+95gEIn/3e+KfWInf/POw1tZQ9fabFDxwP3Vf/Bdbc9NQhygIbieSgYfoqG9wFaZLlvpcmG5T\nums66dLO6aQ9FeLvw4PXTCM2TM+2rHL+9lE2re1i4xhtRCThN9zMmGeeJ3DpRTjb26j79GMK7r+X\nyjffoL2keKhDFAS3EcnAQ9Tv2wf0faFZcVUzucWNTIgPJMrY+2Ti56vlz1dPI2VMENnH6nj23weo\nb2rrUywjjSYoCOOvf8OY51/EeNU1qIOCadrxPUUr/kLJ889izjggSmcLw54YM/AQ9Xs6k8HkviWD\nTemlACw9w3TSnvLxUvPHX07i3xuOsiWznCfXpPP4rXPw91L1+ZojicrHh8AlSwm4YDEtB7Np3LQR\ny5HDtMq5aEKM+C9chP/lFyM+YwnDkUgGHsDR1kZj9kG0UdFojGeeDno2TS0d7M6pIizQh4mJwf2K\nRaVUcu1FEmFBOj7YnM8Df9/OzZeOJ21c3wvmjTQKpRL95CnoJ0+hvayUxm830rR7F7UffUDdZx+j\nnzoN/4WL8JHGjcrBeGF4EsnAA7QcPojTau3zQrMtmWXY7A6WpMW4pUS1QqHgopmxGAN8+OcXObz2\n6SEunhXLlQsTUIkplqfwioom7LobCfnFr2jatRPz9q0079tL8769aMLDCVhwPn5z54sNigSPJ36z\nPUBL5omNbHo/pdRmd/DdgTJ8vFTMTQ13a1zTko38390LCAvS8fWeYl54L5MmS4db32OkUOl8CVy8\nlKmvvETMnx/CMGsOttpaaj54j+PL/0jFP1ZhOZIjxhYEj9WjJwNJkl4EZgNO4G5Zlved9NoS4GnA\nDnwly/KT3Z0jSVIMsBZQARXAtbIst0uSNBl4o/OSn524xmjgtNsxZ2eiDQ7qU2G6fUeqMbV0cOGM\nGHy83P+gFxfux6PXpfHGlzlk5NXyxFv7+O0VqafUPBJ+oFAo8ElKxicpGftvrqFp53Yat26hec8u\nmvfsQh0UhN+cefjNnYc2zL3JWxD645xPBpIkLQSSZFmeA9wM/O1HTf4GXAnMAy6UJGnCWc55Avi7\nLMvnAfnATZ3HVwO3ATOBCZIk6fp3W8NHa34ejpYWgmbO6HX/stPpZGN6CQoFLJ7eu+mkvaHzVnPn\nzyfy8wUJNDS18+w7B/hqdxEOsS/AWan0egIvvJj4p54h5s8P4Td/AQ6LhfovP6fw4QcofnYlpm1b\nsVssQx2qIPToyWAx8CmALMtHJEkKlCTJT5blJkmSEoB6WZZLACRJ+qqzvfFM5wDnA3d0XvdzYLkk\nSR8DelmWD3Qe/42b7m1YMHfuXRA0cwa9ndmfX2aisLKZaclGjAE+7g/uJEqFgmVz40mM9GP1Fzn8\nZ8sxDhfUc8uyCQQaxJaSZ3Py04LjN9dgPrCfpp07sOTm0JafR/V7/0Y/dRqGWbPxnZCKQi2G8oTB\n15O/deHA/pP+XNN5rKnza81Jr1UDiUBIN+f4yrLcflLbCCAeqJck6S0gCfhQluWXzhZQYKAOtbrv\n0x2NRkOfz3Unp9NJ8cFMVD4++E9M7XWpgzfW5wLwiyXJA3pPJ1/baDQweXw4f3s/k705lax4ax93\nXzWVmRMGv8vDU36OP3b2uAyERV8El19Ee00N1d9tpXrzdzTv2U3znt2oDXqC587BeN58/CaMR6Fy\n77Te4fk9GzqjKa6+fAQ5W1/5WN+fAAAgAElEQVRGd6+d6bjipK9jgCuAVmCXJEkbZVk+3N2bNDT0\n/bHaaDRQU9Pc5/Pdqb2slLbKKvRpM1BqNL2Kq87Uxq7sCmJD9YQZtAN2T919v27/yXiSovx4f3M+\nT76xhwWTI/jVoiR03oPzqdaTfo4n611c3ngvuoiY8y+k7fgxmvfuoTl9L1XfbKTqm42oAgIwpM3E\nMHM23mPG9Hua6sj4ng2ekRjX2ZJIT35zy3F9qj8hEtfg75lei+o81tHNOWZJknxkWW49qW0VcFiW\n5ToASZK2AylAt8lgpDD3Y3vLzQdKcTidLEmLGZK57AqFgsXTo0mOCeCfX+SwLauCg8frufGScaQm\n9G+tw2ijUCjwSRyLT+JYjL/+Da1yLk17d2Pev5/GTRto3LQBjTEUw4yZ6Kel4RUXJ9YvCG7Xk6ml\nG4BfAEiSNA0ol2W5GUCW5ULAT5KkeEmS1MCyzvbdnbMJ12AznV+/lmW5ADBIkhQkSZISmALIbro/\nj9aSeaIw3eRendfeYWdrZjl+Og2zJgztYrCYUD2PXp/G5fPiaWrp4K8fZPHW+iNY2kRto75QKJXo\nxk8g/PqbSPzry0T+/o8YZs3G1mSi/qsvKH7qcQoeWE71++tozTsqpqoKbnPOJwNZlndKkrRfkqSd\ngAO4U5KkGwCTLMufAL8F1nU2f1+W5aPA0R+f0/n6Y8AaSZJuB4qAtzuP3wOsxzUN9WtZlrPcc3ue\ny9bYQFvBcXzGjUfl69urc3cersTSbuPyefFo+jF24i5qlZIrzktgWrKRN7480vWUcPWSZKYlh4hP\nsX2kUKu7Vjo72ttpOXQQ84H9tGRn0rjxGxo3foPKzw/91Gnop6Whk8aJwWehzxTOYTg9sKamuc9B\ne0o/YOPW76he+zbGq64hcMnSHsflcDp59J97qG5o5f9+Nxd//cDO5Ont98tmd/DlriK+3FWIze5k\nUmIw1yxNdvtsJ0/5Of7YYMTltNmw5OZgPrAfc8YB7M2u91PqdPhOmox+ylR0E1JR6U6doT2av2d9\nMRLjMhoN3X4yEx8jhog548R4wZRenZdTUE9FnYU5KeEDngj6Qq1S8tP5Y5g5PpR3Nhwl+1gduUV7\nWDY3notnxZ519zWhZxRqNb6pk/BNnUTo/1xPa34e5v3pmA/sp3n3Lpp37wKVCp+kZPSTpuA7ebJY\n4Cack0gGQ8DR1kprbg7a6Bg0Ib0rTLfhxJ4FMwZukZk7RAT7svyqKezJqeK9zfl8vO04Ow9VctXi\nsUxMCBZdR26iUCpdezknSxivupr2oiJaDmZhzsqkNfcIrblHqPlgHZqwcMyzZ6BMmoDP2CTRnSSc\nRvyNGAIthw7htNl6PYuooq6FQ8frSYr2Jz7cb4Cicx+FQsHslHAmJQbz8bbjfJdRxksfZpMyJohf\nXzCW6D7suyB0T6FQ4B0fj3d8PME/+Sk2UyMtB7NpycqiJecQ5Z99DnyO0scH3YQUfFMmoktNRRMk\nZn8JIhkMCXPWiS6i3hWmc8eeBUNB563hfy6UOH9KFO9vzuNwQT2P/WsvCydHcsV5Cfj5aoc6xBFJ\n7R+A//wF+M9fgMNqRVtVTPm2XZizM13dSvvTAdCGR6BLSXX9J41D6eV53Y/CwBPJYJA57XZasrNQ\nBwbhFRfX4/Na2qzsOFRBsJ8XU5NDBjDCgRMdqufeX0/h4PE63t+cz5bMcnbnVHHJrFiWpA1MoT3B\nRanREDh1CrboRIy/uQZrVSUthw5hyTmEJfcIjd9upPHbjSjUarzHJuHbmRy8omNQiLLlo4L47Rtk\nrXlHcbS0YJg5q1f95tuyyumwOlg8P2ZY7ymgUCiYlBjChPggtmaW89n2Aj75voBN+0u5bHYci6ZF\necR02ZFMoVCgDY9AGx5B4JKlOKxW2o7l03L4EJbDh7rGGvjoQ1QGP3TjxuEzbjw6aTyasDAx3jNC\niWQwyLpWHfdie0u7w8Hm/aVoNUrOmxwxUKENKrVKyeLp0cxNDWfDvhK+2VvMe5vz+WZfCT+ZG8/8\nSRFi5tEgUWo06MaNRzduPFz5S2zNTVhycrAcPkjL4cNdm/UAqAIC0EnjuxKEJsQoksMIIZLBIHI6\nnbRkZqD09sZHGtfj8zKO1lLX1M6iaVH4eveumJ2n8/FS89P5Y1g8PZr1u4v4dn8pa76RWb+niEtn\nxzE3NQKNWiSFwaQ2+OE3azZ+s2bjdDqxVlViyT2CJTeXVjm3a28GAHVQsCsxSK5kogkWg9HDlUgG\ng6ijrBRrbQ36tJm9qlC6sXM66ZIB3LNgqOl9NPxy0ViWzojhy51FbM0q4+2vZf67o5CLZ8ayYHIk\nXlrRfTTYTu5SCjj/ApxOJx3l5VhkV1eSRc6laecOmnbuAEAdEoLP2CTXf0nJaCMixZjDMCGSwSDq\n6iLqxV7HhZVN5JWamJgQTERw78pWDEcBei+uuTCZS+fE8c3eYrZklrHu2zw+31nIhTNiuGBaFLoR\n9nQ0nCgUCryiovCKiiLwgiU4HQ46ykpdTw5yLq35eT8sfMO1Kvrk5OAVH49SI2aPeSKRDAaROTMD\nVCp8J07q8Tkb952YTjpynwrOJNDgxVWLk7hsThyb0kv5dn8pH287zvo9RSyYHMkvl44TG3h7AIVS\niVdMLF4xsQQuvcjVrVRZQWteHq35R2nNy6MlO4uWbFe5MYVajVf8mB8SROJY8NA9A0YbkQwGia2x\ngfbCAnTjJ6DS9ewTfqO5nb1HqogI1pEyJmiAI/RMBp2Wny1I4OJZsWzJKOscbC5h474SpiUbuXBG\nLIlRfmIQ00MoFAq0EZFoIyLxX7AQAFtjI635PySHtmP5tOXn0dB5Tll4GJrYMXgnJOKdkIBXTGyv\nN3oS+k8kg0FyoovItxeziLZklGF3DN2eBZ7Ex0vNJbPjWJIWw94jVXyXUU66XEO6XMOYCANLZ8SQ\nJoWKGUgeSB0QgCFtBoa0GUBnOZbjx2nLz6P1+DE6Cgto27ub5r27gc6nh9hYvMe4koP3mEQ0RjFr\naaCJZDBIejteYLXZ+S6jDF9vNXNTRJGxEzRqJfMmRvDTRUnsOFDChn0lZObVsvq/Obyvz+e8SZEs\nnBxJsL/3UIcqdEPp7YPvhBR8J6QAEBKip/xwPm3Hj7mSRMFx2oqKaDt+HL51naPSGzoTQwJecfF4\nx8Wj9vcfwrsYeUQyGASuwnRH8IqJQRPcs9XDe3KqabZYuWRWrJhFcwYKhQIpNhApNpCqBgvfppey\n41AlX+ws4Ovj3xEU1s686DQumTAdtZv3ERbcS6FQoA0LRxsWjt+ceQA4OjpoL3YlhNbjx2grOHbK\n2AOAOjCwKzF4xcV1JoiAobqNYU8kg0FwojCdbw9rETmdTjaml6BUKLhg2ugaOO6LsEAdVy9NZtl5\n0byW/m9KOvJoBr6uKeSbTV8wxjuFKyYsINE4MhbsjQZKrbZrkDmw85jN1EhbQQFtRYW0FxXSVlRE\nS2aGa8fATqqAALy7EkTnE0SASBA9IZLBIDBnHgB6vtfx0ZJGSqrNpI0LFd0dPVTbWs/qg29T1lHB\n2IAxTPebx3cF+6hSHuO4fT8vZO9HZw1jujGNy1Nn4yuKsQ07av8A9FOmnvJ7ZGtsPCk5uP5rycqk\nJSuzq43KPwDvWNeMJ6/oGNcTeli4WP/wIyIZDDCnzUZLdrarMF1szwrTbdjnWmR24TCrTjpU5Pp8\n3jj8Di1WCwui5vCLpMtRKVUsGDuJRouFTw/uIKs+g1avKrabvuT7rRswksDCuBksTEpBpRTdSMOV\nOiAAfYBra9ATbKYTCaKoK1G0HMym5WB2VxuFVos2MgqvmJjOBBGLV3R0j2f6jUQiGQyw1vw8HJYW\nDLN6VpiuurGVzLxa4sMNJEZ5/p4FQ8npdLK1dCcf5X+OAgVXS1cyL2rWKW0CdDpumLUUWMqhsiK+\nPLqdYmcutRqZj8pkPi70Ic5rHBclzWZS1JihuRHBrdT+AegnTUE/6YcEYW9upr20hPaSEtpLi2kv\nKaGjtIT2woJTzw0O7nqCUKQk0+FndM1kGgVPESIZDLAfuoh6Nl6weX8pTmDpDDGd9Gysdivv5H7I\n7op0DFo9t6ZeR2JA/FnPSY2KIzUqDpvDzre5WWwvSadOWUihI4NVcgaqbH+SfCdw8bg5JIWKGVwj\nicpgQDd+ArrxE7qOOW02OiorOhNE538lxV3jEPVfuNopNBrX2onISLwio9BGRqGNikITHDKikoRI\nBgPI6XRizsxw7SzVg8J0re02vs8ux1+vZca40EGIcHhqbDfx0ubXyKsvJNYQzW0TryPQu+eDhGql\niosmTOOiCdNosrTy1ZG9ZNRk0awpI9e2i9xDu9C0B5FkGMfF0iwx8DxCKdRqVxdR9KndsTaTifbS\nEtT1VdTL+XSUl9NRUU57cREnb0Pf1dUUGelKEJGuMh3qoOG5ratIBgOoo6wUW20thhkze7Tn7I6D\nFbS227l4ptg4vjsFpmL+cfBtTB3NzAibxtXjrkSr6vtqVT+dD1dNX8hVLKS62cSXObs43HAYi7aa\nnI6d5BzciaYjkERfiQsS00iJjHXj3QieSO3vj9rfH6NxDl41rn/+nQ4H1poaOsrLaC8vo+PEf2fo\nalJ4eZ+UICJd02YjIlzlvj14mrNIBgOoa9VxD7qIHA4nm/aXolYpWTg1aqBDG5Z2VaTzXu5H2J0O\nrp18JbOCZrr1E1iowZ8bZ10MXEyFqYGvj+zlUEMOrZpKcq27yc3djTLLj1jvscyLncLMMUmoxeDz\nqKBQKtGGhaENC0M/9YffZ6fdjrWmmvayHxJEe3k5bcVFtBUcP/UiKhXa0DBXFdiICDSdSUIbHu4R\nA9ciGQygHwrTTTxn2/TcKqobWpk/KQI/najqeDK7w84n+V/yXel2fNQ+3JFyDQvGTaempvncJ/dR\nhH8gN86+CLiIqiYTG+R9HKw7jFlTQaHzAIVFB/h3vhehqnimR6SyaOxkfL3ENODRRqFSdZX4Znpa\n13GnzUZHdTUdlRVYKyvoqKigo6rza0U5ZJx6HZWfX9d1upJFePigjkuIZDBArA29K0z3+TbXp4jh\nttn9QDNbW3jj0L852pBPuG8Yt0+8nlDd4O4BHebnz7UzlgBLMLW2sEnOILP6MPWKYqqVMuurZL6q\n+AS9LYJJYSnMjZpIQmjYoMYoeBaFWo1XZCRekZGnHHc6ndibTHRUVtJRWeH6WlHRWen1KK1H5dOu\nowkNdT1FhIahjYgg6JLFAxKzSAYDpKWri+jcC81Ka8xk5tUwLjaAmFD9QIc2bJSZK1iV/TZ1bfVM\nCknh+gm/xls9tJ++/X18uXLKfK5kPla7je/zjrCnLJty23FatGXsaihjV8MGlO3+RGjjmBYxngVj\nU9FpxSI3wVV6Q+0fgNo/4LRJJY6ODqzVVackiY7KCtex8nJaOtvp1E40sxe6PTaRDAZIb1Ydb0o/\nsWeBeCo4IaP6IGty3qPDYeWS+CVcOmYJSoVnDaprVGouGDeRC8a5ugHlqjJ2lx7iUM1hWjTVlCmy\nKavM5r/lH+JrCyPRkMjsuIlMioxDOYKmJAruodRqzzi7yel0Ym9uxlpVia2hgdDz59Bgcbj9/UUy\nGAD21lYsuUfwiok9Z2G6ZksHuw5XEh6sY/LYwe3+8EQOp4OvCjayvvBbtCott6Zey5TQc4+5eAIp\nLIr5qeOoqbmIlvY2tuYfJKMyl0prERavCg62V3Dw6HY47E0g0YwLHMv8hFTiQ8Q0YqF7CoUCtZ8f\naj/XIlS1ry9Y3D9eJpLBALAcPgh2e4+6iLZllWO1OVg2PwGlcvjNTXanVlsbb+e8x8HaHEK8g7ht\n0vVE6YfnHH9fL28uTZnBpSmuGv7Hq6v4viAbuTEPk6KMBnU+u8z57Mr+GmWHnhBVFOODxzI/MZVI\nf7GpvDD4RDIYAOaME3sXnH1Kqc3uYPOBMry1KpbOjKWluW0wwvNI1ZYaVmW/TaWlmnGBSdyUeg2+\nGt1Qh+U2CaFhJIS6ymLYHXYOlB5jX8kRCpsLMKuqqVbJVDfKbN3/ZWdyiEYKSmRO/ATigo1DHb4w\nCohk4GZOm42Wg1mog4Lwijn7AqX9cg0Nze0smR6NzlszapNBTp3Mvw6/S6utlQtizuOKxEtHdPE4\nlVLFjNhkZsQmA9Bhs7K3MJ/9ZUcosRRhUVdTrcql2pTL91lfoujQE6gIJzEgnunREikRMWLMQXC7\nHiUDSZJeBGYDTuBuWZb3nfTaEuBpwA58Jcvyk92dI0lSDLAWUAEVwLWyLLefdK11QLssyze44d6G\nRGveURwWC4ZZc865IGpjegkKYPEo2+z+BKfTyabirXx2bD0qpYrrxv+aWRHThzqsQadVa5g/djzz\nx44HoN1mJb0on4zyXIpbimhRVVOvyqe+JZ998iY4rEHvDCVaF8vEsETSYpPRe4s1DkL/nDMZSJK0\nEEiSZXmOJEnjgX8Bc05q8jfgIqAM2CpJ0keAsZtzngD+Lsvyh5IkPQ3cBLze+T5LgUQgx213NwS6\ntrc8x3jBsTITx8ubmDI2hLDAkdMd0lMddiv/zv2Q9KpM/LV+3DbpOuL9RKkHAC+1hnmJ45mX6EoO\nNoed7NJCDpTJFDQVYXJWYdaWkWstI7d0Fx8UK9BaAwnRRJIUGM/U6CTGGsPE04PQKz15MlgMfAog\ny/IRSZICJUnyk2W5SZKkBKBeluUSAEmSvupsbzzTOcD5wB2d1/0cWA68LkmSF/AI8BTwc7fd3SBz\nOp2Ys3pWmG5jumvPgqWj8Kmgvq2B1QfXUNJcxhi/OG6deC3+XqJcd3fUShXTYhOZFpsIuP6eFdRU\ns69UJq+hgFpbOR3aBioU9VSYDrHNBNi0+DqMROgiGR8Sz/SYZIxGw9DeiODRepIMwoH9J/25pvNY\nU+fXmpNeq8b16T6km3N8T+oWqgZOTBV5ENcTQlNPgg4M1KFW971PeaB+KVoKC7HV1hJy3jxCIwK7\nbVfb2Mp+uYa4cAPnpcV2dSd56i+rO+PKrcnnhf2rMbU3c8GYudw8/So0fSw0Nxq+X90JDfVjVsrY\nrj83tVrYkZdDZmkeBY3FmJxVtGjLyLeVkV+5j88rQbnTlwBlGLH+MUyKGsvcxHEEGTxjkeNo/ln2\nxUDE1ZcB5LN1hHf32pmOKwAkSUoC0mRZflySpPN7EkBDg6Unzc7IaDQMWE2bus3bAVCPn3jW9/ho\n6zHsDieLpkZRW2se8Lj6w51xfV+2mw+PfoYTJ79M/ikLo+bSWN8G9H7gfDR8v3orLUIiLULq+nNJ\nfR3pxXnk1RdS3V5Bq6qWetVx6s3HyZS38nYuqDoMGJQhRPpGkBwcx7ToREIMg/uUJn6WvdOfuM6W\nRHqSDMpxfao/IRLX4O+ZXovqPNbRzTlmSZJ8ZFluPantZUCsJEm7AT/AKEnSn2RZfq4HsXmUrsJ0\nqd0vkmq32tmaWY7eR8PsCaOjfo3NYePDvP+yvWw3eo0vN6f+D8mBiUMd1ogXExRMTFAwrnkcEBzs\ny86co2SW5XO8sZgaayXt6gZMqgJMHQUcqdjJZxWgsPqgJ5hQ73ASAmOYGDGGMcGhYgxihOtJMtgA\nrABWSZI0DSiXZbkZQJblQkmS/CRJigdKgWXANbi6iU47R5KkTcCVwDudX7+WZfmfwEsAnU8GNwzH\nRGCtr6e9qBDd+JSzFqbbfbgSc6uVZXPj0GpG7vTJE5o6mvnnwbUcMxUSpY/g9onXE+wTNNRhjUpK\npRIpLAop7IcS6XaHnbzqCrLLCyhoLKGmvYpWZT3NmlKa7aUcq01nYy1g0+BlDyJIYyTaEE5ySCwT\no+IwePsM3Q0JbnXOZCDL8k5JkvZLkrQTcAB3SpJ0A2CSZfkT4LfAus7m78uyfBQ4+uNzOl9/DFgj\nSdLtQBHwtntvZ+i0ZHUWppva/Swip9PJpvRSVEoFi6aO/IHj4uZSVmevoaG9kamhk7h2/K/wUony\n3J5EpVQxLjyaceE//H10Op0U1dVysKKAYw0lVLVWYnbW0u5VRQVVVLQcYl8LOAtBZfXFVxFMqLeR\nOP8opNAYpNAoND3YzEnwLD36icmy/MCPDmWd9No2Tp1q2t05yLJcASw9y/tsAbb0JCZP0zWldHL3\nySCnqIGy2hZmTwgj0DCyq1imV2bwTu6H2Bx2Lk+4mAvjFg3LrQBHI4VCQXyIkfgQIzCz67iptYVD\n5UUcrS2htLmCBmst7aoGmtXFNNuLOVa/n8314MxRorb6YVAGEeodSlxABMnGGJLCwtGoRJLwVOIn\n4wZdheli49AEd19XZtM+13TSJSO4OqnD6eCzY+vZVLwVb5U3t0y6ltSQ8UMdluAG/j6+zEucwLzE\nHzaVdzgcFNXXcriimMLGMqpaq2iy12HVmGhUNtJoP87ROthYdyJJGNArAwjxNhLtF06yMYq5fufe\nH1wYeCIZuIHlkKsw3dkWmlXVW8g6VkdilB8JkSNzTr3FauHNw+vIqZcJ1YVw+8QbCPcVFTlHMqVS\nyZiQUMaEhAI/7PRlc9g5Vl3Bkepiik0VVLfW0OxowKpuwqQyYbIXcawBtjbAahmUNh0+zgACNcGE\n+4YSHxCBFBZDZECAeKIcJCIZuMGJvQvOVqV00/6RvWdBRUsVq7Lfoqa1jgnBEjdOuBqdRgwujlZq\npQopPBop/NSxMZvDTlFdDblVpRQ1VlBlqabZ0UC7woRFW46FcsossN8ClIPTpkFjdz1NBHsHEaEP\nZUxQBMlhkQTpPHMNwHAlkkE/uQrTZaMOCu62MJ2lzcb2gxUEGryYljzyKlAerM3hrcPraLO3c2Hc\nIn6ScJHHbUQjeAa1UkWiMZxE4w8zz0/Mm69uMiFXl1JQX05FSxX1HXVYnCasmgYalfU02o9zzATb\nTUABYNOidbgSRZB3MBG+RuICw0kyRhJiEImit0Qy6KcThen8ZndfmG57djntHXaWzYlDrRo5/0g6\nnU6+LtzMlwUbUCvV3JhyNWlhU4Y6LGGYCvXzJ9TPn/NIOeW41W6joK6KYzUVlJiqqLbU0mhtoE1h\nol1dT4eyjnrbMfJN8L0JKASsWjQOAzqFH4FegRh1wUT7G0kIDic22Ih6BFfF7SuRDPrJ3LXX8Zn3\nLnA4nGzaX4pWrWThlKgzthmO2mztrD3yAZk1Bwn0CuD2SdcTYxg59yd4Do1KTXJoFMmhp//9arNa\nOVZTSUF9BWVN1dS01mKyNtCmaKJDW49VUYfJWUBhC+xrwdX15FCgsunwxoBBHUCQVyBh+mBiAkIZ\nExyBUW8YlQvsRDLoB6fTiTnzgKswXbJ0xjaZ+bXUmtpYOCUSvU/favB4mtrWelZlv0V5SyVjA8Zw\nS+q1GLSeUeNGGF28NRpSImNIiTx9LM5qt1FcX0NBXSWlphqqLXU0djRgcTRhVZqxaCqxUEmVFY40\nAA1AATjtatQ2X3RKf/QqA0HeQYT6BhHlH0J8cChhBv8RmSxEMuiHjtISbHV1GGbORtHNIptN6SNr\nOqlcn88bh96hxWZhQdQcfpF0+YjeiEYYvjQqNYnGCBKNZ946tc5spqC2ipLGKirNddS11dNka6SN\nZuxqM80qE81ARQcc7sCVLArB6VCisunwQo9B7UeAVwAhukAi9CFEB4QQG2TEWzP8FleKZNAP59q7\noLiqmdziRlLiA4kK6b5ExXDgdDrZUrqDj/O/QIGCq6UrmRc1a6jDEoQ+C9brCdbrSeP0OllOpxOb\n1sb+o8coM9VS1VJHfVsjzVYTbU4zNlULrWozrVRSbYWjJsAElIHTCQqbFxqHHh+FHr3GjyCvAEJ8\nA4kwBBMdEExUYJDHLcDzrGiGGXPGAVCp0HVTmG5Teud00hnD+6nAarfynvwJuyvTMWj13Jp6HYkB\n8UMdliAMGIVCQWRAEJox3XftNrZYKKirorSxhqqWeupbG2iymrA4mulQtHSNWTQB5R24ync2AMXg\ndCo6E4YvPko9erWBQG9/QnwCCDMEERUQTHRAyKA+YYhk0EfW+jrai4vQTUhBpTt9p7Kmlg5251QS\nFqQjNaH7Vcmerr61kZcyVlHYVEysIZrbJl5HoHfAUIclCEMuwFfHVN8xTI0dc8bXbQ475aYGSuqr\nKW+upbalgYZ2E2ZrM61OM1aFhQ5NPVZlZ8JoB9qBRqDkxEW0qB06vPDFV2UgwMufO87/CV64P0mI\nZNBHLefoItqSWYbN7mTJ9GiUw3QFZYGpiDd2vkNDm4kZYdO4etyVaPu4EY0gjDZqpYrYwBBiA0O6\nbWNz2KlobKCkoZaKpnpqLA00tptotjbR6jDTobBgVTVjUzXSAlTbYN1ePTekXeT+eN1+xVHCnJUJ\nnHnVsc3u4LsDZfh4qZk3Mfy014eDXeX7eE/+GDsOfj52GRfEnCfKAgiCm6mVKmKCQogJ6j5hOBwO\n6i1mShpqabA0c1laGq3NVvfH4vYrjgJ2i+WHwnRBp3cB7TtSjamlg4tmxuCtHV7fYrvDzsf5X7Cl\ndAc6tQ/3zruVCNXIL7ctCJ5KqVQSovcjRO+qaab39hbJwFN0FaabevpCM6fTyYb0EhQKWDxteP0j\nau5o4Y1D73C08RjhvmHcPvF6UsLHeOTWf4IguJdIBn3Qtep48umlF/LLTBRVNjM92UhIwPAp1FZm\nrmBV9lvUtTUwOSSF6yb8Gm+191CHJQjCIBHJoJdchemyUAefuTDdxq49C4bPU8GB6mzW5rxPh8PK\npfFLuGTMElFoThBGGZEMeslyVMbR2orfnHmnDajWmlrZf7SG2DA9yTGeP/3S4XTwZcFGvi78Fq1K\ny60Tr2OKMXWowxIEYQiIZNBLXVNKzzBesPlAGU6na88CT59502pr4+2cdRysPUKIdxC3T7qBSP3w\nnPkkCEL/iWTQC67CdBkodTp8kpJPea29w862zHL8dBpmjg8bogh7ptpSw6rst6m0VDMuMImbUq/B\nV3P6wjlBEEYPkQx6oWhaOd0AABUoSURBVL2kGFt9HYZZpxem23moAku7jcvnxaNRe25/++E6mTcP\nv0urrZULYs7jisRLRaE5QRBEMuiNri6iyacuNHM4/397dx5eVXkncPyb3CQEEhICCRIIhEX4sSSI\nIEV2EEq1YxeLy4gCKgpjbcduPuM8nWmn2namtqMdrE+r1SpSte5rqVrQuuBSQEPC9ougbAYkJCQk\nNyHbvfPHOYFLvNnvEnp/n+fhyc17zrnndw9vzu+c97z3fZ05CxI8ccw/t2eO6e/3+1m//w2e3/MX\nPPEelo27gmnZU6IdljGmh7Bk0AnVBR86A9PlTzytfPsn5Rwqq2FG3iDSU3tFKbrW1TfV88iup9j8\nWQHpSWmsmric3LQze/A8Y0xoWTLooIYyd2C6CXl4ep/+/YG/unMW9MTJ7stPHOO+ooc5UPUpI9Jy\nuSF/Kem90qIdljGmh7Fk0EHercEHpjtU5mXbx+WMyUknd1DPmoR7d8Un3F+0lqqGamZkT+VyuYTE\nePsvN8Z8np0ZOujUt45PTwY9dc6Ctz59lyeKnwfg8jFfZ86Q6T2+u6sxJnosGXRAU00NNbqLXrnD\nSezf/2S590QDG7cdYkBaMueOzopihKc0+hp5svh53i55n9TEFFbkXc2YjM/P5GSMMYEsGXSAd1uh\nMzBdiyaiN7eWUN/gY8GsHOLjo3/Vfby+ivuL1rKnci9DUrNZlb+cAb37t7+hMSbmWTLogGAT2TT5\nfGzYcpBeiR7mnBN8wu1I2n/8IPcWraGirpLJAydy9bjL6eU58yblNsZEhyWDdjgD0xWSMGAASTmn\nngt8WHyU8uN1XDB5CH2Sozv716bDH/LIridp9DXx1ZEXsih3vj0fMMZ0iiWDdpwcmG7GrNNOsK+6\n3UkXTIne6KQ+v4/n9qxjw/43SfYkc/3EpeRljotaPMaYM1eHkoGI3AWcD/iBm1V1U8CyhcDPgSZg\nnare3to2IjIUWAt4gEPAUlWtE5ErgO8DPmCDqv4wVB+wu7wFHwCnNxF9cug4uw9WMnHUALIHpEQl\nrpqGGv6w/VF2lhczsE8mq/KvYVDKwKjEYow587U7iI6IzAVGq+p0YAWwusUqq4HFwExgkYiMb2Ob\n24B7VHU2sBu4TkT6AL8AFgDTgYUiMr77H637WhuYbv3m6M5ZcMj7GXdsvpud5cVMGDCWW6Z82xKB\nMaZbOjKi2gLgOQBV3QlkiEgagIiMBMpV9YCq+oB17vqtbTMPeMF93xeBhapaA+SrapWq+oEy4PMT\nC0eBMzBdOSn5E08OTFdRXcffdx4he0AfJgyPfE+dwtLt/GrzbyitLWNR7nz+ZeI19Ek8c2ZUM8b0\nTB1pJhoEbAn4vdQtO+7+LA1YdgQYBWS2sk2KqtYFrJsNoKpVACKSDwwH3msroIyMPiQkdH2kzays\njn1TeP/67QAMnjODTHebV7YcpMnn55L5oxk4MLTDOrQVl8/v45kdL/PEthdJ8iTynekrmDHsvJDu\nvytxRZPF1Xk9NTaLq3PCEVdXHiC31U2ltWXByk8rE5HRwKPAElVtaCuAY8dq2gywLVlZfTs8wfuR\nje+Bx0PjsNGUllbR0NjEuo2fkJKcQH5uv5BOFN9WXCca61i78wkKSovI6NWPVROXM7T3kIhMVN+Z\n4xVJFlfn9dTYLK7O6U5cbSWRjiSDEpyr+maDcR7+Bls2xC2rb2WbahHpraq1AesiIjk4zUpLVbWg\nAzGFXUNZGXUH9p82MN17Oz6jqqaBi84fRq/EyMwBcLS2nHsLH6LEe5iz+43g+ryl9E1Kjci+jTGx\noyPPDF4FLgUQkclASXOzjqruBdJEZLiIJAAXu+u3ts16nIfNuD9fdl8/ANyoqh+E4kOFQvXJgemc\n6S39fj/rNx8kPi6OBZMj8+B4V/lH3LFpNSXew8wZMoN/nbTSEoExJizavTNQ1XdEZIuIvIPT9fMm\nEbkGqFTVZ4Ebgcfc1R9X1WKguOU27vIfAw+LyCpgH7BGRMYAs4HbRKR5t3eqavOD5qjwftg8MN0k\nAHR/BQeOVDN17ED6pyWHdd9+v5+/HdzIM7tfIo44loxdzMzB08K6T2NMbOvQMwNVvbVF0daAZW/i\ndAltbxtU9RDwxRbFxUCPmoC3qcZLTfHpA9OdnLMgzKOTNjQ18Cd9lvcOb6ZvUio35C1jVL/hYd2n\nMcbYN5CD8BYVnTYw3ZGKWgo+OsqI7L6MGhy+iWEq6ir5fdFa9h7fz7C+OazMX0ZGcr+w7c8YY5pZ\nMgji1LeOnecFr205iB9nJrNwjflTfPRjfrnpd1TWV/GFQZO5UhaT5InumEfGmNhhyaAFf2Mj3m1F\nJGRmkpSTQ21dI28VlpCemsR5Y8PzLd93Szbxp+JnafI1sfjsi5k/dLYNNGeMiShLBi3U6C5nYLqZ\nzsB0G4sOUVvXxIXTcknwdKTzVcc1+Zp4ZvdL/O3gRlKS+nDt+CWM6z+m/Q2NMSbELBm0UF1wqkup\nz+9n/ZaDJHjimTtpcGj3U+/lgW1/pLhiD9kpZ/Hv827CUxveXkrGGNMaSwYB/H4/3q3uwHRnj2br\nnjKOHKtl9sRs0vqEbqKYT6sPcW/hQ5SdOMY5mRNYNv4KBqVmUVrb877taIyJDZYMAtTt30djeTl9\np00nLiGBv25yu5OeF7rupB8cKWTtjsep9zXw5RFf5KLhC4iPC23zkzHGdJYlgwAnm4jOPZeDpdXs\n3HeMcbkZ5Azs/rd+fX4ff/7kr7y8dwNJniRuyF/GpKy8br+vMcaEgiWDAN6CD4lLSCAlL5+nX98L\nhGbOgtrGE6zZ8RhFR3eSmdyfVROvYXDqoPY3NMaYCLFk4GooO+oMTJeXj9fn4d3tnzGwX2/OGZXZ\nrff9rKaU+wrXcLjmCGMzRnNd3lWkJPaoL1wbY4wlg2anehGdyxsFJTQ0+lgwJYf4+K73999epjy4\n/RFqG09wwdDZfH3Ul/HER2a0U2OM6QxLBi6vmwyS887htT/tJDnJw6yJ2V16L7/fz/r9b/D8nr/g\nifewbNwVTMueEspwjTEmpCwZ0DwwndJr+AgKjjRQUV3PwvNy6N2r84envqmeR3Y9xebPCujXK52V\n+cvITQvv4HbGGNNdlgwAb1HhyYHp1m8+SBywcErnHxyXnzjGfYVrOFBdwsj0XK7PW0Z6r545bZ4x\nxgSyZMCpJqJjg0fz8bZDTDo7k4EZnXvIu7viE35f9DDVDV5mZE/lcrmExHg7vMaYM0PMn62aB6ZL\nzMzilf2NQOfnLHjr03d5ovh5AC4f83XmDJluA80ZY84oMZ8MmgemS5o6nc16lJysVMYO69gcAo2+\nRp4sfp63S94nNTGFFXlXMyZjVJgjNsaY0Iv5ZFDtzl2wo9cQfH4/Xzwvp0NX9cfrq7i/aC17KveS\nkzqYlfnLGdA7I9zhGmNMWMR0MvD7/XgLCojv04d1JR5Se3s4f8JZ7W637/gB7it6mIq6SiYPnMjV\n4y6nlyd0A9kZY0ykxXQyqNu3j8Zj5dTIJKrrfFw8YxiJCW1/Kezvhz/g0V1P0ehr4qsjL2RR7nx7\nPmCMOePFdDKo3ur0ItrkG4gnPo755w5pdV2f38dze9axYf+bJHuSuX7iUvIyx0UqVGOMCauYTgbe\ngg/wezxsaRrA1PyBZPTtFXS9moYa/rD9UXaWF3NWnyxW5S/nrJTwTIFpjDHRELPJoOFoKXUHDlCa\nOZz6+MRW5yw45P2MewsforS2jAkDxnLthCvpndA7wtEaY0x4xWwyqC4oAOCDuEGcPSSdEdlpn1un\nsHQ7D+14jLqmehblzucrI79kE9EYY/4hxXAycLqU7k7J4eoWcxb4/D5e2fsaL33yKonxiVw3YQlT\nzpoUjTCNMSYiYjIZNHm91BYrh5MzSeqfwRTJOrnsRGMda3c+QUFpEf2TM1iZv5yhfQdHMVpjjAm/\nmEwG3qKt4POhfXK4YHIOnnin6edobRn3Fq6hxHuY0f1GsiLvavomdX/KS2OM6eliMhk0T2SzNz2X\nK89xrvp3lX/EH7Y9grexhrk5M1h89ldsIhpjTMyIuWTga2igqrCQioRUZPJYUpITeP3A2zyz+yXi\niGPJ2MXMHDwt2mEaY0xExVwyqCzaRlx9HR+lj2TulMGs3fkE7x/eQt+kVFbmL2Nk+vBoh2iMMREX\nc8lg72sbAagbM4rH9z/MvuMHyO07lBvyl5KR3LHRSo0x5h9Nh5KBiNwFnA/4gZtVdVPAsoXAz4Em\nYJ2q3t7aNiIyFFgLeIBDwFJVrRORq4DvAD7gPlV9IFQfMJDf76di82aaPInomG14j9fwhUGTWSKL\nSfQkhmOXxhhzRmj3G1QiMhcYrarTgRXA6harrAYWAzOBRSIyvo1tbgPuUdXZwG7gOhFJAX4ELATm\nAd8Vkf7d/mRBlO/6iKTaKvYO9VDjr2Xx2RezbNwVlgiMMTGvI1+nXQA8B6CqO4EMEUkDEJGRQLmq\nHlBVH7DOXb+1beYBL7jv+yJOApgGbFLVSlWtBTbiJJaQ2/zaSwDsG5rCTZNWcMGwOTbiqDHG0LFm\nokHAloDfS92y4+7P0oBlR4BRQGYr26Soal3AutmtvEd2WwFlZPQhoZ2hpoPpf04euytKWLH0hww/\na1intw+3rKy+0Q4hKIurc3pqXNBzY7O4OicccXXlAXJbl9KtLQtW3pl1T3PsWE17qwQ1ddYisi5Z\nTGlpFaWlVV16j3DJyurb42ICi6uzempc0HNjs7g6pztxtZVEOtJMVIJz9d5sMM7D32DLhrhlrW1T\nLSK921m3udwYY0yEdCQZvApcCiAik4ESVa0CUNW9QJqIDBeRBOBid/3WtlmP87AZ9+fLwPvAVBHp\nJyKpOM8L3grNxzPGGNMR7TYTqeo7IrJFRN7B6fp5k4hcA1Sq6rPAjcBj7uqPq2oxUNxyG3f5j4GH\nRWQVsA9Yo6oNInIr8ApON9SfqGplCD+jMcaYdsT5/f5ox9BppaVVXQ76H7EdMJwsrs7pqXFBz43N\n4uqcbj4zaPWZrM3UYowxxpKBMcYYSwbGGGOwZGCMMYYz9AGyMcaY0LI7A2OMMZYMjDHGWDIwxhiD\nJQNjjDFYMjDGGIMlA2OMMVgyMMYYQ9cmtzljichdwPk4o6PerKqbIrTfO4DZOMf7v4GvAlOAMneV\nX6rqn0XkKuA7OCO93qeqD4hIIvAQkAs0Adeq6schiGke8CSw3S0qAu4A1gIenPknlqpqXYTjWgEs\nDSg6D9gMpABet+z7qrpFRG4BLuPUaLfrRCQdeBRIB6qBJapa3o148oDngbtU9TciMpRuHiMROQf4\nrRt3oareGKK4HgQSgQbgalU9LCINOFPJNluAcxEYqbgeopt1PUxxPQlkuYv7A+8BP8f5O2iepbFU\nVS9rrU6JyEJ3myZgnare3oW4Wp4bNhGl+hUzdwYiMhcYrarTgRXA6gjtdz6Q5+73QuDX7qJ/V9V5\n7r8/i0gK8COceaHnAd8Vkf7AEqBCVWcBP8OpMKHyRkAM3wZuA+5R1dnAbuC6SMelqg80x4Qz5Pka\nd9G1AbFuEZERwD8Ds3Dm0bhTRDw4fzB/c+N6Bvi3rsbifva7gQ0BxaE4Rr/GuRiZCaSLyEUhiOun\nOCeJucCzwPfc8sqA4zZPVZsiHBd0v66HPC5VvSygnm0G7j+16GSsl7llrdWp1TjzsswEFonI+E7G\nFezcELX6FTPJAOeK6DkAVd0JZIhIWgT2+ybO1StABc4VbrAJnKcBm1S1UlVrca7mZuLE/ay7znq3\nLFzmAS+4r1/EqXzRjOtHQGtXW/OBv6hqvaqW4syPMb5FXM2foavqgC9z+sx78+jGMRKRJGBEwF1p\nV2IMFtc3gafd16XAgDa2j2RcwfSE4wWAiAjQT1X/3sb2n6tTIjISKFfVA6rqA9a563VGsHPDPKJU\nv2KpmWgQp27/wPmDGQQcD+dO3Sux5uaNFTiVpgn4loh8DzgCfMuNpTRg0yNAdmC5qvpExC8iSapa\nH4LwxovICzi3yT8BUlS1rrX9RzAuRGQqcMBt6gC4TUQygZ04V2rtxhVQ1iWq2gg0uvtv1q1j5JYd\nC7Jut+JSVS+Ae3d0E84VJkCyiDyK05TwtKreGcm4XF2u62GOC+BmnLuGZoNE5CmcqXrvUdVHCF6n\ngn2GUZ2MK9i54UvRql+xdGfQUquTPISDiHwN5z/8Wzhtgreq6gVAAfBfQTZpLb5Qxf0RTgL4GrAc\neIDTLw46u/9QH8/rcdpDAf4PuEVV53D6zHnt7T/c/8ehOEYhi9FNBGuB11S1uUnkB8BKYBFwlYic\nF+G4Ql3XQ3m8koBZqvq6W1QG/CdwJc5zvdtFpOWJNOT1v8W5oTv76tbxiqVkUIKTNZsNxnlAE3Yi\n8iXgh8BF7q3eBlUtcBe/AOQHiW+IW3ay3H1gFBeKq29V/VRVH1dVv6ruAQ7jNJ31bm3/kYgrwDzg\nHTfWZ90Ywbnt7dDxCigLperuHCOcOjcgyLqh8CDwkar+pLlAVX+nqtXuncMGWhy7cMfV3boerrhc\nc4GTzUOqWqWqD6pqg6oexXmWMJbgdaq1z9ApLc8NRLF+xVIyeBW4FEBEJgMlqhr2Oe3cngi/BC5W\nt1eLiDzttjmCc9LbBrwPTBWRfiKSitMm+JYbd3O74leA1wkBEblKRH7gvh4EnIVzMlnsrrIYeDnS\ncbnxDAaqVbVeROJEZL2I9HMXz8M5Xq8B/yQiSe76Q4AdLeJq/gyhtJ5uHCNVbQB2icgst/wboYjR\n7W1Sr6o/DigTEXnUPYYJblzbIxxXt+p6uOJyTQW2BsQ6X0TudF+nAJOAYoLUKVXdC6SJyHD32F7s\nrtdhwc4NRLF+xdQQ1iLyP8DJpgZV3drOJqHY50qcW+PigOIHcW4Ja3C6ql2rqkdE5FLgFpwuYXer\n6iPurf/9wGicB2HXqOqBEMTVF6e7XD8gCafJ6EPgYSAZ54HstaraEMm43NimAD9V1Yvc3y/H6cHh\nBT4FVqhqjYh8G7jKjes/VHWD+8fyR5yrowqcLpaV3Yjjf4HhON01P3X39xDdOEZur5N7cS7G3lfV\n79EJrcQ1EDjBqWdgO1T1myLyC+ACnDr/gqr+LMJx3Q3cSjfqepji+gZOnX9bVR9310tw9y84nTx+\nq6oPtlanRGQO8At3N0+r6q86GVewc8NyN4aI16+YSgbGGGOCi6VmImOMMa2wZGCMMcaSgTHGGEsG\nxhhjsGRgjDEGSwbGGGOwZGCMMQb4f/cwEdYdtsdgAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f1818812828>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8tkzxQYKUgRQ"
      },
      "source": [
        "## Regularization                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
        "                                                                                                                                                                                                                                                                                                                      \n",
        "### Label Smoothing\n",
        "\n",
        "During training, we employed label smoothing of value $\\epsilon_{ls}=0.1$ [(cite)](DBLP:journals/corr/SzegedyVISW15).  This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVWDJLXrUgRQ"
      },
      "source": [
        "class LabelSmoothing(nn.Module):\n",
        "    \"Implement label smoothing.\"\n",
        "    def __init__(self, size, padding_idx, smoothing=0.0):\n",
        "        super(LabelSmoothing, self).__init__()\n",
        "        self.criterion = nn.KLDivLoss(size_average=False)\n",
        "        self.padding_idx = padding_idx\n",
        "        self.confidence = 1.0 - smoothing\n",
        "        self.smoothing = smoothing\n",
        "        self.size = size\n",
        "        self.true_dist = None\n",
        "        \n",
        "    def forward(self, x, target):\n",
        "        assert x.size(1) == self.size\n",
        "        true_dist = x.data.clone()\n",
        "        true_dist.fill_(self.smoothing / (self.size - 2))\n",
        "        true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)\n",
        "        true_dist[:, self.padding_idx] = 0\n",
        "        mask = torch.nonzero(target.data == self.padding_idx)\n",
        "        if mask.dim() > 0:\n",
        "            true_dist.index_fill_(0, mask.squeeze(), 0.0)\n",
        "        self.true_dist = true_dist\n",
        "        return self.criterion(x, Variable(true_dist, requires_grad=False))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_gXwt5HVUgRT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254
        },
        "outputId": "3ea1a660-0bc7-4b1e-ba8e-7d3fbf560640"
      },
      "source": [
        "#Example\n",
        "crit = LabelSmoothing(5, 0, 0.5)\n",
        "predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],\n",
        "                             [0, 0.2, 0.7, 0.1, 0], \n",
        "                             [0, 0.2, 0.7, 0.1, 0]])\n",
        "v = crit(Variable(predict.log()), \n",
        "         Variable(torch.LongTensor([2, 1, 0])))\n",
        "\n",
        "# Show the target distributions expected by the system.\n",
        "plt.imshow(crit.true_dist)\n",
        "None"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAADsCAYAAAB+Hb1HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADQNJREFUeJzt3GGo3fV9x/H3mZm0y0iQdRh0pUGR\n7ybpE30wT6RLS7JUrZ2w6IQFWos+cXY4xmBKN7sR0OFwaaMP3INtUsokRYzKdjGhdiy190ptcKUO\n+fpghqVewYxg1CJibs4enHPN4XBzk/M/9+Z/+r3v15Oc/+///5/flz/nfs4v33POv9Pr9ZAk1fMr\nbRcgSVodBrwkFWXAS1JRBrwkFWXAS1JRBrwkFbWuyUkR8avAE8BngAXga5n5PyPHfAT8aGhoe2Yu\nNKxTkjSmRgEP/DHwTmbujoidwEPA7SPHnMzMz09SnCSpuaYtmu3AgcHj7wPXr0w5kqSV0jTgNwHH\nATLzNNCLiItHjvlERPxrRPwoIv58kiIlSeM7Z4smIu4C7hoZ/t2R7c4Sp/4F8F2gBxyOiMOZ+ZNl\npvKeCZI0vqXyt7+jyb1oIuIJ4MnMPDj4wPVoZl6+zPEPA69l5r8s87QG/MDc3FzbJQDQ7XZbr2Xr\n1q2tzg/Q6/XodM76N3TBzM7Otl3CVLwmpsW0XItut3vWF2fTD1kPAbcBB4EvA/8xvDMiAvgmsBu4\niH6P/qmGc0mSGmga8PuB34+IF4EPgTsAIuI+4D8zcy4ijgE/Bk4Dz2Xmj1egXknSeWoU8IPvs39t\nifG/G3r8lxPUJUmakL9klaSiDHhJKsqAl6SiDHhJKsqAl6SiDHhJKsqAl6SiDHhJKsqAl6SiDHhJ\nKsqAl6SiDHhJKsqAl6SiDHhJKsqAl6SiDHhJKsqAl6SiDHhJKsqAl6SiDHhJKsqAl6SiDHhJKsqA\nl6SiDHhJKsqAl6Si1jU9MSL2AtcBPeDezHx5aN8O4EFgAZjJzD2TFipJGk+jFXxEbAOuyswucCew\nb+SQfcAu4HpgZ0RcPVGVkqSxNW3RbAeeAcjM14BLImIDQERcAZzIzGOZeRqYGRwvSbqAmrZoNgFH\nhraPD8beHfx7fGjf28CVDedZk7rdbtslfKztWnq9XqvzL5qWOqZB26+JaTLt16JxD35Ep+E+LWFu\nbq7tEoD+i7ftWrZu3drq/NAP906n/Zfx7Oxs2yVMxWtiWkzLtVjuTaZpi2ae/kp90WXAW2fZd/lg\nTJJ0ATUN+EPArQARcQ0wn5nvAWTmUWBDRGyOiHXAzYPjJUkXUKMWTWbORsSRiJgFTgP3RMQdwMnM\nPADcDTw5OHx/Zr6+ItVKks5b4x58Zt43MvTToX2Hgen+9EGSivOXrJJUlAEvSUUZ8JJUlAEvSUUZ\n8JJUlAEvSUUZ8JJUlAEvSUUZ8JJUlAEvSUUZ8JJUlAEvSUUZ8JJUlAEvSUUZ8JJUlAEvSUUZ8JJU\nlAEvSUUZ8JJUlAEvSUUZ8JJUlAEvSUUZ8JJUlAEvSUWta3piROwFrgN6wL2Z+fLQvqPAMWBhMLQ7\nM99sXqYkaVyNAj4itgFXZWY3In4H+GegO3LYjZn5/qQFSpKaadqi2Q48A5CZrwGXRMSGFatKkjSx\npi2aTcCRoe3jg7F3h8Yej4jNwIvA/ZnZaziXJKmBxj34EZ2R7QeA54ET9Ff6u4CnVmiu8rrd0W5X\ne9qupdebjnXBtNQxDdp+TUyTab8WTQN+nv6KfdFlwFuLG5n5ncXHETEDfBYD/rx1OqPvl+3o9Xqt\n1zI7O9vq/ND/I56bm2u7jKngtThjWq7Fcm8yTXvwh4BbASLiGmA+M98bbG+MiIMRcfHg2G3Aqw3n\nkSQ11GgFn5mzEXEkImaB08A9EXEHcDIzDwxW7S9FxAfAK7h6l6QLrnEPPjPvGxn66dC+bwPfbvrc\nkqTJ+UtWSSrKgJekogx4SSrKgJekogx4SSrKgJekogx4SSrKgJekogx4SSrKgJekogx4SSrKgJek\nogx4SSrKgJekogx4SSrKgJekogx4SSrKgJekogx4SSrKgJekogx4SSrKgJekogx4SSrKgJekogx4\nSSpq3SQnR8QW4Flgb2Y+NrJvB/AgsADMZOaeSeaSJI2n8Qo+ItYDjwIvnOWQfcAu4HpgZ0Rc3XQu\nSdL4JmnRfAjcBMyP7oiIK4ATmXksM08DM8D2CeaSJI2pcYsmM08BpyJiqd2bgOND228DVzada63p\n9Xptl/CxaaqlTd1ut+0SpobX4oxpvxYT9eDH0LlA85TQ6UzH5er1eq3XMjs72+r80P8jnpuba7uM\nqeC1OGNarsVybzKr9S2aefqr+EWXs0QrR5K0elYl4DPzKLAhIjZHxDrgZuDQaswlSVpa4xZNRFwL\nPAJsBj6KiFuB54A3MvMAcDfw5ODw/Zn5+oS1SpLGMMmHrEeAzy+z/zAw3Z9ASFJh/pJVkooy4CWp\nKANekooy4CWpKANekooy4CWpKANekooy4CWpKANekooy4CWpKANekooy4CWpKANekooy4CWpKANe\nkooy4CWpKANekooy4CWpKANekooy4CWpKANekooy4CWpKANekooy4CWpqHWTnBwRW4Bngb2Z+djI\nvqPAMWBhMLQ7M9+cZD5J0vlrHPARsR54FHhhmcNuzMz3m84hSWpukhbNh8BNwPwK1SJJWkGNV/CZ\neQo4FRHLHfZ4RGwGXgTuz8xe0/kkSeOZqAd/Dg8AzwMngGeAXcBTqzhfGb3e9LwPTlMtbep2u22X\nMDW8FmdM+7VYtYDPzO8sPo6IGeCzGPCSdMGsytckI2JjRByMiIsHQ9uAV1djLknS0jpN/wseEdcC\njwCbgY+AN4HngDcy80BE3At8FfgAeAX403P04O0FSNL4OmfdMUU91qkpRJJ+iZw14P0lqyQVZcBL\nUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEG\nvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVZcBLUlEGvCQVtW6SkyPiYeBz\ng+d5KDOfHtq3A3gQWABmMnPPJHNJksbTeAUfEV8AtmRmF7gB+NbIIfuAXcD1wM6IuLpxlZKksU3S\nojkM3DZ4/A6wPiIuAoiIK4ATmXksM08DM8D2iSqVJI2lcYsmMxeAXww276TfhlkYbG8Cjg8d/jZw\nZdO5JEnjm6gHDxARt9AP+J3LHNaZdB5J0ngm/ZD1i8A3gBsy8+TQrnn6q/hFlw/GJEkXSKfX6zU6\nMSI2Aj8EdmTm20vs/2/gS8DPgTlgd2a+vsxTNitEkta2s3ZIJlnB3w58CvheRCyO/QD4WWYeAO4G\nnhyM7z9HuEuSVljjFfwqmJpCJOmXyFlX8P6SVZKKMuAlqSgDXpKKMuAlqSgDXpKKMuAlqSgDXpKK\nMuAlqSgDXpKKMuAlqSgDXpKKMuAlqSgDXpKKMuAlqSgDXpKKMuAlqSgDXpKKMuAlqSgDXpKKMuAl\nqSgDXpKKMuAlqSgDXpKKMuAlqah1k5wcEQ8Dnxs8z0OZ+fTQvqPAMWBhMLQ7M9+cZD5J0vlrHPAR\n8QVgS2Z2I+I3gFeAp0cOuzEz35+kQElSM5O0aA4Dtw0evwOsj4iLJi9JkrQSGq/gM3MB+MVg805g\nZjA27PGI2Ay8CNyfmb2m80mSxjNRDx4gIm6hH/A7R3Y9ADwPnACeAXYBTy3zVJ1Ja5EkndHp9Zov\nqiPii8Ae4IbMPLHMcX8CXJqZ32w8mSRpLI178BGxEfh74ObRcI+IjRFxMCIuHgxtA15tXqYkaVyT\ntGhuBz4FfC8iFsd+APwsMw9ExAzwUkR8QP8bNsu1ZyRJK2yiFo0kaXr5S1ZJKsqAl6SiJv6aZBUR\nsRe4DugB92bmyy2X1JqI2AI8C+zNzMfarqdNy92OYy2JiF8DngAuBT4B7MnMf2u1qJZFxCfpf3lk\nT2Y+0XI5S3IFD0TENuCqzOzS/07/vpZLak1ErAceBV5ou5a2Dd+OA7gB+FbLJbXpy8BPMnMb8EfA\nP7RczzT4K/q/85laBnzfdvo/xiIzXwMuiYgN7ZbUmg+Bm4D5tguZAt6OYyAz92fmw4PNTwM/b7Oe\ntkXEbwNXA//edi3LsUXTtwk4MrR9fDD2bjvltCczTwGnhr76umad5+041pSImAV+C7i57Vpa9gjw\ndeCrbReyHFfwS/O2CfrY0O04vt52LW3LzK3AHwDfjYg1+XcSEV8B5jLzjbZrORcDvm+e/op90WXA\nWy3VoikyuB3HN+jf+vpk2/W0JSKujYhPA2Tmf9H/3/9vtltVa74E3BIRLwF3AX8dETtarmlJtmj6\nDgF/C/xjRFwDzGfmey3XpJYN3Y5jx3L3Wlojfg/4DPBnEXEp8OvA/7VbUjsy8/bFxxHxN8DRzPx+\nexWdnQEPZOZsRBwZ9BdPA/e0XVNbIuJa+v3FzcBHEXEr8IdrNOCWuh3HVzLzf9srqTWPA/8UET8E\nPgnck5mnW65J5+CtCiSpKHvwklSUAS9JRRnwklSUAS9JRRnwklSUAS9JRRnwklSUAS9JRf0/fdse\n4Y9HCc4AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f181876c358>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sHBHTwmjUgRU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 288
        },
        "outputId": "4086db46-2dd9-4566-804e-8e5f80641a69"
      },
      "source": [
        "# Label smoothing starts to penalize the model \n",
        "# if it gets very confident about a given choice\n",
        "crit = LabelSmoothing(5, 0, 0.2)\n",
        "def loss(x):\n",
        "    d = x + 3 * 1\n",
        "    predict = torch.FloatTensor([[0, x / d, 1 / d, 1 / d, 1 / d],\n",
        "                                 ])\n",
        "    #print(predict)\n",
        "    return crit(Variable(predict.log()),\n",
        "                 Variable(torch.LongTensor([1]))).data[0]\n",
        "plt.plot(np.arange(1, 100), [loss(x) for x in range(1, 100)])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f181867fac8>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD8CAYAAABq6S8VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XlwnPd93/H3HrjPBbC4wRv88pRs\nWgcZWZYcqrnsmdS20mbGbapEbtpUaZRk2o7SHM3RNplmPIqddlo7jeOmsZ1MnLFij4+oduKTFkVT\nEiXx+AkgQYK4F/dJnNs/ngW4uIiFCHDx7H5eMxrtPs8DPL8fl/zgh9/zOwLxeBwREfGvYLoLICIi\nd0dBLiLicwpyERGfU5CLiPicglxExOcU5CIiPhdO5SIzex44CcSBZ51z5xLHG4DPJF26D3jOOffZ\nrS6oiIisbcMgN7PHgGbn3CkzOwx8CjgF4JzrBB5PXBcGvgl8cbsKKyIiq6XStXIaeAHAOXcZiJhZ\n6RrXPQX8jXNufOuKJyIiG0mla6UWOJ/0PpY4Nrriuo8AP7LRN5ubm4+Hw6GUCygiIgAE1juRUh/5\nRt/MzE4BV5xzK8N9laGhyU3dLBotIRYb29TXZIJsrTdkb91V7+yy2XpHoyXrnkula6ULrwW+qB7o\nXnHN+4Gvp1wiERHZMqkE+YvAkwBmdgLocs6t/DHyIHBhi8smIiIp2DDInXNngPNmdgb4OPCMmT1l\nZh9IuqwO6NumMoqIyB2k1EfunHtuxaELK84f37ISiYjIpmhmp4iIzynIRUR8TkEuIuJzvgnywdFb\n/PU3W5memU93UUREdhTfBPkPXIyvvtTOlfahdBdFRGRH8U2Qh4LehNKZuYU0l0REZGfxTZDnhL2i\nzs6pa0VEJJkPg1wtchGRZP4J8pCCXERkLf4J8sUW+byCXEQkmf+CXC1yEZFlFOQiIj6nIBcR8Tn/\nBLkedoqIrMk/Qa4WuYjImnwU5N6GzRq1IiKynI+CXC1yEZG1+CfI1UcuIrIm/wS51loREVmTb4I8\nGAwQCgbURy4isoJvghy8Vrm6VkRElguncpGZPQ+cBOLAs865c0nnmoDPAbnAK865f70dBQUFuYjI\nWjZskZvZY0Czc+4U8DTw8RWXfBT4qHPuIWDezHZtfTE9CnIRkdVS6Vo5DbwA4Jy7DETMrBTAzILA\no8AXE+efcc61b1NZyQkF1UcuIrJCKl0rtcD5pPexxLFRIAqMAc+b2QngO865X7vTN4tECgknJvek\nKhotAaAgP4fxW3NL7zNdttRzLdlad9U7u2xVvVPqI18hsOJ1A/Ax4DrwZTN7n3Puy+t98dDQ5KZu\nFo2WEIuNJW4WZ2Z2ful9Jkuud7bJ1rqr3tlls/W+U+in0rXShdcCX1QPdCde9wM3nHNXnXPzwDeA\noymXbJNyQl4feTwe365biIj4TipB/iLwJECi+6TLOTcG4JybA66ZWXPi2ncBbjsKCrcnBc3NK8hF\nRBZtGOTOuTPAeTM7gzdi5Rkze8rMPpC45JeBP0ucHwG+tF2FXVo4SyNXRESWpNRH7px7bsWhC0nn\nWoF3b2Wh1hPWvp0iIqv4a2ZnSOutiIis5Ksgz83RCogiIiv5Ksi1lK2IyGr+CnL1kYuIrOLLIJ9T\ni1xEZIkvg3xGQS4issRfQa4+chGRVfwV5NqAWURkFV8FeVhBLiKyiq+CXKNWRERW81eQh7TWiojI\nSv4K8rCm6IuIrOTTIFeLXERkka+CPFd95CIiq/gqyNUiFxFZzZdBrin6IiK3+SvINbNTRGQVfwW5\n1loREVnFl0GuFrmIyG3+DHKNWhERWeKrIA+rj1xEZJVwKheZ2fPASSAOPOucO5d07jpwE1icbvlh\n51zn1hbTEwgECIeCCnIRkSQbBrmZPQY0O+dOmdlh4FPAqRWX/bhzbnw7CrhSTlhBLiKSLJWuldPA\nCwDOuctAxMxKt7VUd5ATDqqPXEQkSSpdK7XA+aT3scSx0aRj/8vM9gDfBX7NORdf75tFIoWEw6FN\nFTIaLVl6nZ8bYiEeX3YsU2VDHdeTrXVXvbPLVtU7pT7yFQIr3v8W8DVgEK/l/iHg8+t98dDQ5KZu\nFo2WEIuNLb0PBQNMTM0uO5aJVtY7m2Rr3VXv7LLZet8p9FMJ8i68FviieqB78Y1z7s8XX5vZV4Dj\n3CHI71ZOSF0rIiLJUukjfxF4EsDMTgBdzrmxxPsyM/s7M8tNXPsY8Oa2lDRBDztFRJbbsEXunDtj\nZufN7AywADxjZk8BI865LyRa4S+Z2RTwKtvYGgcvyOfm4ywsxAkGV/byiIhkn5T6yJ1zz604dCHp\n3MeAj21loe4knDS7My+4uYemIiKZyFczO0ErIIqIrOS/INfCWSIiy/g3yDVyRUQE8GWQe/3iapGL\niHj8F+QhbfcmIpLMf0GuPnIRkWV8F+S5S0E+v8GVIiLZwXdBroedIiLL+S7Iw+paERFZxndBrj5y\nEZHl/BfkiVErMwpyERHAj0GuFrmIyDIKchERn/NvkGvUiogI4Mcg1+qHIiLL+C/IE2utaIq+iIjH\nh0GuFrmISDL/Bvm8puiLiIAPgzxXLXIRkWV8F+Saoi8islxKmy+b2fPASSAOPOucO7fGNb8PnHLO\nPb6lJVxBo1ZERJbbsEVuZo8Bzc65U8DTwMfXuOYI8J6tL95qi33kmqIvIuJJpWvlNPACgHPuMhAx\ns9IV13wU+PUtLtuaQsEAgYAmBImILEqla6UWOJ/0PpY4NgpgZk8B3wKup3LDSKSQcGIseKqi0ZJl\n73NzQmsezzSZXr87yda6q97ZZavqnVIf+QqBxRdmVgH8LPAE0JDKFw8NTW7qZtFoCbHY2LJj4WCA\nqVtzq45nkrXqnS2yte6qd3bZbL3vFPqpdK104bXAF9UD3YnXPwxEge8AXwBOJB6MbquccFAPO0VE\nElIJ8heBJwHM7ATQ5ZwbA3DOfd45d8Q5dxL4APCKc+5Xtq20CTnhoPrIRUQSNgxy59wZ4LyZncEb\nsfKMmT1lZh/Y9tKtIyccUotcRCQhpT5y59xzKw5dWOOa68Djd1+kjeWE1LUiIrLIdzM7QX3kIiLJ\nfBvkC/E48wsKcxER3wY5aJq+iAgoyEVEfE9BLiLic/4Mcq2AKCKyxJ9Brha5iMgSfwe5ZneKiPg8\nyNUiFxHxaZCrj1xEZIk/gzyxnrmCXETEt0GuPnIRkUX+DvK5+TSXREQk/Xwd5DOzapGLiPgyyIvy\nvdV3J27NprkkIiLp588gL8gBYGJqLs0lERFJP18GeXG+F+TjU2qRi4j4MsgXW+QKchERnwZ5YX6Y\nQADG1UcuIuLPIA8GAhTl5zChFrmIiD+DHLzuFQW5iAiEU7nIzJ4HTgJx4Fnn3Lmkc/8SeBqYBy4A\nzzjn4ttQ1mWKC8L0D08Rj8cJBALbfTsRkR1rwxa5mT0GNDvnTuEF9seTzhUCPw086px7BDgEnNqm\nsi5TnJ/D/EKcqWnN7hSR7JZK18pp4AUA59xlIGJmpYn3k86508652USolwE921baJEsjV/TAU0Sy\nXCpdK7XA+aT3scSx0cUDZvYc8CzwR865a3f6ZpFIIeHE6oWpikZLVh+rKAIgJy9nzfOZIFPrlYps\nrbvqnV22qt4p9ZGvsKpD2jn3B2b2MeArZvZd59z31vvioaHJTd0sGi0hFhtbdTyI1w3f0T1CpODt\nVGNnW6/e2SBb6656Z5fN1vtOoZ9K10oXXgt8UT3QDWBmFWb2HgDn3BTwVeCRlEt2F4o1KUhEBEgt\nyF8EngQwsxNAl3Nu8cdIDvBpMytOvH8IcFteyjUUL623oiAXkey2YZ+Ec+6MmZ03szPAAvCMmT0F\njDjnvmBmvwv8g5nN4Q0//OK2ljihOLEColrkIpLtUupcds49t+LQhaRznwY+vXVFSo1WQBQR8fh2\nZmexhh+KiAA+DnKtgCgi4vFtkOflhMgJBxXkIpL1fBvk4HWvaNSKiGQ7Xwd5UX6O9u0Ukazn6yAv\nLggzNT3P3PxCuosiIpI2Pg/yxBDEWxqCKCLZKyOCXA88RSSb+TrIizRNX0TE50Gerxa5iIivg1xd\nKyIiGRLkGoIoItksI4JcLXIRyWa+DvKixM5AetgpItnM13uk3W6Raxy5iOwcCwtxYsNTdPVP0DUw\nQWf/BLGhKZ54oImHj9Rs+f18HeSF2lxCRNJoIR6nf+QWnbFxuvq9wO6KTdA9OMns3PIZ5znhIDOz\n89tSDl8HeSgYpDAvrK4VEdlW8XicobFpOvsn6IxN0Nk/TmfMa23PzC4P7NxwkPrKIuqriqivKqSh\nqpj6qkKqygoIBlftXb8lfB3k4HWvqEUuIltl4tYsHX3jdMQm6IyN05EI76np5V244VCQ+spC6qNF\nNFR5wd1QVbStgb0e3wd5UUEOg2O3iMfjBAL39g9PRPxrdm6B7oEJbvZ5reuO2DgdsXGGx2eWXRcM\nBKipKODo3goaqopojBbREC0mWp5PKLgzxov4PsiLC3KYm48zPTtPfq7vqyMiW2yxW6QjNs7NREu7\no2+c7oFJFuLxZddGSvI4vq+SxmgRjdFiGqJF1FUWkhMOpan0qfF98hUX3H7gqSAXyW4zs/N09nut\n7I6+xeAeX7VCal5uiL31JTRVlyyFdmO0iMLEsh9+k1LymdnzwEkgDjzrnDuXdO69wO8D84ADPuKc\nu2cLhN9eOGuOqrJ7dVcRSbeRiRlu9o5xs2+c9r5x2nvH6BmcJLmRHQCqIwUc3h2hsbqYxmgxTdXF\nVJblE8ygrtgNg9zMHgOanXOnzOww8CngVNIlnwTe65zrMLO/Bn4M+Mq2lHYNmt0pktkW4nH6hqZo\n7x2jvXec9r4xbvaOMzKxvC87PzfE/oYymqqLl/5rrComL3dnd4tshVRa5KeBFwCcc5fNLGJmpc65\n0cT5dyW9jgGV21DOdWkFRJHMMTe/QGdsghu9Y0vBfbNvnOkV468rS/N4x4EqmqqL2VVTTFNNCVUZ\n1srejFSCvBY4n/Q+ljg2CrAY4mZWB/wI8Jt3+maRSCHhTT44iEZL1j1XX+OdC4SCd7zOjzKtPpuR\nrXXPpnrfmpnjevcoL3+vjasdw1ztHKG9Z5S5+dt9I8FggKbqYvY2lLG/oYy99WXsayijpDA3jSXf\nOlv1eb+dp4OrfuSZWTXwJeDfOOcG7vTFQ0OTm7pZNFpCLDa27vmFOe8ndXds/I7X+c1G9c5k2Vr3\nTK73rZk52nvHudEzxvUer7XdNTCxrD87JxykqbqY3bWl7KopZndNCQ1VReTmLG/43ZqY5tbE9D2u\nwdbb7Od9p9BPJci78Frgi+qB7sU3ZlYKfBX4defciymXaotUlOQBMDBy617fWkTWMD0zT3vfGNe7\nvdC+3jNKz8AkyQP98nJCHGgoY3dNCccPRokU5lBXWbhjxmX7TSpB/iLwO8AnzOwE0OWcS/4x8lHg\neefc17ajgBupKisgAPRtsqUvIndvdm6em30TtHWPcr1nlOs9Y3T1L29p5+eGONhUzu7aEvbUlrC7\ntoSaSOHS7MdM/k3kXtkwyJ1zZ8zsvJmdARaAZ8zsKWAE+DvgZ4BmM/tI4ks+65z75HYVeKWccJCK\n0nz6hqfu1S1FstLCQpyu/gmudY9yvXuUtu4xOmLjzC/cTu28nBDNDWXsqStdCu6aisKsfQh5r6TU\nR+6ce27FoQtJr/O2rjhvT3WkgMs3hpienScvJ/OHGolst3g8zsDoLdq6x2jrGuVa1wg3epePHgmH\ngkthvbeulD21JdRVFt3zdUYkA2Z2AkTLvSCPDU/RGC1Od3FEfGfy1hxt3V5gX+sapa17lNHJ20N6\nAwGorypib10p++pK2VtXSkO0iHBIfdo7QUYEeU2kAIDYkIJcZCMLC3E6+ye42jXCtc5RrnaN0D2w\n/BlTRWkeD1iUvfVecO+uLdESGDtYRnwy0XIvyNVPLrLa2OQMVxOBfbVzhLaeMaZnbneR5OWGOLSr\nnH31ZeyrL2VffSnlxWnvMZVNyIggr44oyEXg9gPJ1s6Rpf/6hpb/u6ivKmJffSkHGsrYV1dKfZX6\ntf0uI4J8qUU+pCCX7DI1Pce17lGudozQ0jnCta4RpqZvt7YL8sIc21vB/oYy9jd43SR+XeFP1pcR\nQV6QF6a0MIeYglwy3NDYNC0dw7TcHKGlc5ibfePLxmzXVhRy4mApzY3l7G8oo65SQ/+yQUYEOUA0\nUsD17jHmFxY0O0wywkI8TvfAJC03h73w7hihP2kGczgUZH9DGc0NZRxoLONABq1BIpuTMUFeXV7A\n1c5RBkanqU50tYj4yfz8Am3do7j228GdvKpnUX6YdxyoormxjOZGb6ZkTliNFsmgIL/dTz6pIBdf\nmJ1LBPfNYd666a3+dytpNEllaT7H91XQ3FhOc1O5uklkXRkT5DWRQsAbS87eNBdGZA0zs/Nc7RrF\ntQ/h2oe52jXK3PztzbSaakrYX1/KwcYyDjaVU1Gan8bSip9kTJBHNQRRdpjZuXlaO73gvtI+zLWu\nkaW1tgNAY3Ux1lSO7fJa3Pt3V2rxKHlbMibIqzUEUdJsbn6Ba12jXLkxxJX2IVo7b7e4A8CumhJs\nlxfcB5vKl3a3ErlbGRPkJYU55OWG1CKXe2ZhIc6N3jEu3xji8o0hWjqGmZm9HdxN1cUc2h3xwrup\nXOO3ZdtkTJAHAgFqygvoGZokHo8T0EMh2WLxeJyewUkuXR/i0vVBXPswk9NzS+cbqoo4tDvCoV1e\neC9uDC6y3TImyMHrJ2/v83bX1loRshVGxqeXgvvSjSGGxm5vMVZVls8Dh6Ic3l3Bod0Ryoo0hlvS\nI6OCPLmfXEEub8f07Dxv3RzmYtsgl64P0hGbWDpXUpjDQ4erObKngsO7I0tDXkXSLaOCfGnkytAU\nB5vK01wa8YN4PM7NvnEutg3yZtsgLR3DSyNLcsJBju6t4OieCo7sidBYXaxx3LIjZVSQ12g5W0nB\n6OQMlxLB/WbbIKMTM0vndtUUe8G9t4KDjWXkhLXjlOx8GRXktZVFAHTGxtNcEtlJFhbiXOsa5Y1r\nA7zZNsD17rGlHd1Li3L5oWO1Sy3vUvVziw9lVJBHSvKoKM2jpWNEI1ey3OjEDG9cG+CNawNcbBtk\n4pY3uiQUDGC7yjm2r5Jjeytoqi7W3xPxvZSC3MyeB04CceBZ59y5pHP5wCeAo865B7allJvQ3FjO\n2Uu99AxOUpdooUvmW4jHudEzxoXWft64NkBb9+0ZkhWleTxwqJrj+yo5vDtCQV5GtV9ENg5yM3sM\naHbOnTKzw8CngFNJl/wh8BpwdHuKuDnNjWWcvdRLa8eIgjzDTU3PcbFtkAtX+3nj2u2+7lAwwKFd\n5dy3v4rj+yqorypSq1syWipNk9PACwDOuctmFjGzUufcaOL8fwQqgQ9vUxk35UBDGQAtHSM8en99\nmksjW61vaJILrQNcuNqPax9mfsHr7S4tyuXdx+u4b38lR/ZUUJivVrdkj1T+ttcC55PexxLHRgGc\nc2NmVpnqDSORQsKbHAkQjZakfG1FZTGF+WHaekY39XU7kd/LfzcW6z6/EKelfYizF3s4e7GHm723\nu0wONJbx4JFaHjhcw4HG8ozYdzJbP3PV++68nWbLXf1rGRqa3NT10WjJpleE21dXypttg1y9PuDb\nUQhvp96ZoqSsgG+fa+fV1n5eb+1ndNLbXCEnHOT+/ZXc31zF/furiJTcnvQ1MOD/kUrZ+pmr3qlf\nv55UgrwLrwW+qB7oTvnuadDcWMabbYO0do5w4mA03cWRFIxNznChdYBXW2JcvD7EzKy3wUJpUS7v\nub+OdxyIcnhPhLwcjesWWSmVIH8R+B3gE2Z2Auhyzu3oH58HGr1ZnS0dwwryHWxg5BavvBXjlbdi\nvNUxvLSJcFNNMcf3VvLO5ir21pdqNqXIBjYMcufcGTM7b2ZngAXgGTN7Chhxzn3BzP4aaALMzL4J\nfNI599ntLPRG9tWVEgoGaOkYSWcxZA3dAxOcdzHOvxXjRs/t9sD+hlJONEd5R3MV9x2qzcpftUXe\nrpT6yJ1zz604dCHp3E9taYm2QF5uiF01JdzoGWN6dl6/jqdRPB6nvXec82/1cd7F6B7wnpGEggGO\n7q3gxMEo72yu0iJnInchY8doNTeW0dY9yvXuUWxXJN3FySrxeJy27jF+4Pr4wZU++kduAZAbDnLi\nYJR3HYxy/4FKbbQgskUyOshfPHeTlo4RBfk9sBD31jP5wZU+zrs+Bka9dbvzckM8dLiaB8ybWZmX\nq9+ORLZaxgb54gPPyzeGeP8P7UlvYTJUPBHe5670ce5K39KmCwV5YU4dreXBQ9Uc3RvRCoIi2yxj\ng7ysKJcDDWVcuTHE4OgtKkrz012kjBCPx7neM8bLl3v5wZXbLe+CvDCPHKvlgUPexgs54WCaSyqS\nPTI2yAEeOV5La+cI37/Yw/tO7Ul3cXxrcfOFly/3ce5KL7Fhr897MbwfTOyaEw4pvEXSIaOD/MFD\nNXz26y18940efuLkbi2ctEndAxOcvdTLy5f76Bn0Rpvk5YY4eaSGBw9Xc2xvpVreIjtARgd5YX6Y\nEwejnL3Uy9Wu0aUFtWR9/SNTvHy5j7OXernZ5017zw0HeeBQNQ8dqua+/ZXkajinyI6S0UEO8Mix\nWs5e6uXMG90K8nWMTsxw7ooX3q2d3iSqUDDA/fsrefhIDfcfqNIa3iI7WMb/6zyyp4Ly4lzOXu7j\np083qzWZMDU9x6stMV661MultiEW4nECwOHdER4+UsOJg1GKCzTOW8QPMj7Ig8EAp47V8tWX2nmt\ntZ+HDteku0hpMzu3wJvXBnjpUi+vtfYzO7cAwN66Uh4+UsODh6qXrSgoIv6Q8UEO8MixOr76Ujvf\neq0r64J8IR6n5eYw37/oDRecnPb2rqytKOTkkRoePlpDTaQwzaUUkbuRFUFeX1XE4d0RLt8Y4o1r\nAxzfl/I+GL60OFzwpUu9nL3UuzRRp6w4lx+5r4mTR2vYXVOiUTwiGSIrghzgp08389t/9jKf+3oL\nh5+OZOSY5/7hKc5e7uWli7109k8AUJAX4t331XHqSA22K5IRu+iIyHJZE+RN1cU8/s4G/uGVTr5x\nvoMffWhXuou0JcYmZ/jBlT6+n9hwGiAcCnDiYJSTR2q4/0ClpsiLZLisCXKADzy6j5cv9fLF77Vx\n8mgtZT7dBu7WzByvtvRz9lIvF9sGmV/wRpwc2lXOyaO1vMuiFGllQZGskVVBXlyQwz9+dB+f+X9v\n8flvtvL0+46ku0gpWxxxcvZyL6+19DOTGHGyu6aEk0dreOhwjUaciGSprApygMffWc+3L3TxvTd6\n2FVdwj96sCndRVrX3PwCl28M8fLlXl55q5+pxIiTmopCHj5czcNHaqirLEpzKUUk3bIuyEPBIP/2\ng8f5L39xns99o4WSwhxOHq3d+Avvkbn5Ba7cGOKNv2/lzOtdTNzywruiNI/H7q/noSPVGnEiIstk\nXZADVJUX8Kv/5B38wWde4U+/fJniwhyO7U3fkMTp2XkuXR/kFRfj1Zb+pbHeZcW5PPGuRh44VM2B\nxjJtQiwia8rKIAdvFMsvfeg4H/2rC3z882/w5GP7eOLBpnsWliPj07x+bYDXWvq52Da41OcdKcnj\nh47X8sTDe6gqzlF4i8iGsjbIAWxXhF968jh/8qVL/OXft/Jaaz8/977DVJUVbPm95uYXuNo5wsXr\n3qSk5B3k6yoLeWeztwnx3vpSgoEA0WiJdpIXkZSkFORm9jxwEogDzzrnziWdewL4r8A88BXn3O9t\nR0G3y7G9lfze0w/zf752hVdb+vmNPznLw0dqeO+JBvbUlr7t7zs9M8+17lFaOoZp7RjhrY5hZma9\nVncoGODQrnLu21/F/Qcq9cBSRO7KhkFuZo8Bzc65U2Z2GPgUcCrpko8DPwp0At8ys79xzl3altJu\nk9KiXH7xg8c582YPf/vdNr7zejffeb2bXdXFHGwqZ09dCbtrSigpzKUgL0xOOMjCQpzp2XmmpucY\nHJumf3iK2PAUnf0TtPeO0zs4STzpHnWVhRzZU8GRPREO7YpoWVgR2TKppMlp4AUA59xlM4uYWalz\nbtTM9gGDzrmbAGb2lcT1vgpygEAgwCPH6zh1tJY32wb51mudXGgdoD2xuUKyUDDA/EJ8je/iKcgL\nL/0AaG4s50BDGaU+nXwkIjtfKkFeC5xPeh9LHBtN/D+WdK4P2H+nbxaJFBLe5JTxaLRkU9ffrZqa\nUk6f3MOtmTmud43ScnOYGz2jjE/OMnFrlqnpOXLDIfLzQhTkhqkoy6e2opCayiKaakqojhRsyfDA\ne13vnSRb6656Z5etqvfb+f3+Tgm1YXoNDU1u6mbpfuhXWZRD5aEoJw9FU/uC+Xn6+1e34jcr3fVO\np2ytu+qdXTZb7zuFfipLAHbhtbwX1QPd65xrSBwTEZF7JJUgfxF4EsDMTgBdzrkxAOfcdaDUzPaY\nWRh4f+J6ERG5RzbsWnHOnTGz82Z2BlgAnjGzp4AR59wXgF8APpe4/K+cc29tW2lFRGSVlPrInXPP\nrTh0Ienct1k+HFFERO6hzNsmR0QkyyjIRUR8TkEuIuJzCnIREZ8LxOPrTzUXEZGdTy1yERGfU5CL\niPicglxExOcU5CIiPqcgFxHxOQW5iIjPKchFRHxuR28ceadNnzONmf034FG8z+T3gXPA/wVCeOu/\n/3Pn3HT6Srh9zKwAeBP4PeAbZE+9Pwz8B2AO+C3gdTK87mZWDPw5EAHygN8BeoD/iffv/HXn3C+k\nr4Rby8yOAX8LPO+c++9m1sQan3Hi78Iv460w+0nn3J9u5j47tkWevOkz8DTeJs8ZyczeCxxL1PXH\ngD8Cfhf4H865R4FW4OfSWMTt9hvAYOJ1VtTbzCqB/wS8G28d/58kO+r+FOCcc+/F2+fgY3h/3591\nzj0ClJnZj6exfFvGzIqAP8ZrnCxa9Rknrvst4AngceBXzKxiM/fasUHOik2fgYiZlaa3SNvm28BP\nJV4PA0V4H+gXE8e+hPchZxwzOwQcAb6cOPQ4WVBvvHp93Tk35pzrds79PNlR936gMvE6gvcDfG/S\nb9uZVO9p4CdYvmva46z+jB9ULnivAAACKUlEQVQGzjnnRpxzU8D3gEc2c6OdHOQrN3Ze3PQ54zjn\n5p1zE4m3TwNfAYqSfq3uA+rSUrjt91HgV5PeZ0u99wCFZvZFM/uOmZ0mC+runPtLYJeZteI1YP4d\nMJR0ScbU2zk3lwjmZGt9xmttYr+pP4OdHOQr3f229Ducmf0kXpD/4opTGVl3M/sZ4PvOubZ1LsnI\neicE8FqmH8Trbvgzltc3I+tuZv8MaHfOHQB+GPiLFZdkZL3XsV5dN/1nsJOD/E6bPmccM/tR4NeB\nH3fOjQDjiYeAkLmbWr8P+Ekzewn4CPCbZEe9AXqBM4lW21VgDBjLgro/AvwdgHPuAlAAVCWdz9R6\nL1rr7/ddb2K/k4N83U2fM42ZlQF/CLzfObf40O/rwIcSrz8EfC0dZdtOzrl/6px70Dl3EvjfeKNW\nMr7eCS8CP2xmwcSDz2Kyo+6teH3CmNluvB9gl83s3YnzHyQz671orc/4LPCgmZUnRvU8AnxnM990\nRy9ja2Z/ALyHxKbPiZ/gGcfMfh74bSB54+p/gRdu+cAN4Gedc7P3vnT3hpn9NnAdr7X252RBvc3s\nX+F1pQH8Z7whpxld90RQfQqowRtq+5t4ww8/gdewPOuc+9X1v4N/mNm78J4B7QFmgU7gw8CnWfEZ\nm9mTwL/HG4L5x865z2zmXjs6yEVEZGM7uWtFRERSoCAXEfE5BbmIiM8pyEVEfE5BLiLicwpyERGf\nU5CLiPjc/wfdnkpfFHoNSQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f18186b34a8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XoyfFgLoUgRW"
      },
      "source": [
        "### Memory Optimization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVKyONFsUgRW"
      },
      "source": [
        "def loss_backprop(generator, criterion, out, targets, normalize):\n",
        "    \"\"\"\n",
        "    Memory optmization. Compute each timestep separately and sum grads.\n",
        "    \"\"\"\n",
        "    assert out.size(1) == targets.size(1)\n",
        "    total = 0.0\n",
        "    out_grad = []\n",
        "    for i in range(out.size(1)):\n",
        "        out_column = Variable(out[:, i].data, requires_grad=True)\n",
        "        gen = generator(out_column)\n",
        "        loss = criterion(gen, targets[:, i]) / normalize\n",
        "        total += loss.data[0]\n",
        "        loss.backward()\n",
        "        out_grad.append(out_column.grad.data.clone())\n",
        "    out_grad = torch.stack(out_grad, dim=1)\n",
        "    out.backward(gradient=out_grad)\n",
        "    return total"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LiTlbMq2UgRY"
      },
      "source": [
        "def make_std_mask(src, tgt, pad):\n",
        "    src_mask = (src != pad).unsqueeze(-2)\n",
        "    tgt_mask = (tgt != pad).unsqueeze(-2)\n",
        "    tgt_mask = tgt_mask & Variable(subsequent_mask(tgt.size(-1)).type_as(tgt_mask.data))\n",
        "    return src_mask, tgt_mask"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sSF9AaKJUgRZ"
      },
      "source": [
        "def train_epoch(train_iter, model, criterion, opt, transpose=False):\n",
        "    model.train()\n",
        "    for i, batch in enumerate(train_iter):\n",
        "        src, trg, src_mask, trg_mask = \\\n",
        "            batch.src, batch.trg, batch.src_mask, batch.trg_mask\n",
        "        out = model.forward(src, trg[:, :-1], src_mask, trg_mask[:, :-1, :-1])\n",
        "        loss = loss_backprop(model.generator, criterion, out, trg[:, 1:], batch.ntokens) \n",
        "                        \n",
        "        model_opt.step()\n",
        "        model_opt.optimizer.zero_grad()\n",
        "        if i % 10 == 1:\n",
        "            print(i, loss, model_opt._rate)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q4kFYs7nUgRb"
      },
      "source": [
        "def valid_epoch(valid_iter, model, criterion, transpose=False):\n",
        "    model.test()\n",
        "    total = 0\n",
        "    for batch in valid_iter:\n",
        "        src, trg, src_mask, trg_mask = \\\n",
        "            batch.src, batch.trg, batch.src_mask, batch.trg_mask\n",
        "        out = model.forward(src, trg[:, :-1], src_mask, trg_mask[:, :-1, :-1])\n",
        "        loss = loss_backprop(model.generator, criterion, out, trg[:, 1:], batch.ntokens) \n",
        "        "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uJo5AZasUgRd"
      },
      "source": [
        "class Batch:\n",
        "    def __init__(self, src, trg, src_mask, trg_mask, ntokens):\n",
        "        self.src = src\n",
        "        self.trg = trg\n",
        "        self.src_mask = src_mask\n",
        "        self.trg_mask = trg_mask\n",
        "        self.ntokens = ntokens\n",
        "    \n",
        "def data_gen(V, batch, nbatches):\n",
        "    for i in range(nbatches):\n",
        "        data = torch.from_numpy(np.random.randint(1, V, size=(batch, 10)))\n",
        "        src = Variable(data, requires_grad=False)\n",
        "        tgt = Variable(data, requires_grad=False)\n",
        "        src_mask, tgt_mask = make_std_mask(src, tgt, 0)\n",
        "        yield Batch(src, tgt, src_mask, tgt_mask, (tgt[1:] != 0).data.sum())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vrpU5b2sUgRe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 604
        },
        "outputId": "00871f60-2fcb-4235-c0ab-8de02a0c069c"
      },
      "source": [
        "V = 11\n",
        "criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)\n",
        "model = make_model(V, V, N=2)\n",
        "model_opt = get_std_opt(model)\n",
        "for epoch in range(2):\n",
        "    train_epoch(data_gen(V, 30, 20), model, criterion, model_opt)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 2.9646920561790466 6.987712429686844e-07\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-41-347a415cdfd4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mmodel_opt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_std_opt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mepoch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m15\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m     \u001b[0mtrain_epoch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata_gen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mV\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m30\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m20\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel_opt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-38-1735085d52f5>\u001b[0m in \u001b[0;36mtrain_epoch\u001b[0;34m(train_iter, model, criterion, opt, transpose)\u001b[0m\n\u001b[1;32m      6\u001b[0m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloss_backprop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mntokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m         \u001b[0mmodel_opt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m         \u001b[0mmodel_opt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mi\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;36m10\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-26-cac760f2f089>\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     16\u001b[0m             \u001b[0mp\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'lr'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_rate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m     70\u001b[0m                 \u001b[0mexp_avg_sq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmul_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbeta2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddcmul_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeta2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 72\u001b[0;31m                 \u001b[0mdenom\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mexp_avg_sq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'eps'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     73\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m                 \u001b[0mbias_correction1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mbeta1\u001b[0m \u001b[0;34m**\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'step'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5S2BcIoOUgRg"
      },
      "source": [
        "# A Real World Example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aP_oq0kLUgRh"
      },
      "source": [
        "# For data loading.\n",
        "from torchtext import data, datasets"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eKynVliVXg6F",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 989
        },
        "outputId": "f471f71a-d329-4fa9-d188-b4dcc9283234"
      },
      "source": [
        "!pip install torchtext spacy\n",
        "!python -m spacy download en\n",
        "!python -m spacy download de"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.6/dist-packages\r\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.6/dist-packages\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from torchtext)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from torchtext)\n",
            "Requirement already satisfied: preshed<2.0.0,>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: murmurhash<0.29,>=0.28 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: numpy>=1.7 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: cymem<1.32,>=1.30 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: msgpack-python==0.5.4 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: html5lib==1.0b8 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: thinc<6.11.0,>=6.10.1 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: pathlib in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: ujson>=1.35 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: dill<0.3,>=0.2 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: msgpack-numpy==0.4.1 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: regex==2017.4.5 in /usr/local/lib/python3.6/dist-packages (from spacy)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext)\n",
            "Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->torchtext)\n",
            "Requirement already satisfied: cytoolz<0.9,>=0.8 in /usr/local/lib/python3.6/dist-packages (from thinc<6.11.0,>=6.10.1->spacy)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.6/dist-packages (from thinc<6.11.0,>=6.10.1->spacy)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.6/dist-packages (from thinc<6.11.0,>=6.10.1->spacy)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from ftfy<5.0.0,>=4.4.2->spacy)\n",
            "Requirement already satisfied: toolz>=0.8.0 in /usr/local/lib/python3.6/dist-packages (from cytoolz<0.9,>=0.8->thinc<6.11.0,>=6.10.1->spacy)\n",
            "Collecting https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)\n",
            "\u001b[K    6% |██                              | 2.4MB 47.2MB/s eta 0:00:01"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "\u001b[K    100% |████████████████████████████████| 37.4MB 50.2MB/s \n",
            "\u001b[?25hInstalling collected packages: en-core-web-sm\n",
            "  Running setup.py install for en-core-web-sm ... \u001b[?25l-\b \b\\\b \b|\b \bdone\n",
            "\u001b[?25hSuccessfully installed en-core-web-sm-2.0.0\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/en_core_web_sm -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/en\n",
            "\n",
            "    You can now load the model via spacy.load('en')\n",
            "\n",
            "Collecting https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-2.0.0/de_core_news_sm-2.0.0.tar.gz (38.2MB)\n",
            "\u001b[K    100% |████████████████████████████████| 38.2MB 50.0MB/s \n",
            "\u001b[?25hInstalling collected packages: de-core-news-sm\n",
            "  Running setup.py install for de-core-news-sm ... \u001b[?25l-\b \b\\\b \b|\b \bdone\n",
            "\u001b[?25hSuccessfully installed de-core-news-sm-2.0.0\n",
            "\n",
            "\u001b[93m    Linking successful\u001b[0m\n",
            "    /usr/local/lib/python3.6/dist-packages/de_core_news_sm -->\n",
            "    /usr/local/lib/python3.6/dist-packages/spacy/data/de\n",
            "\n",
            "    You can now load the model via spacy.load('de')\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lXtYwdHqUgRj",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 395
        },
        "outputId": "31ee7819-a5eb-4cf2-cb06-cc7932bf535a"
      },
      "source": [
        "# Load words from IWSLT\n",
        "\n",
        "#!pip install torchtext spacy\n",
        "#!python -m spacy download en\n",
        "#!python -m spacy download de\n",
        "\n",
        "import spacy\n",
        "spacy_de = spacy.load('de')\n",
        "spacy_en = spacy.load('en')\n",
        "\n",
        "def tokenize_de(text):\n",
        "    return [tok.text for tok in spacy_de.tokenizer(text)]\n",
        "\n",
        "def tokenize_en(text):\n",
        "    return [tok.text for tok in spacy_en.tokenizer(text)]\n",
        "\n",
        "BOS_WORD = '<s>'\n",
        "EOS_WORD = '</s>'\n",
        "BLANK_WORD = \"<blank>\"\n",
        "SRC = data.Field(tokenize=tokenize_de, pad_token=BLANK_WORD)\n",
        "TGT = data.Field(tokenize=tokenize_en, init_token = BOS_WORD, \n",
        "                 eos_token = EOS_WORD, pad_token=BLANK_WORD)\n",
        "\n",
        "MAX_LEN = 100\n",
        "train, val, test = datasets.IWSLT.splits(exts=('.de', '.en'), fields=(SRC, TGT), \n",
        "                                         filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and \n",
        "                                         len(vars(x)['trg']) <= MAX_LEN)\n",
        "MIN_FREQ = 1\n",
        "SRC.build_vocab(train.src, min_freq=MIN_FREQ)\n",
        "TGT.build_vocab(train.trg, min_freq=MIN_FREQ)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "downloading de-en.tgz\n",
            ".data/iwslt/de-en/IWSLT16.TEDX.dev2012.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2013.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2014.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2012.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2010.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.dev2010.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.dev2010.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TEDX.tst2013.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TEDX.dev2012.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2013.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TEDX.tst2013.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TEDX.tst2014.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2014.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2011.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2011.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2010.de-en.de.xml\n",
            ".data/iwslt/de-en/IWSLT16.TEDX.tst2014.de-en.en.xml\n",
            ".data/iwslt/de-en/IWSLT16.TED.tst2012.de-en.de.xml\n",
            ".data/iwslt/de-en/train.tags.de-en.en\n",
            ".data/iwslt/de-en/train.tags.de-en.de\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F8MTIJTWUgRl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 244
        },
        "outputId": "bd6c163a-14a2-4f8c-b0d5-01a033f278c4"
      },
      "source": [
        "# Detail. Batching seems to matter quite a bit. \n",
        "# This is temporary code for dynamic batching based on number of tokens.\n",
        "# This code should all go away once things get merged in this library.\n",
        "\n",
        "BATCH_SIZE = 4096\n",
        "global max_src_in_batch, max_tgt_in_batch\n",
        "def batch_size_fn(new, count, sofar):\n",
        "    \"Keep augmenting batch and calculate total number of tokens + padding.\"\n",
        "    global max_src_in_batch, max_tgt_in_batch\n",
        "    if count == 1:\n",
        "        max_src_in_batch = 0\n",
        "        max_tgt_in_batch = 0\n",
        "    max_src_in_batch = max(max_src_in_batch,  len(new.src))\n",
        "    max_tgt_in_batch = max(max_tgt_in_batch,  len(new.trg) + 2)\n",
        "    src_elements = count * max_src_in_batch\n",
        "    tgt_elements = count * max_tgt_in_batch\n",
        "    return max(src_elements, tgt_elements)\n",
        "\n",
        "class MyIterator(data.Iterator):\n",
        "    def create_batches(self):\n",
        "        if self.train:\n",
        "            def pool(d, random_shuffler):\n",
        "                for p in data.batch(d, self.batch_size * 100):\n",
        "                    p_batch = data.batch(\n",
        "                        sorted(p, key=self.sort_key),\n",
        "                        self.batch_size, self.batch_size_fn)\n",
        "                    for b in random_shuffler(list(p_batch)):\n",
        "                        yield b\n",
        "            self.batches = pool(self.data(), self.random_shuffler)\n",
        "            \n",
        "        else:\n",
        "            self.batches = []\n",
        "            for b in data.batch(self.data(), self.batch_size,\n",
        "                                          self.batch_size_fn):\n",
        "                self.batches.append(sorted(b, key=self.sort_key))\n",
        "\n",
        "def rebatch(pad_idx, batch):\n",
        "    \"Fix order in torchtext to match ours\"\n",
        "    src, trg = batch.src.transpose(0, 1), batch.trg.transpose(0, 1)\n",
        "    src_mask, trg_mask = make_std_mask(src, trg, pad_idx)\n",
        "    return Batch(src, trg, src_mask, trg_mask, (trg[1:] != pad_idx).data.sum())\n",
        "\n",
        "train_iter = MyIterator(train, batch_size=BATCH_SIZE, device=0,\n",
        "                        repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),\n",
        "                        batch_size_fn=batch_size_fn, train=True)\n",
        "valid_iter = MyIterator(val, batch_size=BATCH_SIZE, device=0,\n",
        "                        repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),\n",
        "                        batch_size_fn=batch_size_fn, train=False)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-a98b1f496512>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc_elements\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_elements\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0;32mclass\u001b[0m \u001b[0mMyIterator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mIterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcreate_batches\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'data' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wamR3SPdUgRo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1900
        },
        "outputId": "b3b92f29-2560-4dbb-c2b0-f2a0b71db37c"
      },
      "source": [
        "# Create the model an load it onto our GPU.\n",
        "pad_idx = TGT.vocab.stoi[\"<blank>\"]\n",
        "model = make_model(len(SRC.vocab), len(TGT.vocab), N=6)\n",
        "model_opt = get_std_opt(model)\n",
        "model.cuda()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "AssertionError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-48-b72405416f0a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmake_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mSRC\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTGT\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mN\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m6\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mmodel_opt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_std_opt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36mcuda\u001b[0;34m(self, device)\u001b[0m\n\u001b[1;32m    214\u001b[0m             \u001b[0mModule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m         \"\"\"\n\u001b[0;32m--> 216\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    217\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    218\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    144\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    145\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchildren\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 146\u001b[0;31m             \u001b[0mmodule\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    147\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mparam\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_parameters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    150\u001b[0m                 \u001b[0;31m# Variables stored in modules are graph leaves, and we don't\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    151\u001b[0m                 \u001b[0;31m# want to create copy nodes, so we have to unpack the data.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 152\u001b[0;31m                 \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    153\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_grad\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    154\u001b[0m                     \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_grad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_grad\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(t)\u001b[0m\n\u001b[1;32m    214\u001b[0m             \u001b[0mModule\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    215\u001b[0m         \"\"\"\n\u001b[0;32m--> 216\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mlambda\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    217\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    218\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/_utils.py\u001b[0m in \u001b[0;36m_cuda\u001b[0;34m(self, device, async)\u001b[0m\n\u001b[1;32m     67\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m             \u001b[0mnew_type\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 69\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mnew_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0masync\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     70\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py\u001b[0m in \u001b[0;36m_lazy_new\u001b[0;34m(cls, *args, **kwargs)\u001b[0m\n\u001b[1;32m    356\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mstaticmethod\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    357\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_lazy_new\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 358\u001b[0;31m     \u001b[0m_lazy_init\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    359\u001b[0m     \u001b[0;31m# We need this method only for lazy init, so we can remove it\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m     \u001b[0;32mdel\u001b[0m \u001b[0m_CudaBase\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__new__\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py\u001b[0m in \u001b[0;36m_lazy_init\u001b[0;34m()\u001b[0m\n\u001b[1;32m    118\u001b[0m         raise RuntimeError(\n\u001b[1;32m    119\u001b[0m             \"Cannot re-initialize CUDA in forked subprocess. \" + msg)\n\u001b[0;32m--> 120\u001b[0;31m     \u001b[0m_check_driver\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    121\u001b[0m     \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cuda_init\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    122\u001b[0m     \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cuda_sparse_init\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py\u001b[0m in \u001b[0;36m_check_driver\u001b[0;34m()\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[0mFound\u001b[0m \u001b[0mno\u001b[0m \u001b[0mNVIDIA\u001b[0m \u001b[0mdriver\u001b[0m \u001b[0mon\u001b[0m \u001b[0myour\u001b[0m \u001b[0msystem\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mPlease\u001b[0m \u001b[0mcheck\u001b[0m \u001b[0mthat\u001b[0m \u001b[0myou\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     61\u001b[0m \u001b[0mhave\u001b[0m \u001b[0man\u001b[0m \u001b[0mNVIDIA\u001b[0m \u001b[0mGPU\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0minstalled\u001b[0m \u001b[0ma\u001b[0m \u001b[0mdriver\u001b[0m \u001b[0;32mfrom\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 62\u001b[0;31m http://www.nvidia.com/Download/index.aspx\"\"\")\n\u001b[0m\u001b[1;32m     63\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     64\u001b[0m             \u001b[0;31m# TODO: directly link to the alternative bin that needs install\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mAssertionError\u001b[0m: \nFound no NVIDIA driver on your system. Please check that you\nhave an NVIDIA GPU and installed a driver from\nhttp://www.nvidia.com/Download/index.aspx"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SStCCZoiUgRp",
        "outputId": "a551e444-2fe5-4cd5-f46a-3de3505a2545"
      },
      "source": [
        "\n",
        "criterion = LabelSmoothing(size=len(TGT.vocab), padding_idx=pad_idx, smoothing=0.1)\n",
        "criterion.cuda()\n",
        "for epoch in range(15):\n",
        "    train_epoch((rebatch(pad_idx, b) for b in train_iter), model, criterion, model_opt)\n",
        "    valid_epoch((rebatch(pad_idx, b) for b in valid_iter), model, criterion)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 9.299771845340729 6.987712429686844e-07\n",
            "11 9.415135336574167 4.192627457812107e-06\n",
            "21 8.813630282878876 7.686483672655528e-06\n",
            "31 9.112178653478622 1.118033988749895e-05\n",
            "41 8.607461810112 1.4674196102342371e-05\n",
            "51 8.913826749660075 1.8168052317185794e-05\n",
            "61 8.701497752219439 2.1661908532029216e-05\n",
            "71 8.373274087905884 2.515576474687264e-05\n",
            "81 8.454237446188927 2.8649620961716057e-05\n",
            "91 7.6996782422065735 3.214347717655948e-05\n",
            "101 8.037408232688904 3.56373333914029e-05\n",
            "111 7.704962134361267 3.913118960624633e-05\n",
            "121 7.699015600606799 4.262504582108975e-05\n",
            "131 7.367554426193237 4.611890203593317e-05\n",
            "141 7.2071177661418915 4.961275825077659e-05\n",
            "151 7.106400920893066 5.310661446562001e-05\n",
            "161 6.804656069725752 5.660047068046343e-05\n",
            "171 6.390337720513344 6.0094326895306855e-05\n",
            "181 5.687528342008591 6.358818311015028e-05\n",
            "191 6.122820109128952 6.70820393249937e-05\n",
            "201 5.829070374369621 7.057589553983712e-05\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1NO9lsw2UgRt"
      },
      "source": [
        "\n",
        "OTHER"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B8BVm-hEUgRw"
      },
      "source": [
        "BOS_WORD = '<s>'\n",
        "EOS_WORD = '</s>'\n",
        "BLANK_WORD = \"<blank>\"\n",
        "SRC = data.Field()\n",
        "TGT = data.Field(init_token = BOS_WORD, eos_token = EOS_WORD, pad_token=BLANK_WORD) # only target needs BOS/EOS\n",
        "\n",
        "MAX_LEN = 100\n",
        "train = datasets.TranslationDataset(path=\"/n/home00/srush/Data/baseline-1M_train.tok.shuf\", \n",
        "                                    exts=('.en', '.fr'),\n",
        "                                    fields=(SRC, TGT), \n",
        "                                    filter_pred=lambda x: len(vars(x)['src']) <= MAX_LEN and \n",
        "                                         len(vars(x)['trg']) <= MAX_LEN)\n",
        "SRC.build_vocab(train.src, max_size=50000)\n",
        "TGT.build_vocab(train.trg, max_size=50000)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eFdZyOIzUgRx",
        "outputId": "bc543dba-c343-47bc-e81e-f74a25688668"
      },
      "source": [
        "pad_idx = TGT.vocab.stoi[\"<blank>\"]\n",
        "print(pad_idx)\n",
        "model = make_model(len(SRC.vocab), len(TGT.vocab), pad_idx, N=6)\n",
        "model_opt = get_opt(model)\n",
        "model.cuda()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "EncoderDecoder(\n",
              "  (encoder): Encoder(\n",
              "    (layers): ModuleList(\n",
              "      (0): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (1): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (2): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (3): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (4): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (5): EncoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (norm): LayerNorm(\n",
              "    )\n",
              "  )\n",
              "  (decoder): Decoder(\n",
              "    (layers): ModuleList(\n",
              "      (0): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (1): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (2): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (3): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (4): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (5): DecoderLayer(\n",
              "        (self_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (src_attn): MultiHeadedAttention(\n",
              "          (linears): ModuleList(\n",
              "            (0): Linear(in_features=512, out_features=512)\n",
              "            (1): Linear(in_features=512, out_features=512)\n",
              "            (2): Linear(in_features=512, out_features=512)\n",
              "            (3): Linear(in_features=512, out_features=512)\n",
              "          )\n",
              "        )\n",
              "        (feed_forward): PositionwiseFeedForward(\n",
              "          (w_1): Linear(in_features=512, out_features=2048)\n",
              "          (w_2): Linear(in_features=2048, out_features=512)\n",
              "          (dropout): Dropout(p=0.1)\n",
              "        )\n",
              "        (sublayer): ModuleList(\n",
              "          (0): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (1): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "          (2): SublayerConnection(\n",
              "            (norm): LayerNorm(\n",
              "            )\n",
              "            (dropout): Dropout(p=0.1)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (norm): LayerNorm(\n",
              "    )\n",
              "  )\n",
              "  (src_embed): Sequential(\n",
              "    (0): Embeddings(\n",
              "      (lut): Embedding(50002, 512)\n",
              "    )\n",
              "    (1): PositionalEncoding(\n",
              "      (dropout): Dropout(p=0.1)\n",
              "    )\n",
              "  )\n",
              "  (tgt_embed): Sequential(\n",
              "    (0): Embeddings(\n",
              "      (lut): Embedding(50004, 512)\n",
              "    )\n",
              "    (1): PositionalEncoding(\n",
              "      (dropout): Dropout(p=0.1)\n",
              "    )\n",
              "  )\n",
              "  (generator): Generator(\n",
              "    (proj): Linear(in_features=512, out_features=50004)\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cinuTkbtUgRz"
      },
      "source": [
        "criterion = LabelSmoothing(size=len(TGT.vocab), padding_idx=pad_idx, label_smoothing=0.1)\n",
        "criterion.cuda()\n",
        "for epoch in range(15):\n",
        "    train_epoch(train_iter, model, criterion, model_opt)\n",
        "    valid_epoch()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PsoeJn4bUgR1"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4LadFBIEUgR3",
        "outputId": "ba9a812f-b5f6-4972-af91-215d91a223d1"
      },
      "source": [
        "print(pad_idx)\n",
        "print(len(SRC.vocab))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1\n",
            "50002\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kP_Au0bHUgR7",
        "outputId": "d5ad5d88-d512-4947-b5b8-f18eaba4f9ff"
      },
      "source": [
        "torch.save(model, \"/n/rush_lab/trans_ipython.pt\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type EncoderDecoder. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type Encoder. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type EncoderLayer. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type MultiHeadedAttention. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type PositionwiseFeedForward. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type SublayerConnection. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type LayerNorm. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type Decoder. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type DecoderLayer. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type Embeddings. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type PositionalEncoding. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/torch/serialization.py:158: UserWarning: Couldn't retrieve source code for container of type Generator. It won't be checked for correctness upon loading.\n",
            "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nqKKIhoOUgR-",
        "outputId": "2c087978-9803-4639-886b-88df91e62096"
      },
      "source": [
        "#weight = torch.ones(len(TGT.vocab))\n",
        "#weight[pad_idx] = 0\n",
        "#criterion = nn.NLLLoss(size_average=False, weight=weight.cuda())\n",
        "criterion = LabelSmoothing(size=len(TGT.vocab), padding_idx=pad_idx, label_smoothing=0.1)\n",
        "criterion.cuda()\n",
        "for epoch in range(15):\n",
        "    train_epoch(train_iter, model, criterion, model_opt)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1 3.269582842476666 0.0005377044714644026\n",
            "101 3.300532897672383 0.0005726430336128369\n",
            "201 3.3047672072425485 0.0006075815957612711\n",
            "301 2.7151080842595547 0.0006425201579097052\n",
            "401 2.6975380268413574 0.0006774587200581396\n",
            "501 3.051631387323141 0.0007123972822065737\n",
            "601 2.554425454698503 0.000747335844355008\n",
            "701 2.6254820519825444 0.0007822744065034422\n",
            "801 2.868743653933052 0.0008172129686518764\n",
            "901 2.5978208642918617 0.0008521515308003106\n",
            "1001 2.5955790174775757 0.0008870900929487448\n",
            "1101 2.6764775353949517 0.000922028655097179\n",
            "1201 2.464000296778977 0.0009569672172456132\n",
            "1301 2.0503073083236814 0.0009919057793940475\n",
            "1401 2.295472824771423 0.0010268443415424816\n",
            "1501 2.245281406212598 0.0010617829036909158\n",
            "1601 2.2577588511630893 0.00109672146583935\n",
            "1701 2.2232908592559397 0.0011316600279877844\n",
            "1801 2.357596361427568 0.0011665985901362186\n",
            "1901 2.121352154412307 0.0012015371522846527\n",
            "2001 2.5742998471250758 0.001236475714433087\n",
            "2101 2.2518509055953473 0.0012714142765815214\n",
            "2201 2.2251326659170445 0.0013063528387299553\n",
            "2301 2.078994876006618 0.0013412914008783896\n",
            "2401 2.068276036065072 0.001376229963026824\n",
            "2501 2.31435151558253 0.0013907788851585368\n",
            "2601 1.9106871648691595 0.0013738752565588634\n",
            "2701 2.183084836578928 0.0013575733592730722\n",
            "2801 2.4668076275847852 0.0013418383196400342\n",
            "2901 1.963176985620521 0.0013266380295186675\n",
            "3001 2.2140520309330896 0.0013119428705609764\n",
            "3101 2.6989458349489723 0.0012977254713568687\n",
            "3201 2.1293521663174033 0.0012839604929174666\n",
            "3301 2.1402786187827587 0.0012706244386700126\n",
            "3401 2.041781216394156 0.0012576954857216498\n",
            "3501 2.051893091876991 0.0012451533346344698\n",
            "3601 1.5498304846696556 0.001232979075358713\n",
            "3701 2.763939742697403 0.001221155067309524\n",
            "3801 2.7611468499198963 0.0012096648318570434\n",
            "3901 1.7321470333263278 0.0011984929557393293\n",
            "4001 2.139603299088776 0.0011876250041103701\n",
            "4101 2.1966493157087825 0.0011770474421074978\n",
            "4201 2.0962203710805625 0.0011667475639689723\n",
            "4301 1.9717675620922819 0.0011567134288575545\n",
            "4401 2.097687987901736 0.0011469338026529508\n",
            "4501 1.9319786678534001 0.001137398105067946\n",
            "4601 1.8846281475271098 0.0011280963615221983\n",
            "4701 1.9817245414596982 0.0011190191592759865\n",
            "4801 1.7659185670199804 0.0011101576073853326\n",
            "4901 2.188665813198895 0.0011015033000912066\n",
            "5001 2.1391192222399695 0.0010930482833001135\n",
            "5101 1.8125874139368534 0.0010847850238522342\n",
            "5201 1.6616800595074892 0.0010767063813072288\n",
            "5301 1.6544548005331308 0.0010688055820075176\n",
            "5401 1.9542939933016896 0.0010610761952049212\n",
            "5501 2.218412609123334 0.0010535121110594244\n",
            "5601 1.838119359650591 0.001046107520339004\n",
            "5701 1.892627771012485 0.0010388568956672375\n",
            "5801 2.2462481096954434 0.0010317549741811346\n",
            "5901 1.4471426841337234 0.0010247967414755423\n",
            "6001 1.9312338004237972 0.0010179774167228303\n",
            "6101 1.7303275546291843 0.001011292438867507\n",
            "6201 1.8833909621462226 0.0010047374538051973\n",
            "6301 1.8943474531406537 0.0009983083024640838\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/n/home00/srush/.conda/envs/py3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: generator 'Iterator.__iter__' raised StopIteration\n",
            "  \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "1 1.5940000533591956 0.0009927515780513657\n",
            "101 1.7524283765815198 0.0009865483707369156\n",
            "201 1.900138527940726 0.0009804600111146078\n",
            "301 1.8419977760640904 0.0009744829985071481\n",
            "401 1.9621913449373096 0.0009686139798247046\n",
            "501 2.226916428655386 0.0009628497416600543\n",
            "601 1.7190162097394932 0.0009571872028951208\n",
            "701 1.8589106332874508 0.0009516234077802563\n",
            "801 1.8107321247807704 0.000946155519450957\n",
            "901 1.6531266793608665 0.0009407808138497059\n",
            "1001 1.4840005157748237 0.0009354966740233614\n",
            "1101 1.7578403616789728 0.0009303005847689719\n",
            "1201 1.3920216620899737 0.0009251901276031373\n",
            "1301 1.6626927084289491 0.0009201629760320567\n",
            "1401 1.7256765578058548 0.0009152168911012566\n",
            "1501 1.6049046433763579 0.0009103497172056578\n",
            "1601 1.6955451717367396 0.000905559378142174\n",
            "1701 1.6796367820352316 0.0009008438733884249\n",
            "1801 1.5794002648835885 0.0008962012745924116\n",
            "1901 1.9637197174597532 0.0008916297222591652\n",
            "2001 1.4656428614398465 0.0008871274226214399\n",
            "2101 1.567156056407839 0.0008826926446824871\n",
            "2201 1.542241255287081 0.0008783237174198395\n",
            "2301 1.690121710913445 0.0008740190271398465\n",
            "2401 1.357302049640566 0.0008697770149734477\n",
            "2501 1.9049871656461619 0.0008655961745043597\n",
            "2601 2.240402895025909 0.0008614750495214811\n",
            "2701 1.7940634173137369 0.0008574122318878972\n",
            "2801 1.7314323161263019 0.0008534063595194054\n",
            "2901 1.6064868164248765 0.0008494561144659686\n",
            "3001 1.7515187719254754 0.0008455602210899614\n",
            "3101 1.552100334316492 0.0008417174443354889\n",
            "3201 1.6221882179379463 0.0008379265880834463\n",
            "3301 1.5139061958470847 0.0008341864935873445\n",
            "3401 1.6668659402348567 0.0008304960379852562\n",
            "3501 2.1993618682026863 0.0008268541328835436\n",
            "3601 1.823760490231507 0.0008232597230083089\n",
            "3701 1.8189842144493014 0.0008197117849207771\n",
            "3801 1.689056838164106 0.0008162093257930558\n",
            "3901 1.5656833801185712 0.0008127513822409492\n",
            "4001 1.5621904337021988 0.0008093370192107105\n",
            "4101 1.4836799805052578 0.0008059653289168093\n",
            "4201 1.47899504378438 0.000802635429827976\n",
            "4301 1.6922758186701685 0.0007993464656989501\n",
            "4401 1.636858390578709 0.000796097604645519\n",
            "4501 1.5558803144613194 0.0007928880382605766\n",
            "4601 1.5102424336364493 0.00078971698076907\n",
            "4701 1.541241532890126 0.0007865836682198282\n",
            "4801 1.5931309935403988 0.000783487357712386\n",
            "4901 1.2315586884506047 0.0007804273266570247\n",
            "5001 1.527937745093368 0.0007774028720663579\n",
            "5101 1.31743333209306 0.0007744133098768835\n",
            "5201 1.5960889644484269 0.0007714579742990187\n",
            "5301 1.4181096099782735 0.0007685362171942096\n",
            "5401 1.4596448407392018 0.0007656474074777987\n",
            "5501 1.4594163084111642 0.0007627909305463981\n",
            "5601 1.62109798315214 0.0007599661877285873\n",
            "5701 1.586864550015889 0.0007571725957578231\n",
            "5801 1.5062829439411871 0.0007544095862665088\n",
            "5901 1.4292167258172412 0.0007516766053002225\n",
            "6001 1.4355267270930199 0.0007489731128511653\n",
            "6101 1.4162533966591582 0.0007462985824099354\n",
            "6201 1.6518787188415445 0.0007436525005347853\n",
            "6301 1.5916137372114463 0.0007410343664375577\n",
            "1 1.202994157327339 0.0007387531385993765\n",
            "101 1.4649722938484047 0.0007361862332058686\n",
            "201 1.1459896704182029 0.0007336459004644837\n",
            "301 1.417104929103516 0.0007311316850490442\n",
            "401 1.373963651509257 0.0007286431424819469\n",
            "501 1.6432027550181374 0.0007261798388040814\n",
            "601 1.4122836171882227 0.0007237413502569408\n",
            "701 1.6119428309611976 0.0007213272629763972\n",
            "801 1.5545603609643877 0.0007189371726976359\n",
            "901 1.5427279596333392 0.0007165706844707772\n",
            "1001 1.5437391183004365 0.0007142274123867243\n",
            "1101 1.9743895339342998 0.0007119069793128112\n",
            "1201 1.730805973522365 0.0007096090166378355\n",
            "1301 1.5635135210759472 0.0007073331640260875\n",
            "1401 1.206731209764257 0.000705079069180001\n",
            "1501 1.4495476994197816 0.0007028463876110714\n",
            "1601 1.2935033895773813 0.0007006347824187037\n",
            "1701 1.1734203454107046 0.000698443924076667\n",
            "1801 1.202259551268071 0.0006962734902268488\n",
            "1901 1.7874216835407424 0.0006941231654800159\n",
            "2001 1.5438914835685864 0.0006919926412233024\n",
            "2101 1.5168145569041371 0.000689881615434157\n",
            "2201 1.5306344364071265 0.0006877897925004977\n",
            "2301 1.5227781175635755 0.0006857168830468271\n",
            "2401 1.3308223116910085 0.0006836626037660786\n",
            "2501 1.4871021673316136 0.0006816266772569715\n",
            "2601 1.3415705130901188 0.0006796088318666611\n",
            "2701 1.2746119699440897 0.0006776088015384847\n",
            "2801 1.3439618053671438 0.0006756263256646049\n",
            "2901 1.3065503026737133 0.0006736611489433701\n",
            "3001 1.4918707825127058 0.0006717130212412112\n",
            "3101 1.4003087060991675 0.0006697816974589058\n",
            "3201 1.3473156996478792 0.0006678669374020495\n",
            "3301 1.3869949235959211 0.0006659685056555759\n",
            "3401 1.5086751837225165 0.000664086171462178\n",
            "3501 1.4735991460911464 0.00066221970860449\n",
            "3601 1.3997832712557283 0.0006603688952908887\n",
            "3701 1.5196008981074556 0.0006585335140447885\n",
            "3801 1.2834229312138632 0.0006567133515973014\n",
            "3901 1.3874705795169575 0.0006549081987831418\n",
            "4001 1.6422591609880328 0.0006531178504396635\n",
            "4101 1.305389653716702 0.0006513421053089143\n",
            "4201 1.5159487561322749 0.0006495807659426053\n",
            "4301 1.3981374967552256 0.0006478336386098913\n",
            "4401 1.7390631912276149 0.0006461005332078655\n",
            "4501 1.3604947600979358 0.0006443812631746732\n",
            "4601 1.7799529591429746 0.000642675645405156\n",
            "4701 1.3463407127128448 0.0006409835001689394\n",
            "4801 1.4632963918847963 0.0006393046510308797\n",
            "4901 1.1903231081087142 0.0006376389247737917\n",
            "5001 1.3287691511941375 0.0006359861513233783\n",
            "5101 1.3445309301023372 0.0006343461636752915\n",
            "5201 1.5431754024625661 0.0006327187978242499\n",
            "5301 1.3343850841192761 0.0006311038926951474\n",
            "5401 1.1768817943520844 0.0006295012900760858\n",
            "5501 1.6530805606771537 0.0006279108345532683\n",
            "5601 1.2646167293241888 0.000626332373447694\n",
            "5701 1.3651119051501155 0.000624765756753594\n",
            "5801 1.831987822048177 0.0006232108370785525\n",
            "5901 1.3451470380132378 0.0006216674695852594\n",
            "6001 1.5295006221767835 0.0006201355119348414\n",
            "6101 1.2796215488779126 0.0006186148242317232\n",
            "6201 1.3307579715619795 0.0006171052689699666\n",
            "6301 1.5296110774725094 0.0006156067109810445\n",
            "1 1.355640010209754 0.0006142969713181733\n",
            "101 1.3438594869803637 0.0006128187302418007\n",
            "201 1.3398014856502414 0.0006113511097561582\n",
            "301 1.2453488917089999 0.0006098939832926246\n",
            "401 1.74672898178801 0.0006084472263842588\n",
            "501 1.348103358541266 0.0006070107166211413\n",
            "601 1.2492765338683967 0.0006055843336068713\n",
            "701 1.568915182055207 0.0006041679589161831\n",
            "801 1.3617599749704823 0.0006027614760536461\n",
            "901 1.3296397840604186 0.0006013647704134199\n",
            "1001 1.506301498040557 0.0005999777292400283\n",
            "1101 1.1846984136500396 0.000598600241590126\n",
            "1201 1.1235853107646108 0.0005972321982952243\n",
            "1301 1.3506322290195385 0.0005958734919253515\n",
            "1401 1.5431637589354068 0.0005945240167536175\n",
            "1501 1.4227895765798166 0.0005931836687216574\n",
            "1601 1.2444980588334147 0.0005918523454059284\n",
            "1701 1.37204463215312 0.000590529945984835\n",
            "1801 1.3662666375166737 0.0005892163712066582\n",
            "1901 1.758998476434499 0.0005879115233582672\n",
            "2001 1.3996043455335894 0.0005866153062345879\n",
            "2101 1.409632071852684 0.0005853276251088103\n",
            "2201 1.3139934270293452 0.0005840483867033116\n",
            "2301 1.2863777373568155 0.0005827774991612753\n",
            "2401 1.1966209802776575 0.0005815148720189864\n",
            "2501 1.3174833165830933 0.0005802604161787846\n",
            "2601 1.406668136944063 0.0005790140438826557\n",
            "2701 1.31760111481708 0.0005777756686864456\n",
            "2801 1.22686495014932 0.0005765452054346768\n",
            "2901 1.4871160766715548 0.0005753225702359537\n",
            "3001 1.3321835576352896 0.0005741076804389384\n",
            "3101 1.349290698301047 0.0005729004546088824\n",
            "3201 1.0498975263908505 0.0005717008125046992\n",
            "3301 1.4295434548403136 0.0005705086750565621\n",
            "3401 1.3862976277887356 0.0005693239643440145\n",
            "3501 1.3612052928074263 0.0005681466035745775\n",
            "3601 1.3539716337691061 0.0005669765170628427\n",
            "3701 1.3053378225304186 0.0005658136302100359\n",
            "3801 1.2067344364186283 0.0005646578694840415\n",
            "3901 1.417662046442274 0.0005635091623998715\n",
            "4001 1.2578378450434684 0.0005623674375005725\n",
            "4101 1.2363171717152 0.0005612326243385544\n",
            "4201 1.3426340871083084 0.0005601046534573332\n",
            "4301 1.3097076122212457 0.000558983456373675\n",
            "4401 1.0131576862186193 0.0005578689655601316\n",
            "4501 1.4332989812392043 0.000556761114427959\n",
            "4601 1.4043821960221976 0.0005556598373104054\n",
            "4701 1.373746110650245 0.0005545650694463629\n",
            "4801 1.2657524709356949 0.0005534767469643717\n",
            "4901 1.1224889098666608 0.0005523948068669684\n",
            "5001 1.2615516305086203 0.000551319187015369\n",
            "5101 1.409785834257491 0.0005502498261144795\n",
            "5201 1.3791224808810512 0.0005491866636982242\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "5301 1.2408291140163783 0.0005481296401151859\n",
            "5401 1.3008261130889878 0.0005470786965145471\n",
            "5501 1.1700160388209042 0.0005460337748323287\n",
            "5601 1.2999350049067289 0.000544994817777915\n",
            "5701 1.3322585223941132 0.0005439617688208604\n",
            "5801 1.254337038320955 0.0005429345721779703\n",
            "5901 1.773689029644629 0.000541913172800649\n",
            "6001 1.3898115772462916 0.0005408975163625087\n",
            "6101 1.4735579792177305 0.0005398875492472326\n",
            "6201 1.05738219874911 0.000538883218536687\n",
            "6301 1.0802461032290012 0.0005378844719992749\n",
            "1 1.286231731530279 0.0005370101533168812\n",
            "101 1.2250633136718534 0.0005360217659787991\n",
            "201 1.239320948603563 0.0005350388161199592\n",
            "301 1.4140636462761904 0.0005340612540665886\n",
            "401 1.442663955502212 0.0005330890307779102\n",
            "501 1.4505203103472013 0.0005321220978358095\n",
            "601 1.2115196966333315 0.0005311604074347066\n",
            "701 1.2035035027656704 0.0005302039123716286\n",
            "801 1.3747974793659523 0.0005292525660364788\n",
            "901 1.36490419106849 0.0005283063224024965\n",
            "1001 1.1864821948111057 0.0005273651360169036\n",
            "1101 1.1623371304303873 0.000526428961991735\n",
            "1201 1.1043747729854658 0.0005254977559948457\n",
            "1301 1.6982813560443901 0.0005245714742410941\n",
            "1401 1.2719842366641387 0.0005236500734836944\n",
            "1501 1.2951120301149786 0.0005227335110057353\n",
            "1601 1.580276207998395 0.0005218217446118628\n",
            "1701 1.218743062199792 0.0005209147326201215\n",
            "1801 1.1479590674862266 0.0005200124338539494\n",
            "1901 1.2872504810075043 0.0005191148076343284\n",
            "2001 1.8993003838438653 0.0005182218137720798\n",
            "2101 1.2762204335303977 0.0005173334125603075\n",
            "2201 1.6183682525045242 0.0005164495647669814\n",
            "2301 1.2522982619411778 0.0005155702316276618\n",
            "2401 1.2925108795752749 0.0005146953748383575\n",
            "2501 1.340747339767404 0.0005138249565485178\n",
            "2601 1.340512964350637 0.0005129589393541545\n",
            "2701 1.1672844442073256 0.0005120972862910908\n",
            "2801 1.257948145037517 0.0005112399608283344\n",
            "2901 1.510728154462413 0.0005103869268615725\n",
            "3001 1.4130934766726568 0.0005095381487067851\n",
            "3101 1.2367545471934136 0.0005086935910939762\n",
            "3201 1.3846962348325178 0.0005078532191610173\n",
            "3301 1.2582954101526411 0.0005070169984476032\n",
            "3401 1.1545094328466803 0.0005061848948893172\n",
            "3501 1.295005505089648 0.0005053568748118022\n",
            "3601 1.3319955187034793 0.0005045329049250373\n",
            "3701 1.3548947679810226 0.000503712952317716\n",
            "3801 1.4635376840888057 0.0005028969844517252\n",
            "3901 1.6542128916307774 0.0005020849691567213\n",
            "4001 1.3512894048908493 0.0005012768746248036\n",
            "4101 1.397591198408918 0.0005004726694052806\n",
            "4201 1.3055676214280538 0.0004996723223995292\n",
            "4301 1.3375271083787084 0.000498875802855943\n",
            "4401 1.2366086341207847 0.0004980830803649704\n",
            "4501 1.2439679206581786 0.0004972941248542376\n",
            "4601 1.352382222772576 0.0004965089065837576\n",
            "4701 1.7570512742054234 0.0004957273961412208\n",
            "4801 1.232903058291413 0.0004949495644373684\n",
            "4901 1.015858386293985 0.0004941753827014446\n",
            "5001 1.381107110035373 0.000493404822476726\n",
            "5101 0.9564947709441185 0.0004926378556161293\n",
            "5201 1.228621664486127 0.0004918744542778926\n",
            "5301 1.182083563413471 0.0004911145909213302\n",
            "5401 1.2583643229590962 0.0004903582383026592\n",
            "5501 1.404046923678834 0.0004896053694708976\n",
            "5601 1.2389367091745953 0.0004888559577638302\n",
            "5701 1.119320425321348 0.00048810997680404295\n",
            "5801 1.586507015679672 0.0004873674004950231\n",
            "5901 1.112720330056618 0.0004866282030173253\n",
            "6001 1.3577893248293549 0.0004858923588248005\n",
            "6101 1.217524498468265 0.0004851598426408882\n",
            "6201 1.3229771983387764 0.0004844306294549693\n",
            "6301 1.5693217546272535 0.0004837046945187796\n",
            "1 1.1786362157727126 0.00048306134975017534\n",
            "101 1.28241519164294 0.00048234154403106603\n",
            "201 1.1411214591062162 0.00048162494648183897\n",
            "301 1.2352831599419005 0.0004809115333417623\n",
            "401 1.1032181181944907 0.00048020128109574806\n",
            "501 1.18390864826506 0.00047949416647109663\n",
            "601 1.2226583541632863 0.00047879016643429347\n",
            "701 1.0373018080717884 0.0004780892581878584\n",
            "801 1.2819566036341712 0.00047739141916724456\n",
            "901 1.1648676298791543 0.0004766966270377871\n",
            "1001 1.1654199322802015 0.00047600485969170105\n",
            "1101 1.2386636545270449 0.00047531609524512704\n",
            "1201 1.2253044219105504 0.0004746303120352227\n",
            "1301 1.375744077755371 0.0004739474886173019\n",
            "1401 1.1551300736318808 0.0004732676037620178\n",
            "1501 1.5255512128351256 0.00047259063645259034\n",
            "1601 1.255034319277911 0.00047191656588207824\n",
            "1701 1.1623500876303297 0.0004712453714506923\n",
            "1801 1.2958592986833537 0.00047057703276315175\n",
            "1901 1.1341320046922192 0.0004699115296260807\n",
            "2001 1.1937441515619867 0.0004692488420454462\n",
            "2101 1.7062073841661913 0.00046858895022403485\n",
            "2201 1.2566360468044877 0.00046793183455896863\n",
            "2301 1.2216275975806639 0.0004672774756392595\n",
            "2401 1.2636524712725077 0.0004666258542434008\n",
            "2501 1.2113699619076215 0.00046597695133699556\n",
            "2601 1.1559934263350442 0.00046533074807042176\n",
            "2701 1.256740387296304 0.0004646872257765319\n",
            "2801 1.3039579528664262 0.0004640463659683885\n",
            "2901 1.2651300196012016 0.0004634081503370334\n",
            "3001 1.2652980692801066 0.000462772560749291\n",
            "3101 1.1218284339411184 0.00046213957924560355\n",
            "3201 1.2543016897689085 0.0004615091880379007\n",
            "3301 1.2131407480192138 0.0004608813695074994\n",
            "3401 1.2994702684518415 0.0004602561062030357\n",
            "3501 1.2115506358095445 0.00045963338083842724\n",
            "3601 1.1760960748360958 0.00045901317629086643\n",
            "3701 1.0682971130590886 0.00045839547559884254\n",
            "3801 1.0764332090620883 0.00045778026196019347\n",
            "3901 1.1835216325707734 0.0004571675187301866\n",
            "4001 1.3529939632862806 0.00045655722941962654\n",
            "4101 1.3684578015236184 0.0004559493776929923\n",
            "4201 1.2233722301607486 0.0004553439473666001\n",
            "4301 1.2596116681525018 0.0004547409224067939\n",
            "4401 1.2757911044172943 0.00045414028692816196\n",
            "4501 1.2199301174841821 0.0004535420251917793\n",
            "4601 1.3471774608151463 0.0004529461216034753\n",
            "4701 1.475795219448628 0.0004523525607121267\n",
            "4801 1.1835241899825633 0.0004517613272079745\n",
            "4901 1.1791616377497576 0.0004511724059209659\n",
            "5001 1.3126113665202865 0.0004505857818191191\n",
            "5101 1.2516068609402282 0.0004500014400069121\n",
            "5201 1.178165558274486 0.0004494193657236937\n",
            "5301 1.6013869942435122 0.0004488395443421177\n",
            "5401 1.2677101592962572 0.00044826196136659916\n",
            "5501 1.1976667390699731 0.0004476866024317922\n",
            "5601 1.1990807302790927 0.00044711345330108884\n",
            "5701 1.1415361673789448 0.0004465424998651406\n",
            "5801 1.2389779405202717 0.0004459737281403985\n",
            "5901 1.1746156329172663 0.0004454071242676752\n",
            "6001 1.1718775559565984 0.0004448426745107265\n",
            "6101 1.1669323876558337 0.00044428036525485275\n",
            "6201 1.22836275130976 0.0004437201830055194\n",
            "6301 1.1068585112225264 0.000443162114386997\n",
            "1 1.1908240653865505 0.00044267275186678196\n",
            "101 1.156728027795907 0.0004421186210736662\n",
            "201 1.151486962888157 0.0004415665660409348\n",
            "301 1.1075408830074593 0.0004410165738412884\n",
            "401 1.1251853418070823 0.00044046863165985925\n",
            "501 1.224421168473782 0.0004399227267929559\n",
            "601 1.1097798637929372 0.00043937884664682695\n",
            "701 0.992531725903973 0.0004388369787364407\n",
            "801 1.2762772621936165 0.0004382971106842813\n",
            "901 1.154728337773122 0.00043775923021916087\n",
            "1001 0.9699444866273552 0.00043722332517504866\n",
            "1101 1.1039727496681735 0.0004366893834899152\n",
            "1201 1.2997219555545598 0.0004361573932045913\n",
            "1301 1.5713044246076606 0.00043562734246164385\n",
            "1401 1.1782071397465188 0.00043509921950426545\n",
            "1501 1.256332863289117 0.0004345730126751789\n",
            "1601 1.162631830346072 0.00043404871041555687\n",
            "1701 1.1123517343075946 0.00043352630126395546\n",
            "1801 1.0946980192093179 0.0004330057738552615\n",
            "1901 1.120711475959979 0.0004324871169196544\n",
            "2001 1.1385652619646862 0.0004319703192815812\n",
            "2101 1.0391206528292969 0.0004314553698587452\n",
            "2201 1.1468603002722375 0.00043094225766110786\n",
            "2301 1.1944863148819422 0.0004304309717899036\n",
            "2401 1.1445480604888871 0.00042992150143666746\n",
            "2501 1.160092411795631 0.0004294138358822756\n",
            "2601 0.905779943568632 0.00042890796449599795\n",
            "2701 1.2337692737637553 0.0004284038767345632\n",
            "2801 1.2654334787439439 0.00042790156214123586\n",
            "2901 1.2613030684588011 0.0004274010103449054\n",
            "3001 1.1566388571663992 0.0004269022110591865\n",
            "3101 1.1506170178181492 0.0004264051540815317\n",
            "3201 1.1042177192866802 0.00042590982929235444\n",
            "3301 1.268968387885252 0.00042541622665416415\n",
            "3401 1.1708880871301517 0.0004249243362107117\n",
            "3501 1.1094016103306785 0.00042443414808614573\n",
            "3601 1.3188527839665767 0.0004239456524841804\n",
            "3701 1.2144307589042 0.0004234588396872726\n",
            "3801 1.1827894128946355 0.0004229737000558104\n",
            "3901 0.9924444004427642 0.00042249022402731095\n",
            "4001 1.1228576390712988 0.0004220084021156294\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "4101 1.1924936635477934 0.0004215282249101765\n",
            "4201 1.1275967326655518 0.0004210496830751471\n",
            "4301 1.0625419117277488 0.0004205727673487576\n",
            "4401 1.1389823842037003 0.00042009746854249313\n",
            "4501 1.339291847194545 0.00041962377754036395\n",
            "4601 1.0302886090357788 0.0004191516852981713\n",
            "4701 1.7122778899138211 0.00041868118284278167\n",
            "4801 1.2910672437865287 0.0004182122612714111\n",
            "4901 1.0494152382598259 0.00041774491175091685\n",
            "5001 1.1782474033534527 0.0004172791255170995\n",
            "5101 1.040663594380021 0.0004168148938740118\n",
            "5201 0.9901199785526842 0.00041635220819327733\n",
            "5301 1.5490817801910453 0.00041589105991341656\n",
            "5401 1.1346296942792833 0.00041543144053918197\n",
            "5501 1.223581779631786 0.00041497334164089994\n",
            "5601 0.958975835936144 0.0004145167548538224\n",
            "5701 1.238148811913561 0.0004140616718774844\n",
            "5801 1.1881117207813077 0.000413608084475071\n",
            "5901 1.1295715225860476 0.0004131559844727907\n",
            "6001 1.0610488005331717 0.000412705363759257\n",
            "6101 1.2371235750615597 0.00041225621428487707\n",
            "6201 1.1479767516138963 0.00041180852806124783\n",
            "6301 1.2059640220250003 0.0004113622971605593\n",
            "1 1.1765662879188312 0.0004109663692823915\n",
            "101 1.219779463717714 0.0004105228675028437\n",
            "201 0.972488499362953 0.00041008079847008953\n",
            "301 1.1617506150214467 0.0004096401544864815\n",
            "401 1.0883347367926035 0.0004092009279121472\n",
            "501 1.1085488446406089 0.00040876311116443343\n",
            "601 1.1023443718672752 0.0004083266967173559\n",
            "701 1.018611608800711 0.00040789167710105623\n",
            "801 1.1658196483003849 0.00040745804490126497\n",
            "901 1.2917855954219704 0.0004070257927587706\n",
            "1001 1.186474629881559 0.00040659491336889525\n",
            "1101 1.127356821874855 0.00040616539948097586\n",
            "1201 1.1975307842949405 0.00040573724389785204\n",
            "1301 1.1174790017685154 0.0004053104394753595\n",
            "1401 1.0863252188792103 0.0004048849791218294\n",
            "1501 1.0622602235816885 0.000404460855797593\n",
            "1601 1.1195327076129615 0.00040403806251449327\n",
            "1701 1.2447982146404684 0.00040361659233540054\n",
            "1801 1.2607627244724426 0.0004031964383737348\n",
            "1901 1.278331945562968 0.00040277759379299307\n",
            "2001 1.025594950420782 0.0004023600518062819\n",
            "2101 1.115274733179831 0.0004019438056758561\n",
            "2201 1.1167924739420414 0.0004015288487126612\n",
            "2301 1.0995151306560729 0.000401115174275883\n",
            "2401 1.1547567544039339 0.00040070277577250023\n",
            "2501 1.1590442548913416 0.00040029164665684384\n",
            "2601 1.1047132272506133 0.00039988178043016053\n",
            "2701 1.0620037270709872 0.00039947317064018093\n",
            "2801 1.0939110746548977 0.00039906581088069363\n",
            "2901 0.9693534299731255 0.0003986596947911227\n",
            "3001 1.2754340882529505 0.0003982548160561108\n",
            "3101 2.0011723663365046 0.0003978511684051071\n",
            "3201 1.1672507325711194 0.0003974487456119586\n",
            "3301 0.8547125565819442 0.00039704754149450736\n",
            "3401 1.0948779656609986 0.00039664754991419163\n",
            "3501 1.1662541554399013 0.0003962487647756509\n",
            "3601 1.242357063729287 0.00039585118002633614\n",
            "3701 1.142975198366912 0.000395454789656124\n",
            "3801 1.2055247909738682 0.0003950595876969351\n",
            "3901 1.2129587295930833 0.0003946655682223565\n",
            "4001 1.239052205113694 0.0003942727253472687\n",
            "4101 1.1285374546132516 0.0003938810532274764\n",
            "4201 1.3123125492420513 0.00039349054605934306\n",
            "4301 1.2088057449145708 0.00039310119807943006\n",
            "4401 1.0897887839237228 0.00039271300356413926\n",
            "4501 1.2131327866227366 0.00039232595682935973\n",
            "4601 0.9877158605959266 0.000391940052230118\n",
            "4701 1.2145014504967548 0.0003915552841602323\n",
            "4801 1.1513765653944574 0.0003911716470519708\n",
            "4901 1.1751896993955597 0.0003907891353757127\n",
            "5001 1.1771067604749987 0.0003904077436396139\n",
            "5101 1.3224336538114585 0.0003900274663892758\n",
            "5201 1.355893952131737 0.0003896482982074174\n",
            "5301 1.1996791153214872 0.0003892702337135512\n",
            "5401 1.206806231304654 0.0003888932675636631\n",
            "5501 1.280991047504358 0.0003885173944498942\n",
            "5601 0.9168080711970106 0.0003881426091002278\n",
            "5701 1.1924245768459514 0.0003877689062781782\n",
            "5801 1.1940782586170826 0.0003873962807824836\n",
            "5901 1.2784329915011767 0.0003870247274468023\n",
            "6001 1.1448089724872261 0.00038665424113941134\n",
            "6101 1.1181278676813236 0.0003862848167629092\n",
            "6201 1.2686003018752672 0.00038591644925392126\n",
            "6301 1.0795898175565526 0.000385549133582808\n",
            "1 1.199639980099164 0.00038523407955521927\n",
            "101 1.0967486363369972 0.00038486870703897236\n",
            "201 1.2039306252845563 0.00038450437215947677\n",
            "301 1.106695241353009 0.0003841410700146326\n",
            "401 1.0133273621067929 0.00038377879573470126\n",
            "501 1.1209047999582253 0.0003834175444820315\n",
            "601 1.073817516444251 0.00038305731145078797\n",
            "701 1.1002086726948619 0.00038269809186668256\n",
            "801 1.044247637852095 0.00038233988098670897\n",
            "901 1.1538543488713913 0.00038198267409887953\n",
            "1001 1.1638364950194955 0.00038162646652196454\n",
            "1101 1.1222136826545466 0.00038127125360523515\n",
            "1201 1.0936923912668135 0.0003809170307282081\n",
            "1301 1.0790816302178428 0.0003805637933003932\n",
            "1401 1.0973517978854943 0.0003802115367610436\n",
            "1501 1.3543376340385294 0.00037986025657890806\n",
            "1601 1.0638599513913505 0.0003795099482519871\n",
            "1701 1.095864930888638 0.0003791606073072896\n",
            "1801 1.3785420355270617 0.00037881222930059356\n",
            "1901 1.0656100183841772 0.0003784648098162084\n",
            "2001 1.083574770949781 0.0003781183444667399\n",
            "2101 1.7192173111798184 0.0003777728288928577\n",
            "2201 1.090653446619399 0.0003774282587630644\n",
            "2301 1.1122783308528597 0.00037708462977346826\n",
            "2401 0.95696423901245 0.0003767419376475568\n",
            "2501 1.2160079335735645 0.0003764001781359734\n",
            "2601 1.2086039673304185 0.00037605934701629616\n",
            "2701 1.0832101813866757 0.00037571944009281874\n",
            "2801 1.013074157119263 0.00037538045319633314\n",
            "2901 1.0823434699559584 0.00037504238218391556\n",
            "3001 1.1612105248786975 0.0003747052229387128\n",
            "3101 0.9126896761590615 0.0003743689713697328\n",
            "3201 1.3341417479550728 0.00037403362341163505\n",
            "3301 0.9600269035436213 0.0003736991750245252\n",
            "3401 1.1916928359714802 0.0003733656221937497\n",
            "3501 1.4976106537505984 0.0003730329609296942\n",
            "3601 1.1840611910447478 0.0003727011872675824\n",
            "3701 1.3150727476167958 0.0003723702972672783\n",
            "3801 1.221188339870423 0.00037204028701308904\n",
            "3901 1.2026138188084587 0.0003717111526135708\n",
            "4001 1.3793403076779214 0.00037138289020133557\n",
            "4101 1.0772470782976598 0.0003710554959328607\n",
            "4201 1.0168679640773917 0.0003707289659882998\n",
            "4301 1.1685322925950459 0.00037040329657129513\n",
            "4401 1.1224966624286026 0.00037007848390879306\n",
            "4501 1.0253048577578738 0.00036975452425085955\n",
            "4601 1.2101853350850433 0.00036943141387049916\n",
            "4701 1.050108958443161 0.0003691091490634741\n",
            "4801 1.2717001468117815 0.00036878772614812674\n",
            "4901 1.1231291381409392 0.00036846714146520227\n",
            "5001 1.1141781120968517 0.00036814739137767423\n",
            "5101 0.9994667179416865 0.00036782847227057074\n",
            "5201 1.3557415267750912 0.0003675103805508032\n",
            "5301 1.504937146051816 0.0003671931126469962\n",
            "5401 1.0444834220979828 0.00036687666500931896\n",
            "5501 1.0159150707913795 0.0003665610341093186\n",
            "5601 1.4691102042561397 0.00036624621643975515\n",
            "5701 1.2679062836105004 0.0003659322085144373\n",
            "5801 1.1070539963402553 0.0003656190068680607\n",
            "5901 1.2043958652066067 0.0003653066080560474\n",
            "6001 1.1217296464601532 0.00036499500865438625\n",
            "6101 1.2132740695233224 0.00036468420525947586\n",
            "6201 1.452793362134571 0.00036437419448796804\n",
            "6301 1.0731251265387982 0.00036406497297661317\n",
            "1 1.1998733360724145 0.00036379350826718935\n",
            "101 1.1498671763110906 0.0003634857615296514\n",
            "201 1.1022596344992053 0.00036317879447701637\n",
            "301 1.0011312331771478 0.00036287260382257964\n",
            "401 1.0373523531015962 0.0003625671862990008\n",
            "501 1.0644397677097004 0.000362262538658157\n",
            "601 1.1183270415349398 0.000361958657670998\n",
            "701 0.9439565273642074 0.00036165554012740277\n",
            "801 1.0483181119780056 0.0003613531828360362\n",
            "901 1.10791165966657 0.00036105158262420917\n",
            "1001 0.9895736404578201 0.00036075073633773743\n",
            "1101 1.0942751504917396 0.00036045064084080426\n",
            "1201 1.0274720180314034 0.0003601512930158222\n",
            "1301 1.049829078532639 0.0003598526897632977\n",
            "1401 1.0599873741302872 0.00035955482800169595\n",
            "1501 1.1110646773013286 0.00035925770466730756\n",
            "1601 1.1195740707335062 0.0003589613167141159\n",
            "1701 1.1175601889844984 0.0003586656611136664\n",
            "1801 1.27650167158572 0.00035837073485493607\n",
            "1901 1.1153064398095012 0.000358076534944205\n",
            "2001 1.4756392514391337 0.000357783058404929\n",
            "2101 0.8967400579713285 0.0003574903022776121\n",
            "2201 1.0554919667192735 0.0003571982636196827\n",
            "2301 1.1484477042686194 0.000356906939505368\n",
            "2401 1.0967925000586547 0.00035661632702557175\n",
            "2501 1.275310180048109 0.0003563264232877516\n",
            "2601 1.084061863599345 0.00035603722541579873\n",
            "2701 1.076280384673737 0.00035574873054991784\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "2801 1.200010517553892 0.0003554609358465082\n",
            "2901 1.059172057226533 0.000355173838478046\n",
            "3001 1.0261963941156864 0.000354887435632968\n",
            "3101 0.9737269952893257 0.0003546017245155551\n",
            "3201 1.109491402952699 0.0003543167023458187\n",
            "3301 1.0379895093501545 0.0003540323663593864\n",
            "3401 1.0210047286818735 0.00035374871380738974\n",
            "3501 1.3062616163679195 0.0003534657419563522\n",
            "3601 1.1297821700572968 0.00035318344808807914\n",
            "3701 1.095454223890556 0.0003529018294995477\n",
            "3801 1.0263875699602067 0.00035262088350279793\n",
            "3901 1.0200049103004858 0.00035234060742482575\n",
            "4001 1.139240143907955 0.00035206099860747537\n",
            "4101 1.185085133272878 0.00035178205440733397\n",
            "4201 1.0887831579057092 0.0003515037721956263\n",
            "4301 1.7533796445081862 0.000351226149358111\n",
            "4401 1.1149956742010545 0.00035094918329497705\n",
            "4501 1.1544227619015146 0.0003506728714207421\n",
            "4601 1.1121961249154992 0.00035039721116415036\n",
            "4701 1.0929120010696352 0.0003501221999680728\n",
            "4801 1.118996370700188 0.000349847835289407\n",
            "4901 0.9860638748505153 0.00034957411459897886\n",
            "5001 1.004449057742022 0.0003493010353814441\n",
            "5101 1.2927988782757893 0.00034902859513519165\n",
            "5201 0.98420900356723 0.0003487567913722472\n",
            "5301 1.1210692654858576 0.000348485621618178\n",
            "5401 1.163566045666812 0.00034821508341199766\n",
            "5501 1.17701395630138 0.0003479451743060731\n",
            "5601 1.0291424575298151 0.00034767589186603104\n",
            "5701 1.2543358486509533 0.0003474072336706659\n",
            "5801 1.2299754508348997 0.00034713919731184855\n",
            "5901 1.1936145080917413 0.0003468717803944353\n",
            "6001 0.9138605990447104 0.00034660498053617827\n",
            "6101 1.1037570714397589 0.00034633879536763624\n",
            "6201 1.04157008238235 0.00034607322253208626\n",
            "6301 1.1919174637878314 0.0003458082596854357\n",
            "1 0.9797578346915543 0.00034557559511139293\n",
            "101 0.9891712895187084 0.00034531177274179953\n",
            "201 1.1815550197497942 0.00034504855367990236\n",
            "301 1.1358435613219626 0.0003447859356297997\n",
            "401 1.0717519058380276 0.000344523916307803\n",
            "501 1.172375235328218 0.00034426249344235384\n",
            "601 1.0603420100815129 0.00034400166477394084\n",
            "701 1.0992282917286502 0.000343741428055018\n",
            "801 1.04559408465866 0.0003434817810499231\n",
            "901 1.0248865495998416 0.0003432227215347973\n",
            "1001 1.0598302248690743 0.0003429642472975047\n",
            "1101 1.0938185814011376 0.0003427063561375535\n",
            "1201 1.291374852447916 0.000342449045866017\n",
            "1301 1.0285806620959193 0.0003421923143054557\n",
            "1401 1.17992360109929 0.0003419361592898398\n",
            "1501 0.9615428688703105 0.0003416805786644727\n",
            "1601 1.2486475716141285 0.0003414255702859144\n",
            "1701 0.9794061238644645 0.0003411711320219065\n",
            "1801 1.1059001302346587 0.00034091726175129706\n",
            "1901 0.9131126408465207 0.00034066395736396637\n",
            "2001 0.9930663524428383 0.0003404112167607534\n",
            "2101 1.0812218267819844 0.0003401590378533823\n",
            "2201 1.0715046060213353 0.0003399074185643907\n",
            "2301 1.1185214965953492 0.00033965635682705713\n",
            "2401 1.05721441534115 0.00033940585058533\n",
            "2501 1.0936700437378022 0.00033915589779375693\n",
            "2601 1.07034515045234 0.00033890649641741454\n",
            "2701 1.0813248989579733 0.00033865764443183875\n",
            "2801 1.0510470166627783 0.0003384093398229561\n",
            "2901 0.9623011860530823 0.0003381615805870148\n",
            "3001 1.1494725269731134 0.00033791436473051725\n",
            "3101 1.2335324875239166 0.0003376676902701525\n",
            "3201 1.0398004396120086 0.00033742155523272933\n",
            "3301 1.0589452146668918 0.0003371759576551101\n",
            "3401 1.084106142167002 0.00033693089558414497\n",
            "3501 1.0406659920408856 0.0003366863670766065\n",
            "3601 0.9325393754988909 0.00033644237019912526\n",
            "3701 1.0507557194505353 0.0003361989030281253\n",
            "3801 0.945562198292464 0.0003359559636497606\n",
            "3901 1.0489263081690297 0.0003357135501598519\n",
            "4001 1.0528855019947514 0.00033547166066382383\n",
            "4101 1.1952165458351374 0.0003352302932766432\n",
            "4201 1.0958911241032183 0.00033498944612275674\n",
            "4301 1.077680416405201 0.0003347491173360301\n",
            "4401 1.2972540742484853 0.0003345093050596873\n",
            "4501 1.039087069220841 0.0003342700074462501\n",
            "4601 0.9597453814931214 0.00033403122265747876\n",
            "4701 1.1728105103829876 0.00033379294886431207\n",
            "4801 1.1258336059836438 0.0003335551842468092\n",
            "4901 1.0011352995352354 0.0003333179269940906\n",
            "5001 0.9672558718448272 0.00033308117530428074\n",
            "5101 1.0522874586749822 0.0003328449273844502\n",
            "5201 1.0063609674107283 0.0003326091814505589\n",
            "5301 1.0480196221014921 0.00033237393572739917\n",
            "5401 1.0445173801199417 0.00033213918844854004\n",
            "5501 1.417743748796056 0.00033190493785627127\n",
            "5601 1.0902981872641249 0.000331671182201548\n",
            "5701 1.0301871165866032 0.00033143791974393625\n",
            "5801 1.4090074983541854 0.00033120514875155805\n",
            "5901 1.1904211936052889 0.0003309728675010378\n",
            "6001 1.1052953382313717 0.0003307410742774485\n",
            "6101 1.133972127106972 0.00033050976737425853\n",
            "6201 1.4820798086614104 0.00033027894509327907\n",
            "6301 1.1476092239608988 0.00033004860574461153\n",
            "1 1.0870586273958907 0.00032984630526377065\n",
            "101 1.00110832543578 0.00032961686928176145\n",
            "201 0.9876259700540686 0.0003293879114110055\n",
            "301 1.2413200094233616 0.0003291594299932822\n",
            "401 0.9424758276436478 0.00032893142337841173\n",
            "501 1.0926933737646323 0.00032870388992420444\n",
            "601 1.0706572374765528 0.0003284768279964114\n",
            "701 1.0879125816572923 0.00032825023596867546\n",
            "801 1.262531905740616 0.0003280241122224816\n",
            "901 1.050877535046311 0.0003277984551471088\n",
            "1001 1.135452825037646 0.0003275732631395822\n",
            "1101 1.0099836179433623 0.0003273485346046242\n",
            "1201 1.1641883124248125 0.0003271242679546084\n",
            "1301 1.0249766572378576 0.00032690046160951133\n",
            "1401 1.216345368164184 0.0003266771139968662\n",
            "1501 1.4009095890432945 0.00032645422355171653\n",
            "1601 0.9214687866624445 0.00032623178871657\n",
            "1701 1.050587208737852 0.0003260098079413526\n",
            "1801 1.0106142781150993 0.0003257882796833635\n",
            "1901 0.9388233295176178 0.00032556720240723\n",
            "2001 1.1458081254386343 0.0003253465745848626\n",
            "2101 1.0818304931126477 0.00032512639469541087\n",
            "2201 0.8635702040046453 0.0003249066612252194\n",
            "2301 1.0180434776411857 0.00032468737266778394\n",
            "2401 1.0467977939988486 0.0003244685275237081\n",
            "2501 1.083000476603047 0.0003242501243006605\n",
            "2601 1.1069668279960752 0.00032403216151333166\n",
            "2701 1.086160118225962 0.00032381463768339173\n",
            "2801 1.0924749624973629 0.0003235975513394485\n",
            "2901 1.009744831302669 0.00032338090101700554\n",
            "3001 0.9085385013604537 0.0003231646852584205\n",
            "3101 1.1706041378201917 0.00032294890261286426\n",
            "3201 1.032271361502353 0.00032273355163627964\n",
            "3301 1.2584509218577296 0.00032251863089134133\n",
            "3401 1.2436874122358859 0.000322304138947415\n",
            "3501 1.0451832013077365 0.0003220900743805179\n",
            "3601 1.0900762653473066 0.00032187643577327854\n",
            "3701 1.1192542002827395 0.00032166322171489793\n",
            "3801 1.00253647408681 0.0003214504308011099\n",
            "3901 1.2160937447333708 0.0003212380616341424\n",
            "4001 1.0416435159859248 0.0003210261128226793\n",
            "4101 1.3598752447869629 0.00032081458298182156\n",
            "4201 1.0555532689650136 0.0003206034707330495\n",
            "4301 1.1295962483854964 0.00032039277470418526\n",
            "4401 0.9410244208120275 0.0003201824935293548\n",
            "4501 0.8939700378105044 0.00031997262584895135\n",
            "4601 0.908640876179561 0.000319763170309598\n",
            "4701 1.128680162204546 0.0003195541255641112\n",
            "4801 1.0497526655672118 0.00031934549027146444\n",
            "4901 1.0289937005145475 0.0003191372630967521\n",
            "5001 0.9918764412868768 0.0003189294427111535\n",
            "5101 1.2255524442298338 0.0003187220277918973\n",
            "5201 1.3292681298672733 0.0003185150170222263\n",
            "5301 0.878005885053426 0.00031830840909136197\n",
            "5401 1.0740752452165907 0.00031810220269447\n",
            "5501 1.0994656120092259 0.00031789639653262544\n",
            "5601 1.159670107124839 0.0003176909893127784\n",
            "5701 0.8859041188843548 0.0003174859797477199\n",
            "5801 1.084522244927939 0.00031728136655604814\n",
            "5901 1.4824702723776682 0.0003170771484621346\n",
            "6001 1.230977819112013 0.0003168733241960908\n",
            "6101 1.0119965468620649 0.00031666989249373517\n",
            "6201 1.1002646164814678 0.00031646685209656003\n",
            "6301 1.1440792203939054 0.00031626420175169897\n",
            "1 1.0551144047021808 0.0003160882122689669\n",
            "101 1.0408390048833098 0.00031588628797931984\n",
            "201 1.0153360446565785 0.0003156847501772966\n",
            "301 1.01748421555385 0.0003154835976315582\n",
            "401 1.1267781729111448 0.0003152828291162512\n",
            "501 1.0327468327741371 0.0003150824434109757\n",
            "601 0.9960314880299848 0.0003148824393007546\n",
            "701 1.0631007702104398 0.00031468281557600267\n",
            "801 1.0372768385277595 0.00031448357103249544\n",
            "901 1.031023440795252 0.0003142847044713392\n",
            "1001 0.9323838343843818 0.0003140862146989404\n",
            "1101 0.850510573014617 0.0003138881005269756\n",
            "1201 1.0943628003296908 0.0003136903607723615\n",
            "1301 1.0844742289336864 0.00031349299425722566\n",
            "1401 1.4024500491796061 0.00031329599980887637\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "1501 1.1463393379817717 0.00031309937625977405\n",
            "1601 1.0650637014914537 0.0003129031224475018\n",
            "1701 1.1410465109511279 0.00031270723721473664\n",
            "1801 1.0148204645956866 0.0003125117194092209\n",
            "1901 0.9743651752360165 0.0003123165678837336\n",
            "2001 0.990701739974611 0.00031212178149606226\n",
            "2101 1.1128740338463103 0.00031192735910897496\n",
            "2201 1.5508626039809315 0.0003117332995901923\n",
            "2301 1.01486110695987 0.00031153960181235955\n",
            "2401 0.9611194784665713 0.0003113462646530196\n",
            "2501 1.0847897573257796 0.0003111532869945851\n",
            "2601 1.2803321699175285 0.00031096066772431187\n",
            "2701 1.0769891024538083 0.0003107684057342714\n",
            "2801 1.0039243546780199 0.00031057649992132457\n",
            "2901 1.0824949400266632 0.00031038494918709473\n",
            "3001 0.9644128995714709 0.00031019375243794144\n",
            "3101 0.9868561172188492 0.00031000290858493437\n",
            "3201 1.0903575613629073 0.00030981241654382685\n",
            "3301 1.2633273452374851 0.0003096222752350304\n",
            "3401 1.0088038056419464 0.000309432483583589\n",
            "3501 1.1047235757578164 0.0003092430405191533\n",
            "3601 1.163446888080216 0.00030905394497595545\n",
            "3701 1.018205283649877 0.00030886519589278384\n",
            "3801 1.0408570388099179 0.00030867679221295824\n",
            "3901 1.0657447287230752 0.00030848873288430483\n",
            "4001 0.930832964291767 0.0003083010168591314\n",
            "4101 1.4587874389944773 0.00030811364309420327\n",
            "4201 1.1227497690124437 0.0003079266105507184\n",
            "4301 1.1970081577601377 0.0003077399181942835\n",
            "4401 1.1083186157047749 0.00030755356499488986\n",
            "4501 1.0473160178516991 0.00030736754992688985\n",
            "4601 1.0928417469840497 0.0003071818719689727\n",
            "4701 1.0073067757184617 0.00030699653010414117\n",
            "4801 1.1523846584532293 0.00030681152331968824\n",
            "4901 1.1988015054084826 0.0003066268506071739\n",
            "5001 1.036811558995396 0.00030644251096240176\n",
            "5101 1.0675939484208357 0.0003062585033853964\n",
            "5201 1.0752534797684348 0.00030607482688038056\n",
            "5301 1.1183025861246279 0.0003058914804557523\n",
            "5401 1.0365500289481133 0.0003057084631240629\n",
            "5501 1.129350705537945 0.00030552577390199393\n",
            "5601 1.06671357084997 0.0003053434118103358\n",
            "5701 1.0859052878222428 0.0003051613758739652\n",
            "5801 1.0723684425465763 0.00030497966512182316\n",
            "5901 1.0603833084605867 0.0003047982785868937\n",
            "6001 1.0581669194652932 0.0003046172153061818\n",
            "6101 1.2675022858026068 0.0003044364743206923\n",
            "6201 1.0536423055746127 0.0003042560546754083\n",
            "6301 1.2465427681345318 0.00030407595541926996\n",
            "1 0.8441335130482912 0.00030391413923858634\n",
            "101 0.9350836968515068 0.00030373464611573535\n",
            "201 1.0421846130630001 0.0003035554706461405\n",
            "301 1.008769184758421 0.0003033766118939749\n",
            "401 0.9787611065548845 0.000303198068927267\n",
            "501 1.0469113910803571 0.00030301984081788013\n",
            "601 1.0306272330635693 0.00030284192664149214\n",
            "701 0.8391264111269265 0.00030266432547757535\n",
            "801 0.9852646276758605 0.00030248703640937665\n",
            "901 0.9640902730752714 0.00030231005852389745\n",
            "1001 1.1280414578068303 0.00030213339091187405\n",
            "1101 0.9939612101297826 0.0003019570326677579\n",
            "1201 1.0867023059399799 0.00030178098288969626\n",
            "1301 1.0411206257122103 0.00030160524067951265\n",
            "1401 1.0711584523378406 0.0003014298051426879\n",
            "1501 1.0796883448783774 0.0003012546753883405\n",
            "1601 1.027666941517964 0.00030107985052920836\n",
            "1701 1.082986782770604 0.00030090532968162913\n",
            "1801 1.1074479344606516 0.0003007311119655219\n",
            "1901 1.1305997944582487 0.0003005571965043686\n",
            "2001 1.345339710366943 0.0003003835824251953\n",
            "2101 1.001590578132891 0.00030021026885855383\n",
            "2201 1.0054450589232147 0.0003000372549385033\n",
            "2301 1.0041432639409322 0.00029986453980259265\n",
            "2401 1.0640304164699046 0.00029969212259184163\n",
            "2501 1.1201181028736755 0.0002995200024507235\n",
            "2601 0.8765224074013531 0.00029934817852714696\n",
            "2701 1.0152945614827331 0.0002991766499724386\n",
            "2801 1.2956223709957158 0.0002990054159413251\n",
            "2901 1.097986907014274 0.0002988344755919157\n",
            "3001 1.2291080165232415 0.00029866382808568526\n",
            "3101 1.1140619127836544 0.0002984934725874564\n",
            "3201 1.0722678579004423 0.0002983234082653825\n",
            "3301 1.03946516571159 0.0002981536342909311\n",
            "3401 0.9876702070032479 0.0002979841498388662\n",
            "3501 0.8540887358831242 0.00029781495408723205\n",
            "3601 0.9894529741141014 0.00029764604621733594\n",
            "3701 1.0710785342380404 0.000297477425413732\n",
            "3801 1.2710673977526312 0.00029730909086420423\n",
            "3901 1.0929949116252828 0.0002971410417597504\n",
            "4001 1.1222996068827342 0.0002969732772945655\n",
            "4101 1.164539644116303 0.00029680579666602566\n",
            "4201 1.033734397671651 0.000296638599074672\n",
            "4301 1.093015514779836 0.0002964716837241944\n",
            "4401 1.0481613585725427 0.0002963050498214161\n",
            "4501 1.0937337041832507 0.00029613869657627706\n",
            "4601 1.1815662420112858 0.000295972623201819\n",
            "4701 1.1428916620570817 0.0002958068289141693\n",
            "4801 1.1009264337189961 0.0002956413129325257\n",
            "4901 0.8777621657354757 0.00029547607447914055\n",
            "5001 1.0156730558082927 0.00029531111277930595\n",
            "5101 0.9942513670539483 0.00029514642706133804\n",
            "5201 1.1302866424011881 0.00029498201655656206\n",
            "5301 1.0087051462905947 0.0002948178804992971\n",
            "5401 1.0660143050336046 0.0002946540181268415\n",
            "5501 1.0107977577135898 0.00029449042867945755\n",
            "5601 1.0777363086963305 0.0002943271114003569\n",
            "5701 0.856828257907182 0.00029416406553568584\n",
            "5801 1.1287360956775956 0.0002940012903345107\n",
            "5901 1.366358119645156 0.00029383878504880313\n",
            "6001 1.0964845723065082 0.00029367654893342604\n",
            "6101 1.34646458978159 0.00029351458124611887\n",
            "6201 1.2197050878312439 0.0002933528812474836\n",
            "6301 1.0830936049751472 0.0002931914482009704\n",
            "1 1.2537849597129025 0.00029304477550482497\n",
            "101 0.9655292083625682 0.00029288385030021\n",
            "201 1.1372923650196753 0.0002927231899204302\n",
            "301 0.9451354363700375 0.0002925627936399378\n",
            "401 0.9661671929707154 0.00029240266073596516\n",
            "501 1.0162687979172915 0.0002922427904885108\n",
            "601 0.9551542007829994 0.0002920831821803257\n",
            "701 0.9841917234880384 0.0002919238350969\n",
            "801 1.061543255826109 0.00029176474852644945\n",
            "901 0.985908080736408 0.0002916059217599022\n",
            "1001 1.0337736615701942 0.0002914473540908853\n",
            "1101 0.9899544979416532 0.0002912890448157118\n",
            "1201 1.1052642236463726 0.00029113099323336726\n",
            "1301 0.9609193275682628 0.00029097319864549706\n",
            "1401 1.0143777604680508 0.0002908156603563932\n",
            "1501 1.0131031578639522 0.0002906583776729816\n",
            "1601 1.0399196342332289 0.00029050134990480915\n",
            "1701 1.3567781529854983 0.00029034457636403104\n",
            "1801 1.1145936762022757 0.0002901880563653981\n",
            "1901 1.0984270876506343 0.0002900317892262443\n",
            "2001 1.0226630433771788 0.00028987577426647405\n",
            "2101 1.1856945900362916 0.0002897200108085499\n",
            "2201 1.125245438015554 0.00028956449817748025\n",
            "2301 1.126162831991678 0.00028940923570080693\n",
            "2401 0.971063018507266 0.00028925422270859307\n",
            "2501 0.8773902256507427 0.00028909945853341086\n",
            "2601 0.8763325407635421 0.0002889449425103295\n",
            "2701 1.1415061227562546 0.0002887906739769035\n",
            "2801 1.052460735765635 0.0002886366522731603\n",
            "2901 1.5205009760629764 0.00028848287674158846\n",
            "3001 0.9414957111439435 0.0002883293467271265\n",
            "3101 0.9435001520323567 0.0002881760615771502\n",
            "3201 1.1403545759221743 0.0002880230206414618\n",
            "3301 1.053262686386006 0.00028787022327227786\n",
            "3401 1.0832857484929264 0.000287717668824218\n",
            "3501 1.1019753235159442 0.00028756535665429354\n",
            "3601 1.055358653771691 0.0002874132861218958\n",
            "3701 1.0278627741499804 0.00028726145658878504\n",
            "3801 1.0083879251906183 0.0002871098674190792\n",
            "3901 1.0555589499126654 0.0002869585179792425\n",
            "4001 1.0509156602493022 0.00028680740763807453\n",
            "4101 1.0385481148259714 0.0002866565357666993\n",
            "4201 1.0417402729653986 0.0002865059017385537\n",
            "4301 1.082753369351849 0.0002863555049293774\n",
            "4401 1.0459589868987678 0.0002862053447172013\n",
            "4501 1.0592919351911405 0.00028605542048233684\n",
            "4601 1.1034397517796606 0.0002859057316073656\n",
            "4701 1.1311876591207692 0.00028575627747712837\n",
            "4801 1.070465801298269 0.00028560705747871445\n",
            "4901 1.199700104945805 0.00028545807100145134\n",
            "5001 1.2379299406893551 0.00028530931743689397\n",
            "5101 1.1045849512156565 0.00028516079617881457\n",
            "5201 0.9958119990806154 0.000285012506623192\n",
            "5301 1.620139996672151 0.00028486444816820157\n",
            "5401 1.0613090786646353 0.0002847166202142048\n",
            "5501 1.175794189737644 0.0002845690221637393\n",
            "5601 1.1241089710965753 0.00028442165342150834\n",
            "5701 1.160597581154434 0.000284274513394371\n",
            "5801 1.0310369414401066 0.0002841276014913322\n",
            "5901 0.9960121748881647 0.0002839809171235324\n",
            "6001 0.9698299318188219 0.00028383445970423817\n",
            "6101 1.1415085992775857 0.0002836882286488319\n",
            "6201 1.0641657677479088 0.0002835422233748022\n",
            "6301 1.0828722650112468 0.00028339644330173413\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "35JC6i9QUgSB"
      },
      "source": [
        "1 10.825187489390373 6.987712429686844e-07\n",
        "101 9.447168171405792 3.56373333914029e-05\n",
        "201 7.142856806516647 7.057589553983712e-05\n",
        "301 6.237934365868568 0.00010551445768827134\n",
        "401 5.762486848048866 0.00014045301983670557\n",
        "501 5.415792358107865 0.00017539158198513977\n",
        "601 5.081815680023283 0.000210330144133574\n",
        "701 4.788327748770826 0.00024526870628200823\n",
        "801 4.381739928154275 0.0002802072684304424\n",
        "901 4.55433791608084 0.00031514583057887664\n",
        "1001 4.911875109748507 0.0003500843927273108\n",
        "1101 4.0579032292589545 0.0003850229548757451\n",
        "1201 4.2276234351193125 0.0004199615170241793\n",
        "1301 3.932735869428143 0.00045490007917261356\n",
        "1401 3.8179439397063106 0.0004898386413210477\n",
        "1501 3.3608515430241823 0.000524777203469482\n",
        "1601 3.832796103321016 0.0005597157656179162\n",
        "1701 2.907085266895592 0.0005946543277663504\n",
        "1801 3.5280659823838505 0.0006295928899147847\n",
        "1901 2.895841649500653 0.0006645314520632189\n",
        "2001 3.273784235585481 0.000699470014211653\n",
        "2101 3.181488689899197 0.0007344085763600873\n",
        "2201 3.4151616653980454 0.0007693471385085215\n",
        "2301 3.4343731447588652 0.0008042857006569557\n",
        "2401 3.0505455391539726 0.0008392242628053899\n",
        "2501 2.8089329147478566 0.0008741628249538242\n",
        "2601 2.7827929875456903 0.0009091013871022583\n",
        "2701 2.4428516102489084 0.0009440399492506926\n",
        "2801 2.4015486147254705 0.0009789785113991267\n",
        "2901 2.3568112018401735 0.001013917073547561\n",
        "3001 2.6349758653668687 0.0010488556356959952\n",
        "3101 2.5981983028614195 0.0010837941978444295\n",
        "3201 2.666826274838968 0.0011187327599928637\n",
        "3301 3.0092043554177508 0.0011536713221412978\n",
        "3401 2.4580375660589198 0.0011886098842897321\n",
        "3501 2.586465588421561 0.0012235484464381662\n",
        "3601 2.5663993963389657 0.0012584870085866006\n",
        "3701 2.9430236657499336 0.0012934255707350347\n",
        "3801 2.464644919440616 0.001328364132883469\n",
        "3901 2.7124062888276512 0.0013633026950319032\n",
        "4001 2.646443709731102 0.0013971932312809247\n",
        "4101 2.7294750874862075 0.001380057517579748\n",
        "4201 2.1295202329056337 0.0013635372009002666\n",
        "4301 2.596563663915731 0.001347596306985731\n",
        "4401 2.1265982036820787 0.0013322017384983986\n",
        "4501 2.3880532500334084 0.0013173229858148\n",
        "4601 2.6129120760888327 0.0013029318725783852\n",
        "4701 2.2873719420749694 0.001289002331178292\n",
        "4801 2.4949760700110346 0.0012755102040816328\n",
        "4901 2.496607314562425 0.001262433067573089\n",
        "5001 2.1889712483389303 0.0012497500749750088\n",
        "5101 1.8677761815488338 0.0012374418168536253\n",
        "5201 2.2992054556962103 0.0012254901960784316\n",
        "5301 2.664361578106707 0.0012138783159049418\n",
        "5401 2.705850490485318 0.0012025903795063202\n",
        "5501 2.581445264921058 0.0011916115995949978\n",
        "5601 2.2480602325085783 0.0011809281169581616\n",
        "5701 1.9289666265249252 0.0011705269268863989\n",
        "5801 2.4863578918157145 0.0011603958126073107\n",
        "5901 2.632946971571073 0.0011505232849492607\n",
        "6001 2.496141305891797 0.0011408985275576757\n",
        "6101 2.6422974687084206 0.0011315113470699342\n",
        "6201 2.448802186456305 0.0011223521277270118"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}