_12_1_RNN_인코더_디코더.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/yoonjong8739/NLP_study/blob/master/_12_1_RNN_%EC%9D%B8%EC%BD%94%EB%8D%94_%EB%94%94%EC%BD%94%EB%8D%94.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IatqtWk_c3A"
      },
      "source": [
        "# Sequence-to-Seqeunce\n",
        "\n",
        "* https://wikidocs.net/24996"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VgLvc6qn_u1a"
      },
      "source": [
        "* Sequence-to-sequence(Seq2Seq)는 입력된 시퀀스로부터 다른 도메인의 시퀀스를 출력하는 모델\n",
        "* 예를 들어, 한국어 도메인을 가지는 문장을 입력해 중국어 도메인에 해당하는 문장을 얻을 수 있음\n",
        "* Seq2Seq는 다른 특별한 기술을 이용하는 것이 아닌 지금까지 배운 RNN 기술들을 조합해 만들며, encoder와 decoder로 구성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9R3B2xOHI9a"
      },
      "source": [
        "# 기계 번역 데이터"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-SapxBVjHNYB"
      },
      "source": [
        "* 일반적인 자연어 처리의 경우, 입력 시퀀스와 출력 시퀀스의 길이가 동일함\n",
        "* Seq2Seq는 입력 시퀀스와 출력 시퀀스의 길이가 다를 수 있다고 가정\n",
        "* Seq2Seq에는 인코더의 입력, 디코더의 입력, 디코더의 출력에 해당하는 데이터가 필요\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EcRSvvCxIEpo"
      },
      "source": [
        "* 데이터는 영어 문장과 그에 해당하는 프랑스어 문장이 존재하는 기계 번역 데이터를 사용\n",
        "* url 주소에서 데이터를 받아오고, 필요없는 열(lic)은 제거\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iwmLeOAuM4Y-"
      },
      "source": [
        "import os\n",
        "import shutil\n",
        "import zipfile\n",
        "\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import urllib3\n",
        "\n",
        "from keras.utils import pad_sequences, to_categorical\n",
        "from keras.layers import Input, LSTM, Embedding, Dense\n",
        "from keras.models import Model\n",
        "import numpy as np"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WDoEpzFPDzMc"
      },
      "source": [
        "* http://www.manythings.org/anki/fra-eng.zip"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# 에러 발생 시 그냥 알집에서 꺼내쓰세요...\n",
        "http = urllib3.PoolManager()\n",
        "url ='http://www.manythings.org/anki/fra-eng.zip'\n",
        "filename = 'fra-eng.zip'\n",
        "path = os.getcwd()\n",
        "zipfilename = os.path.join(path, filename)\n",
        "with http.request('GET', url, preload_content=False) as r, open(zipfilename, 'wb') as out_file:\n",
        "    shutil.copyfileobj(r, out_file)\n",
        "\n",
        "with zipfile.ZipFile(zipfilename, 'r') as zip_ref:\n",
        "    zip_ref.extractall(path)\n"
      ],
      "metadata": {
        "id": "nCCoEy8ZbrIo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "outputId": "1516961e-9fa2-44ea-a22b-4105f7ad5630"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "BadZipFile",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mBadZipFile\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-2-660b54dfaad4>\u001b[0m in \u001b[0;36m<cell line: 9>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      7\u001b[0m     \u001b[0mshutil\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopyfileobj\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0;32mwith\u001b[0m \u001b[0mzipfile\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mZipFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mzipfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mzip_ref\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m     \u001b[0mzip_ref\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextractall\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.10/zipfile.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, file, mode, compression, allowZip64, compresslevel, strict_timestamps)\u001b[0m\n\u001b[1;32m   1267\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1268\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'r'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1269\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_RealGetContents\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1270\u001b[0m             \u001b[0;32melif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m'w'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'x'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1271\u001b[0m                 \u001b[0;31m# set the modified flag so central directory gets written\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.10/zipfile.py\u001b[0m in \u001b[0;36m_RealGetContents\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1334\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mBadZipFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"File is not a zip file\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1335\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mendrec\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1336\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mBadZipFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"File is not a zip file\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1337\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1338\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mendrec\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mBadZipFile\u001b[0m: File is not a zip file"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ls"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ocmDp5uRuJ4i",
        "outputId": "b1171d03-eedb-47fd-d16e-923ec246faed"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_about.txt  fra.txt  sample_data\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TJyvXEu8M05W",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2623c13f-6733-4353-d82f-1bce19c82e67"
      },
      "source": [
        "lines = pd.read_csv('fra.txt', names=['src', 'tar', 'lic'], sep='\\t')\n",
        "\n",
        "del lines['lic']\n",
        "print('전체 샘플의 개수 :',len(lines))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "전체 샘플의 개수 : 227815\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lines = lines.loc[:, 'src':'tar']\n",
        "lines = lines[0:60000] # 6만개만 저장\n",
        "lines.sample(10)"
      ],
      "metadata": {
        "id": "EfHAgfEbpa9G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "outputId": "f7551958-605a-442a-9b5a-ea0973107089"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                          src                              tar\n",
              "17185        Take an aspirin.            Prenez de l'aspirine.\n",
              "40960    I lost the election.        J'ai perdu les élections.\n",
              "54805  Did you sleep all day?   As-tu dormi toute la journée ?\n",
              "16939        Nothing's on TV.      Il n'y a rien à la téloche.\n",
              "29726      Tom dyes his hair.        Tom se teint les cheveux.\n",
              "26623      I hate mosquitoes.       Je déteste les moustiques.\n",
              "20120       Here you go, sir.                 Voilà, monsieur.\n",
              "2595              Tom forgot.                      Tom oublia.\n",
              "23743       Try to go slower.  Essayez d'aller plus lentement.\n",
              "40657    I have an open mind.         Je suis ouvert d'esprit."
            ],
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-5fa05608-df55-4c39-a546-ef57f0a01c88\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>src</th>\n",
              "      <th>tar</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>17185</th>\n",
              "      <td>Take an aspirin.</td>\n",
              "      <td>Prenez de l'aspirine.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40960</th>\n",
              "      <td>I lost the election.</td>\n",
              "      <td>J'ai perdu les élections.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>54805</th>\n",
              "      <td>Did you sleep all day?</td>\n",
              "      <td>As-tu dormi toute la journée ?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16939</th>\n",
              "      <td>Nothing's on TV.</td>\n",
              "      <td>Il n'y a rien à la téloche.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29726</th>\n",
              "      <td>Tom dyes his hair.</td>\n",
              "      <td>Tom se teint les cheveux.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26623</th>\n",
              "      <td>I hate mosquitoes.</td>\n",
              "      <td>Je déteste les moustiques.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20120</th>\n",
              "      <td>Here you go, sir.</td>\n",
              "      <td>Voilà, monsieur.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2595</th>\n",
              "      <td>Tom forgot.</td>\n",
              "      <td>Tom oublia.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23743</th>\n",
              "      <td>Try to go slower.</td>\n",
              "      <td>Essayez d'aller plus lentement.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40657</th>\n",
              "      <td>I have an open mind.</td>\n",
              "      <td>Je suis ouvert d'esprit.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5fa05608-df55-4c39-a546-ef57f0a01c88')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-fac2c4fc-0570-4285-a76f-d7ffc6a1a8a1\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-fac2c4fc-0570-4285-a76f-d7ffc6a1a8a1')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-fac2c4fc-0570-4285-a76f-d7ffc6a1a8a1 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5fa05608-df55-4c39-a546-ef57f0a01c88 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5fa05608-df55-4c39-a546-ef57f0a01c88');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ps4YN9SOOHle"
      },
      "source": [
        "*   데이터를 모두 사용할 경우 많은 시간이 소요되기 때문에, 일부 데이터만 사용\n",
        "*   목표 데이터에는 시작과 끝을 나타내는 토큰이 포함되어야 함\n",
        "*   여기서는 '\\t'와 '\\n'을 각각 시작과 끝을 나타내는 토큰으로 사용"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a2VkcWatNMMG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "outputId": "27d4f896-f2b1-4a5c-f9d3-5dd7915a283f"
      },
      "source": [
        "lines.tar = lines.tar.apply(lambda x : '\\t '+ x + ' \\n')\n",
        "lines.sample(10)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                          src                               tar\n",
              "27057      I owe you so much.            \\t Je te dois tant. \\n\n",
              "5785            Is it nearby?  \\t Est-ce dans les environs ? \\n\n",
              "28979      That was a secret.  \\t Il s'agissait d'un secret. \\n\n",
              "55134  Don't eat the oysters.   \\t Ne mange pas les huîtres. \\n\n",
              "41088    I quickly ate lunch.    \\t J'ai rapidement déjeuné. \\n\n",
              "53720   You are not Japanese.  \\t Vous n'êtes pas japonaise. \\n\n",
              "18481        We've upset Tom.            \\t On a énervé Tom. \\n\n",
              "24914       You're prisoners.      \\t Vous êtes prisonniers. \\n\n",
              "56416  I don't like studying.    \\t Je n'aime pas apprendre. \\n\n",
              "34382     I'm being promoted.             \\t Je suis promue. \\n"
            ],
            "text/html": [
              "\n",
              "\n",
              "  <div id=\"df-d04228be-6b77-4889-8e3d-6f2760108521\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>src</th>\n",
              "      <th>tar</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>27057</th>\n",
              "      <td>I owe you so much.</td>\n",
              "      <td>\\t Je te dois tant. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5785</th>\n",
              "      <td>Is it nearby?</td>\n",
              "      <td>\\t Est-ce dans les environs ? \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28979</th>\n",
              "      <td>That was a secret.</td>\n",
              "      <td>\\t Il s'agissait d'un secret. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>55134</th>\n",
              "      <td>Don't eat the oysters.</td>\n",
              "      <td>\\t Ne mange pas les huîtres. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41088</th>\n",
              "      <td>I quickly ate lunch.</td>\n",
              "      <td>\\t J'ai rapidement déjeuné. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53720</th>\n",
              "      <td>You are not Japanese.</td>\n",
              "      <td>\\t Vous n'êtes pas japonaise. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18481</th>\n",
              "      <td>We've upset Tom.</td>\n",
              "      <td>\\t On a énervé Tom. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24914</th>\n",
              "      <td>You're prisoners.</td>\n",
              "      <td>\\t Vous êtes prisonniers. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>56416</th>\n",
              "      <td>I don't like studying.</td>\n",
              "      <td>\\t Je n'aime pas apprendre. \\n</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34382</th>\n",
              "      <td>I'm being promoted.</td>\n",
              "      <td>\\t Je suis promue. \\n</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d04228be-6b77-4889-8e3d-6f2760108521')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "\n",
              "\n",
              "\n",
              "    <div id=\"df-c805089a-0ab6-46d1-ad08-5288598dde68\">\n",
              "      <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c805089a-0ab6-46d1-ad08-5288598dde68')\"\n",
              "              title=\"Suggest charts.\"\n",
              "              style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "      </button>\n",
              "    </div>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "    background-color: #E8F0FE;\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: #1967D2;\n",
              "    height: 32px;\n",
              "    padding: 0 0 0 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: #E2EBFA;\n",
              "    box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: #174EA6;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "    background-color: #3B4455;\n",
              "    fill: #D2E3FC;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart:hover {\n",
              "    background-color: #434B5C;\n",
              "    box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "    filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "    fill: #FFFFFF;\n",
              "  }\n",
              "</style>\n",
              "\n",
              "    <script>\n",
              "      async function quickchart(key) {\n",
              "        const containerElement = document.querySelector('#' + key);\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      }\n",
              "    </script>\n",
              "\n",
              "      <script>\n",
              "\n",
              "function displayQuickchartButton(domScope) {\n",
              "  let quickchartButtonEl =\n",
              "    domScope.querySelector('#df-c805089a-0ab6-46d1-ad08-5288598dde68 button.colab-df-quickchart');\n",
              "  quickchartButtonEl.style.display =\n",
              "    google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "}\n",
              "\n",
              "        displayQuickchartButton(document);\n",
              "      </script>\n",
              "      <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d04228be-6b77-4889-8e3d-6f2760108521 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d04228be-6b77-4889-8e3d-6f2760108521');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F2DHfeeSOsAo"
      },
      "source": [
        "\n",
        "\n",
        "*   해당 예제에서는 글자 단위로 예측, 따라서 글자 집합을 구축해주어야 함\n",
        "*   구축한 다음, 정렬해 인덱스를 부여해 글자에 해당하는 사전을 만듬\n",
        "*   사전은 글자를 모델에 투입하도록 변환하거나 예측시 반환되는 인덱스들을 글자로 변환할 때 사용\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T18GljqANlT1"
      },
      "source": [
        "# 문자 집합 구축\n",
        "src_vocab = set()\n",
        "for line in lines.src: # 1줄씩 읽음\n",
        "    for char in line: # 1개의 문자씩 읽음\n",
        "        src_vocab.add(char)\n",
        "\n",
        "tar_vocab = set()\n",
        "for line in lines.tar:\n",
        "    for char in line:\n",
        "        tar_vocab.add(char)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HCIiItcXNoEh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6626e5b9-a226-418d-9a19-46b553ad1105"
      },
      "source": [
        "src_vocab = sorted(list(src_vocab))\n",
        "tar_vocab = sorted(list(tar_vocab))\n",
        "\n",
        "src_vocab_size = len(src_vocab) + 1  # 공백 word를 고려\n",
        "tar_vocab_size = len(tar_vocab) + 1\n",
        "\n",
        "print('source 문장의 char 집합 :',src_vocab_size)\n",
        "print('target 문장의 char 집합 :',tar_vocab_size)\n",
        "\n",
        "src_to_index = dict([(word, i+1) for i, word in enumerate(src_vocab)])\n",
        "tar_to_index = dict([(word, i+1) for i, word in enumerate(tar_vocab)])\n",
        "\n",
        "print(src_to_index)\n",
        "print(tar_to_index)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "source 문장의 char 집합 : 80\n",
            "target 문장의 char 집합 : 104\n",
            "{' ': 1, '!': 2, '\"': 3, '$': 4, '%': 5, '&': 6, \"'\": 7, ',': 8, '-': 9, '.': 10, '/': 11, '0': 12, '1': 13, '2': 14, '3': 15, '4': 16, '5': 17, '6': 18, '7': 19, '8': 20, '9': 21, ':': 22, '?': 23, 'A': 24, 'B': 25, 'C': 26, 'D': 27, 'E': 28, 'F': 29, 'G': 30, 'H': 31, 'I': 32, 'J': 33, 'K': 34, 'L': 35, 'M': 36, 'N': 37, 'O': 38, 'P': 39, 'Q': 40, 'R': 41, 'S': 42, 'T': 43, 'U': 44, 'V': 45, 'W': 46, 'X': 47, 'Y': 48, 'Z': 49, 'a': 50, 'b': 51, 'c': 52, 'd': 53, 'e': 54, 'f': 55, 'g': 56, 'h': 57, 'i': 58, 'j': 59, 'k': 60, 'l': 61, 'm': 62, 'n': 63, 'o': 64, 'p': 65, 'q': 66, 'r': 67, 's': 68, 't': 69, 'u': 70, 'v': 71, 'w': 72, 'x': 73, 'y': 74, 'z': 75, 'é': 76, 'ï': 77, '’': 78, '€': 79}\n",
            "{'\\t': 1, '\\n': 2, ' ': 3, '!': 4, '\"': 5, '$': 6, '%': 7, '&': 8, \"'\": 9, '(': 10, ')': 11, ',': 12, '-': 13, '.': 14, '0': 15, '1': 16, '2': 17, '3': 18, '4': 19, '5': 20, '6': 21, '7': 22, '8': 23, '9': 24, ':': 25, '?': 26, 'A': 27, 'B': 28, 'C': 29, 'D': 30, 'E': 31, 'F': 32, 'G': 33, 'H': 34, 'I': 35, 'J': 36, 'K': 37, 'L': 38, 'M': 39, 'N': 40, 'O': 41, 'P': 42, 'Q': 43, 'R': 44, 'S': 45, 'T': 46, 'U': 47, 'V': 48, 'W': 49, 'X': 50, 'Y': 51, 'a': 52, 'b': 53, 'c': 54, 'd': 55, 'e': 56, 'f': 57, 'g': 58, 'h': 59, 'i': 60, 'j': 61, 'k': 62, 'l': 63, 'm': 64, 'n': 65, 'o': 66, 'p': 67, 'q': 68, 'r': 69, 's': 70, 't': 71, 'u': 72, 'v': 73, 'w': 74, 'x': 75, 'y': 76, 'z': 77, '\\xa0': 78, '«': 79, '»': 80, 'À': 81, 'Ç': 82, 'É': 83, 'Ê': 84, 'Ô': 85, 'à': 86, 'â': 87, 'ç': 88, 'è': 89, 'é': 90, 'ê': 91, 'ë': 92, 'î': 93, 'ï': 94, 'ô': 95, 'ù': 96, 'û': 97, 'œ': 98, '\\u2009': 99, '‘': 100, '’': 101, '\\u202f': 102, '‽': 103}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Flh0swsdPOhc"
      },
      "source": [
        "*   인코더에 입력될 입력 데이터를 구성\n",
        "*   문장의 글자 하나씩을 사전을 이용해 인덱스로 변환해 리스트에 넣음\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rgtp58H-NpBm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8a53974a-5eb8-48c2-f180-7c01ba9d946b"
      },
      "source": [
        "encoder_input = []\n",
        "\n",
        "# 1개의 문장\n",
        "for line in lines.src:\n",
        "  encoded_line = []\n",
        "  # 각 줄에서 1개의 char\n",
        "  for char in line:\n",
        "    # 각 char을 정수로 변환\n",
        "    encoded_line.append(src_to_index[char])\n",
        "  encoder_input.append(encoded_line)\n",
        "print('source 문장의 정수 인코딩 :',encoder_input[:5])"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "source 문장의 정수 인코딩 : [[30, 64, 10], [30, 64, 10], [30, 64, 10], [30, 64, 10], [31, 58, 10]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "agsrqdb7PQP5"
      },
      "source": [
        "\n",
        "\n",
        "*   디코더에 입력될 입력 데이터를 구성\n",
        "*   인코더 입력 데이터 처리와 동일하나, 목표 데이터에 해당하는 사전을 사용해주어야 함\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ge1w7o54NqVA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c71fee19-a7da-43b5-9344-3aa207ae58f0"
      },
      "source": [
        "decoder_input = []\n",
        "for line in lines.tar:\n",
        "  encoded_line = []\n",
        "  for char in line:\n",
        "    encoded_line.append(tar_to_index[char])\n",
        "  decoder_input.append(encoded_line)\n",
        "print('target 문장의 정수 인코딩 :',decoder_input[:5])"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "target 문장의 정수 인코딩 : [[1, 3, 48, 52, 3, 4, 3, 2], [1, 3, 39, 52, 69, 54, 59, 56, 14, 3, 2], [1, 3, 31, 65, 3, 69, 66, 72, 71, 56, 3, 4, 3, 2], [1, 3, 28, 66, 72, 58, 56, 3, 4, 3, 2], [1, 3, 45, 52, 63, 72, 71, 3, 4, 3, 2]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XOCf-xuEPSm1"
      },
      "source": [
        "\n",
        "*   디코더의 출력과 비교할 목표 데이터를 구성\n",
        "*   디코더의 입력 데이터를 구성할 때와 동일하나, 시작 토큰을 제외해주어야 함\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KogHdxHsNrl2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "074b40cd-3c55-4629-bc88-c0c2a015c339"
      },
      "source": [
        "decoder_target = []\n",
        "for line in lines.tar:\n",
        "  timestep = 0\n",
        "  encoded_line = []\n",
        "  for char in line:\n",
        "    if timestep > 0:\n",
        "      encoded_line.append(tar_to_index[char])\n",
        "    timestep = timestep + 1\n",
        "  decoder_target.append(encoded_line)\n",
        "print('target 문장 레이블의 정수 인코딩 :',decoder_target[:5])"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "target 문장 레이블의 정수 인코딩 : [[3, 48, 52, 3, 4, 3, 2], [3, 39, 52, 69, 54, 59, 56, 14, 3, 2], [3, 31, 65, 3, 69, 66, 72, 71, 56, 3, 4, 3, 2], [3, 28, 66, 72, 58, 56, 3, 4, 3, 2], [3, 45, 52, 63, 72, 71, 3, 4, 3, 2]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5a5A0lzdPWad"
      },
      "source": [
        "\n",
        "\n",
        "* 각각의 데이터를 동일한 길이로 맞춰줌\n",
        "* 길이를 맞춰줄 때는 해당 데이터의 최대 길이로 맞춰줌\n",
        "* 원 핫 인코딩을 통해 원 핫 벡터로 변환\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8BAyplv9Nuan",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4ded040a-f14c-4a1b-9ebb-cb73e286c17d"
      },
      "source": [
        "max_src_len = max([len(line) for line in lines.src])\n",
        "max_tar_len = max([len(line) for line in lines.tar])\n",
        "\n",
        "print('source 문장의 최대 길이 :',max_src_len)\n",
        "print('target 문장의 최대 길이 :',max_tar_len)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "source 문장의 최대 길이 : 22\n",
            "target 문장의 최대 길이 : 76\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "encoder_input = pad_sequences(encoder_input, maxlen=max_src_len, padding='post')\n",
        "decoder_input = pad_sequences(decoder_input, maxlen=max_tar_len, padding='post')\n",
        "decoder_target = pad_sequences(decoder_target, maxlen=max_tar_len, padding='post')"
      ],
      "metadata": {
        "id": "GkhrCNnbrFKr"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "urrwjhZhNvse"
      },
      "source": [
        "encoder_input = to_categorical(encoder_input)\n",
        "decoder_input = to_categorical(decoder_input)\n",
        "decoder_target = to_categorical(decoder_target)"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xc68H8R9EXrK"
      },
      "source": [
        "## 인코더(Encoder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L6Jtuq5QGHdm"
      },
      "source": [
        "![encoder.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAUcAAADoCAYAAAB8dCbkAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAhdEVYdENyZWF0aW9uIFRpbWUAMjAyMDowOToyMiAyMjo1MTozNtm7QSYAABnBSURBVHhe7d0LmFxlfcfxd/a+ySbcDESEhJvcNJBgFaVBgrRcRCRRnhq0KgUtDVWLz4NgfSCUwONj0HopFqotlLZo0NomUpVLpUKCKFDZKGIEQlwSpNwScs9esjud37vvu3t29p2Zc87O7uzMfD/PM8+c885lZ37zzv+85z1ndzPZHAMAGKHBXQMAIiiOABBAcQSAAIojAARQHAEggOIIAAEURwAIoDgCQADFEQACKI4AEEBxBIAAiiMABFAcASCA4ggAARRHAAigOAJAAMURAAIojgAQQHEEgACKIwAEUBwBIIDiCAABFEcACKA4AkAAxREAAiiOABBAcQSAAIojAARQHAEggOIIAAEURwAIoDgCQADFEQACKI4AEEBxBIAAiiMABFAcASCA4ggAARRHAAigOAJAAMURAAIym6+Zl3XLAACHkSMABAyNHPdf9rhtAIB6t2XpSYwcASCE4ggAARRHAAigOAJAAMURAAIojgAQQHEEgIAJK47bv/Gn9tyhibLr+9fbn9fz+CrXAgDxle0kcBWhXauWubVc1Z02w+z7mXvd2mBx3Pv734z6OTu+dbnpe2q1WwtrfcsiM/X8a9zaYOHr+cVKtzbS9EtuM02z5w7dZ+rCpab1pIXu1snP51FtrxuoJWU7CdwXxuZj3mmLny+AcUaK0z701aHH5F9UIIrR7fmPUWEsZesXzxp6bX45elFbtfIjdEbMwNiUpTj2rnvAXqvQeRrtyZ7/+Qd7PVloZDaw4xVbSKM04vQFNjrirTbTL73DNL3h+BGjeADJ1dUBmb3PrbW7rL5w16opZ19hrzW1ACCdshTHluMW2GuNyjw/J9j+rr+w1yHa9cvfpY1eko5+oo8NzUn2rP0ve9069zx7XYo/qKP3FX3u/NGw35UN3Z6/257/WP8z/CW781V3y7D8+0SfI9qma91XUwsaPfY9/ZC7F4CkylIcdeBA838alfkvq+TvuubT4/yubOji5xx1cCeO6GNDo8P+F5+yzxWal9x+68VDrz1/xNX/wrqh55U9D3zTXouKnw40+fnP6DypnsvvwvvXpMf651dBUxH3c7XatddzRakw6z7tC/7c3kf31XPkzynmbwwaZx5jf7ZGywCSK9tudX6hSzNv50dI/os/sPVFe10uA9tfMQ3Tw4U2OucYPTIuzUfPd0vGjshERUevUwVIBcsfWda1Rsu+AEaLtH9eP6Lre2bwun3+RfZaRVvPFaUNjgq6H4E3vf5Ye+3neT39nOhr9xuU/s1d9hpAMpN6zlGFRxr2nWmvJxMVHV+8GzoOsNchoVGvf18q1lLoCLsf9en+flQbHbVGFZu+AJBcWYujH/mVa1dOoyCNhvyoLJ/mJH3R8BfN/xWiUaMvSOU0sHOzWxrNF8Ko/IIZzSs056j7+1Gtv0TPDAgJ/VwA8U3YyFGnmOhL7akg5Bc2P28WKnq6+CLii2boop9TSLnn4TRaU+HSrq+fCtC15hL97m10LjB/V9vvrvsDRXpsdM7RH1jRa44ehCm2AfA0vyqFNiwAiqvYbrW++KHiVuxSaPczLn+U2hejqOgBGV3i0tyqCqQv6NEj7HrN4p/TH1jxu8AqoJpjVLt/bP6coz9vUbvT/nla33qBu7UwFdn85wIQX1n/h4xGRvlHTfPpaO5EjGb8a8n/eb7d/5phLaqH9wiMJw1C6vIfbPnf856oQj2RfGGMjlABJFO3xREAilFxnNSn8gBApVAcASCA4ggAARRHAAigOAJAAMURAAI4lWcSyPbuNn3rHjA9v/qR6X/5dya7Z5ttq1aZplaTad/HNB10pGk58VzTfNwCk2mZ4m4tL7Ibm+zuraZ33U9y+f3QDLz6/GB+e3vcrdUl09RiGtqmmcyM2ab1hHNNy3Gnm8yUfd2tyZTtPEc6aDrZvb1mz4P/ZLofvsNkMplcZnvcLbUj09Jmsv39pu3ti037aR8zmVznLQeyGxv9sZTuB75hujt/YDINDSbbU73f1xD7fc1mTeu8c03bgkuL/uWskDEXRzpoev0vrzfbb7vUmL49JtvX7VprlzY4JneZ9uGbTNOhc1xrOmQ3Nn3PPmJ2fvvTudoxYEzuO1zTGltsber4wI2m+ZhTXWNpYzoJXB1065fOMT25wmhyHbQWC6Nke3Nfvv4+0/PId83WL59n9m56wt2S3t6ux832b340t0vzWl18uUW7atnu7WbH7Zea3ifuc63JkV367KT3l3ebnd+6fDC7Wi+M0t9r89v575813T+/0zXGk6o40kHTd1BtVHbc8cma3ZiUov6y6/vLUm1kyC59dqIRox5frXOKY6E+s+fHX7fzq3ElLo500PQdVNMQ2h2s1+w8zUfvuONTuY3NDtdSGtkNSpOdaI7R7krXYWH0lN3O711t+rdsci3FJSqOdNBBaTuo5mc1T4ac/h6z+yfhf/kQQnYRCbMTHXyxc4x1LjOw1+y5/2a3Vlyi4kgHjUjYQVVQdeCqXqYhStFcbs9j37OnkpRCdiMlyU50Px2Vros5xhKy/X2m76k1ZmD7S66lsNjFkQ46UtIOqlOddNQMwxoam2LNAZHdaHGzE91Pp+tgUDabNb1P3u/WCoudGB10tCQdVOeA1vt0RL6Bnl2mp/Mut1YY2Y0WNzvRCd61dh7jmOT2fnt/mRtJlxC7ONJBR0vSQXVyPEbrf3WjWyqM7MLiZCf6zReM1L+1jLvVdNCwuB1UvzWEgNwGphSyKyBGdkJ+o+m0vFLizzkScFjcDtrLbk1Itr/0QQKyC4uTndTz6TsFDfS7hcLiF0c6aFDcDgqgusQujgBQTyiOABBAcQSAAIojAARQHAEggOIIAAEURwAIoDgCQADFEQACKI4AEEBxBIAAiiMABFAcASCA4ggAARRHAAigOAJAAMURAAIym6+Zl9XC/ssetw2FbFl6kltCvlLZSbnz2++zg//18LUvnG6vo/Jfz/ZbLzFTF15rGg+Y5VqG9W/eaLZ9beHQY/ZuesJs/8eP2mXP39b72wfNzm9/2i6X00T3vekf/xfTdOgcm8ve5zpd66B9/mrViJy6H/mOadhnpmk59jTXMpJem39Mdve2UZ+Hv83nXG4T3fc6PvgVm8XuH33RdP98hWsd5HP11F8Gtr1o2k7+gGsZSflPOfPyocfkv07/fKFcy6FYdnotFRs56o3rxTXNnudahqlD6TZ/mXLuVfZDibZFL9HH+KIR5W/Tda3T+9cXUR+uLip2oi+m1vVlF11rPfqFVSeMdm5R9vVCfVKFzGfns9IGQev6Mosy9ffxlF1myj6m7e0XuhZj+3Zog1SL9P30GxzlouIpu3+4fERWI/qm2zApO8nva/l9caJNut1qOmh6eq96//2vDP+nSI0C80dHhajjSrSTNh/1jqH2WudHeJ6+2LrEMbBnm+1/LXPOdi3GtP3hR2yb//LXssYZh9v36ftad25UmWQvQ7mrr3m+D1ay70264kgHTc93TO32RDcQSUQ7qd+w9K3/mV2vdeo/er8aBSXV0L6P6du4dsRop3nWXNtWD/T90oZZg5uk9Dj1MWXv9yTVB/1eT6VMuuJYCx100aJFZu3aynwp/O7MlHd/xk4l+M4WV7ST+g1L76/vc7eOv0pm56cYtHFRdkmnFLp/+q/2Wo/Txklfet82kSqRofZQ1Ff03VN2Sb+/fgCkPuc3yr1P3GPbKmXSFcda6KCrVq0y8+bNq0gn1e6Mpho0GS7TL7k1UYH0hVCdtBIblkpmJ8rOb2B0ICFJ/9PIXQVCox7tvWgUHndKo5wqlaEOmvhpL31/kxZIu9eS63N+o6y+XEmTrjhKtXXQTCYz4uJFO+lE03yPz7DlzWfa6ziUle+k2rDs3fCou2X8TLbs/AZGmg4+3l7HpY2JRj26ROd+x0s0u0IZTiT1H2Wn76DmIZPQXov6nPqe+mClTcriKNXUQUvp6upyS+NLI8TonE9D7v3LQMKO5jvpZNh6T1R2Ej2bwY+2B3Ztsddx+b2USu1SV0r+mSC+/yThd6312ErvUsukK47V2EGz2eyIizd37lyzcuVK09k5PiNXvT9NPfhL48yjh+Z8dNGoW0f7kxY430knape6EtmJphx8Vto70Zy1X9dtOiCQ9LxOP/KeiD0WiWZXKMPx4Oe0dfG7z35dF73//PNl41Dmk2GjLBU7CVwjnPzzmPRFbs3tCutL7ymsaMh2dOQ6brRdRVWd258s6ousn8PU+X862BM9r69c8rPTrsy1115rFi4c/lnlzq+WRPMju2RC39v8DMkvrFjNU2b8hkwZlMpOyK8w+l569L30ShXHSTvnCACVRHEEgACKIwAEUBwBIIDiCAABFEcACKA4AkAAxREAAiiOABBAcQSAAIojAARQHAEggOIIAAEURwAIoDgCQADFEQACKI4AEEBxBICA2MUx09TqlhCVaWpxS8WRX1ic/MguLH7fi3e/utLQ6BYKi18c24f/6RUiWqa6heLIr4AY+ZFdATH7XkPbNLcEL9M23S0VFrs4Nh10pFtCVOP+B7ul4sgvLE5+ZBcWt+9lZsx2S/Aa9z3QLRUWuzi2nHiuybS0uzVIpnWKaZlzjlsrjvxGi5sf2Y2WpO+1nqD8prg1ZFracn3qPLdWWOzi2HzcApMdGHBrkOzePtPypjPcWnHkN1rc/MhutCR9r+W40/Xf/90astlMrOzizznmtjxtJ/8Jk+NOprnVtM59j2mYfpBrKY78RkqSH9mNlLjvTdnXtM47N7cvyYEZHYhpOXp+rOxiF0dpP+1jxtBBBzU0m/YzLnMr8ZBfRML8yC4iRd9rW3CpLQx1r6HJtP/xJ9xKcYmKY6Ztmpn24ZtyW64211KfNP/V8cEvm4aOA1xLPOQ3KE1+ZDcobd/T/TsuuME+vl5pD6Tj/GtM4/6HupbiEhVHaTp0jpm68G/qtpOqc009/1rTfPgfuJZkyC99fmQ3tr6nucf2d11mi0S90Xtuf9cS03Liu11LaYmLo7TMOdNM+8jN9vwzHfmpB5rnybR2mI4Pfc2+/7Egv/T5kd3Y+l7bKR8yU99/vTHN7SbT2Oxaa5imEppazdTzPmffexKZzdfMs4ex9l/2uG1IItu9w+y+/2bT84tVxjQ2GdOzy91SO3TKhI4MagJc8zxJd2eKIb/0yG5s+rdsMnty+fU9tcZkdSS7b4+7pTZow2mPSh89384xxt2V9rYsPWlsxdEb2Lk5F/Jq09N5l+l/daPtqNn+Xndr9bG/btUy1Z5kq3PJdNg/7pHBNMgvPbIbm4HtL5neJ+83vb/8genf+lJuo7M919jvbq0yuVGifvNFJ3jrPMaxZFe24lgpa9euNV1dXWbhwoWuBXGRXXpkl0415Vb1xXHRokU27M7OTteCuMguPbJLp5pyU3FMdUBmMtBWaNWqVUPXiI/s0iO7dKoxt6otjtddd51bGrmM0sguPbJLpxpzq8rimL/1YSseH9mlR3bpVGtuQ3OO1eTD391gfvTUNrc2aM7MdvPAx491ayiE7NIju3SqNbeqLI7eAdcPTuzm3oO9Rnxklx7ZpVNtuWWy9gzQ6pTJZOx1Fb+FiiG79MgunWrLrWoPyADAeKI4AkAAxREAAiiOABBAcQSAgMRHq9f+fqf56x/8zjy2abvZvGuva60f+01pMm89dLq57pzZ5u2zS//v23zklz4/siO7tNJkl6g43vzTF8yVd20wu3qr9E8aldG01kZz+WmHmGXnHOZaSiO/YUnzI7thZJdekuxi71Zry0PAw3b09JuvPPi8eWD9VtdSHPmNlCQ/shuJ7NJLkl3s4qghOQGPtDMX9LL7nnNrxZHfaHHzI7vRyC69uNnFLo6aq8Bo2jLHQX5hcfIjuzCySy9OdrGLYz1O4sbx2u54uZBfWJz8yC6M7NKLkx2n8gBAAMURAAIojgAQMGmK49Ofe5v51Dvf4NaSyX7lNLc0km+/6X1Hme9f8ma7XKsKZVCKMlf2+aLtP7t8XurPphqQXXq1nF1djBw/+Z/rzfm3/tqtIal3fLXT/N3q37s1JEF26VU6O3arASAg9q8PZj79oFsqTruwF5/8ert82yP/Z0dt2qU96ZAOs/+UZrOnr9+sePxlc+FJB5r25kbztw9sMkvv7rJD6c27+8wJr++wj/XtH5g3w/z9+99oDpjabO5/+jXzR7f8yszer83895ITzBtntJtnXtljr/X6CrX7XWqNHrUb8PPnttvfr9R9jv78o/a2H+ced8bR+5nNu/rsuk4SjbvVirNrETe/6Ov4y/94xnyn85Wh3Qz/vh7Jvf73nTDDZrnotifNmg3b7GtQPv6xvj30eUQz1fOJcijUrp//9YcGs/jM6Yfa60P2bTV3/XqzzbRQ7nGVyo/sCiO78cuurCNHvdiFc15njv/CY/Zy1rH72zbZsnuvmXrVGnP3ui22MCrEi+/8rVk870B7u7y8o8/eR+1/9raZtu36cw63H5Z/0/qdyFsXH23WvbTbtvnwpFB7PhVn3WdP34Cd09BzzuhoMYcte8S+rkrR65ja2mhfm16H3runjulf8wkHd9h8leUVrtPIXU9utvfR+7vxvUcU/Dy+9N4jbfHXfZWXV6g9qr25wXzw39bZrP7w8MFf4I+b+3giu/TILqysxfH8OQfYyt619GR7USVXm3Rt6bbXj23aYQulti7aOkXd/8xr9lrt+jBEz3HnR463VV5bpxNzH9CsXNB+DjE6uivUns/f5l+TnvPW3Nbtude67evS66sEvQ6NaPVeV39yrn3vnnITvWZd9Fp9m+ffl7bSB+RG6YU+D2Xr7+szl0LtUf6z08/3OcXNfTyRXXpkF1bW4tj5/E7zqxd22equaq5Lkhd9xhv3s9cazWlLIRouv/OmtUPPpzC0+61hu0SPZhVqL2VjLnCNZuXUI/bJ7f432eWJptehXQb/XnVJwr9nTSPouQp9HspWuzLiM5dC7aWkzb2cyC49sgsra3FUZV/97Fbzm8++1W6FdEniuIOm2McsPXO2ueKuZ23bNXf/zqy8+E1Dz6cQ9FdGVMy0/t43DY5MpVB7KX6Lp8fduvgYuyXTBzzR9DqmtjQMvVc/5xPXJ+a/wT5OOV5y59MFP49/fvRFc9viY82u5aeaA6c12zYp1F5Kfu6ae5poZJce2YWV/YBMtdPIUcX4dVc/7FpK8x9+MfWSn+avzn/z68yJX/xf11JaqfzIrjCyGzQe2ZV15FitNBpVULqoMOqPgyI+7U75/HSA7fM/jvdn3EB2YzHe2TFyLAN9OKWQX2Gl8iO7wsguvVLZMXIEgACKIwAEUBwBICB2cdS/NsRoHS2Nbqk48guLkx/ZhZFdenGyi10c9T9fMdpbZk1zS8WRX1ic/MgujOzSi5Nd7OKof4at//mKYdPbmswVCw5xa8WR32hx8yO70cguvbjZxS6O+t1L/TPsDoK2FPDHTp5p3hPzN3HIb6Qk+ZHdSGSXXpLsYp/n6OmfYesvaDy+aafZ1l2ZP9BQSZqr0JBcW564hTGK/NLnR3Zkl1aa7BIXRwCoB5zKAwABFEcACKA4AkAAxREAAqq6OK5du9asWrXKrSEJskuP7NKpttyq+mj1okWLTFdXl+ns7HQtiIvs0iO7dKouNxXHapQLWEXdXlauXOlaEQfZpUd26VRjblVbHBcuXDgU9ty5c10r4iC79MgunWrMrSqLY3Qr5C9sxeMhu/TILp1qza0qi2N0K+QvbMXjIbv0yC6das2tqg/IZDIZe13Fb6FiyC49skun2nLjPEcACKA4AkAAxREAAiiOABBAcQSAAIojAARQHAEggOIIAAEURwAIoDgCQADFEQACKI4AEEBxBIAAiiMABFT1nywDgPHCyBEAAiiOABBAcQSAAIojAARQHAEggOIIYMLceOON5qijjnJrk1vVFseHHnrI/jezDRs2uBYAKB9GjgAQQHEEUJB2g88++2y3Nnn5PclyojgCQADFsUZddtlldkvqL3feeae7xdh5Wj9fG72Ptr4SbdPIAYOKZVrN/KjLX/w8vpavuuoqc++999plfyBFOeSPJn1f8n3I02P88xYageb/fD1/lJ5DWUfzjx7U0fOeeuqpdtnfXpZjEfrd6mq0Zs0a/U549tlnn3Ut8FasWGEvnpajWela69GPf/ny5Xb9yCOPHLofGQ8rlWm18n1Bn7W3ZMkStzTYL8466yy3Nki357eFnkfr0efy/Ul9zAv1Md0efZzWdYnmX+i5y4niWCeUle+4oY4satOXISq/U2JYKMNqU6qopC2Oely0CHr57aH+pefIv09+v8x/nlLvIw12q2tU/q6KPP/88/baO/jgg93SsFmzZrmlYRs3bnRL9S1OptVm/vz5Jlfo7PvJ350di66uLnPmmWe6tcJyRdVceOGFI3LVLrLao/L7ZaiflhvFsQZpDuaiiy7SZnTogrGp5Uzvuece+35uueUWW5wmep45N+obkau/VBrFscZoIloT6Pfdd59rGWxDevWSqQqSCpUOwhRz2GGHuaVhL7zwglsapPuEMtKIMiq3a2wefvhhtza5UBxrzBFHHGGvH330UXst5dxdqke1nKmmCqJH3TVNoILlafd1/fr1bm3QKaecYjcW0SPTGlVHXXDBBfY+0VGofo5Gp1E33HCDLcbR59Jy0tGrnyIq50aL4liDtPWPzuNcffXV7hakVauZas7x9ttvH3pfeo/RYrh48WJ7rdv86TN6zPLly+3coH+cniNKGxTNG6rw+fusXr3aPi5Kz79ixYoRz6WCeeWVV7p7xKOft2TJElvY9RzlKJL8mwQAGMWY/wfDaMVnmP6A7wAAAABJRU5ErkJggg==)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQYNntGgExi5"
      },
      "source": [
        "\n",
        "\n",
        "* encoder는 입력 문장을 받는 여러 개의 RNN cell\n",
        "* 입력은 단어 토큰화로 단어 단위로 쪼개지고, 이는 각 시점의 encoder 입력이 됌\n",
        "* encoder는 모든 단어를 입력받고 마지막 시점의 은닉 상태를 decoder RNN cell의 첫번째 은닉 상태로 넘겨주며,\n",
        "이를 컨텍스트 벡터(context vector)라고 함\n",
        "* encoder는 입력 시퀀스를 컨텍스트 벡터라는 고정 길이 벡터로 압축하고자 함\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2nWXvDTvBcxh"
      },
      "source": [
        "## 인코더 모델 구성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XmoZbTM68l3n"
      },
      "source": [
        "\n",
        "\n",
        "*   encoder의 구성은 일반 lstm 모델과 동일\n",
        "*   lstm 안의 return_state는 은닉 상태를 반환해줘 seq2seq 모델을 구성할 때 필요함\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "puPexCSJEZb_"
      },
      "source": [
        "encoder_inputs = Input(shape=(None, src_vocab_size))\n",
        "encoder_lstm = LSTM(units=256, return_state=True)\n",
        "\n",
        "# encoder_outputs은 여기서는 불필요\n",
        "encoder_outputs, state_h, state_c = encoder_lstm(encoder_inputs)  # state_h(은닉상태), state_c(셀 상태)\n",
        "\n",
        "# LSTM은 바닐라 RNN과는 달리 상태가 두 개. 은닉 상태와 셀 상태.\n",
        "encoder_states = [state_h, state_c]  # 켄텍스트 벡터"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rjbam2VaEa4q"
      },
      "source": [
        "## 디코더(Decoder)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i5xvLcDgF_u3"
      },
      "source": [
        "![decoder.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZoAAADoCAYAAADWmyIdAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAhdEVYdENyZWF0aW9uIFRpbWUAMjAyMDowOToyMiAyMjo1MTozNtm7QSYAACSwSURBVHhe7d0PuBRlvQfw9/zhjwjIPxUB4XhApCwBSeghDoY+KVS3ON2nK3irx5uVQlraRagbWRSWcOnP1S5cLYv7dC/SHy9UpoCFhmRpKlAaKgc4AoIa/0H+HeDc+b77/g7vzpndnZnd2d3Z+X6eZ57dmZ2dnfmdd97f+74zu6eq1aGIiIgiUm0eicinqqoqtXTpUjNHRLnEItHgpMbJvWXLFrOEohLnWK9du7bo+47PYuIhyo49GqoYN9xwg3ryySdVfX29WUJE5YDXaKgizJ8/Xz/OnDlTP9rQ40ACGjdunFmSH2zvwQcfVFOmTDFLolHo/SYqFfZoqCKMHTvWM8kQUenFItHI2Lsblsk0ZMgQs5QQC4mLtPRh4sSJavr06WYuxX1dwyvWeJ9sz/3+coD9a2hoaNtHORYcuxyLvI5rKXJdBcdqw7FhWza5ZiWT+z3gXi7bl8ldNvE5mNzbzrXf5cTeX+E+bncswX3MmNwQG3nNvQ3EphzLIGUX2x4NCiGGLzDyh+maa65hsnHgxEQsJC7grhCCkJNatjd+/HjPyrZUcLy4JiP7hzIxePBgfczo4UgMMASF50GGu1CpTZ06VW3evLlt+3PnzjWvZoaYyfqYZJlt1apVas2aNW3rXHvttfrvBvnud5QkASDO9rUwxBtxl/2V/bcThcRTXsc0bdo0vT2BdRAbeX327NlpSVZ6rXgPE06MOH/MsucUXpRaM9fa6hROPblhHadSMHPJ5JzsrU4lYObSOZVZu7hJbCVu7ljjPfPmzTNz5QXHieN1wz7bMcDx4LgEjtW9DBAbvFdgHa9Yupd7bcuG+NnbdX8OuP8OkGu7xSLxcu+fzaucyPvs517Hg7+hvDfTue0F78E23bGk8hPLHg1aT4sWLWprXckEO3fu1I9JhRY3Wo0Sj3yhRTlr1iy9vXx6RlHYtm2b7m3YZQDTypUr9Wv5kGMdPXq0fgwKLXnZH8SvqanJvJLivjOuX79+5ll5wSgBeipOXaGnTHf04fiknMiE9wFiKeel140N6Mk1Nzfr5zNmzGg7t3P1nKXnhzKK9dEbovIU26EzpzXTVvjtKel36GCIBXFwWoX65Mt3OBHxxPYQb1Qc2GY5cVqzaX9/mUp1YwAqR8QIt1rLviB2cYUE4vRC9DFhytbYcHp5aX8DmYLcbi7DoNiWXJ/K9JlyvUdiXaq/OeUWy0SDwrh69WozR14WLlyoTz6QMW6vE37Hjh3mWXbSekTFXi4tx4EDB+reS1CZKj67QpN13D3kbBUtLFmyRCd5+5qKtNbjShob6D1KY8Pd20CDBtecMpEem1cvBddk6urqzFyK3WBasGCBWZqCazPYh8WLF+t13L1FKj+xTDToXqOCcVd47jtUksh9gRSVw4ABA/RzXMjHsIRdWWKYLRv39nBSo4IvB6iMUPG5/+7ueazjTqhImPaFfSRjd9JCJYdWtU0u2GeCChMVp0DFipiH4bXfpSS9DUyIi12OMHyF47Qv3ON1KT94r1c85RyW3oi7vGEbdhKS9bEPK1as0M8pBpw/WNmTC6U2ubhoT5S6QG7HxH0x2znZ0153X4R2x1rmZZKLtuXEqZDT9hH7bLNjYsfDfg/igsl9YdkdLyl37u3Yn4ltyPp4jphhH4V8lk22K38HyLTf5cpdVuxjFoiFvY473u7z2h0niqdY/DIAWoVoCcVgV4mIyCUWQ2cYPsAwAhERxU8sEg3Gf/18UY6IiMpP2SYauU0UEy7Alss3o4mIKBj+ejMREUUqlrc3E1E8rV+/Xi1fvtzMUVLEPtGw4IbH2IXH2IUzZ84cPVGyxH7orLGxUX/zet26dWYJ+cXYhcfYBYfkPHLkSP182bJlavLkyfo5Vb5Y92ikVcnWZXCMXXiMXTh2T4a9mmSJdaJhwQ2PsQuPsQvOnZSZpJMltomGBTc8xi48xi4cr4TMJJ0csb1GgzFy9wk+YsQIjpn7wNiFx9jlB9+LA36rIllifzMAC254jF14jF04jFsy8Xs0REQUKSYaIiKKFBMNERFFiomGiIgixURDRESRYqIhIqJIMdEQEVGkmGiIiChSTDRERBQpJhoiIooUEw0REUWKiYaIiCLFRENERJFioiGiosGvNvOXm5Mn9v8mgIiIyht7NEREFCndo7l3Q6OZJSIi263Dl5lnFBZ7NEREFKm0Hg0zNxFRCuvFwmGPhoiIIsVEQ0REkWKiISKiSDHREBFRpJhoiIgoUkw0REQUqaImmh//7dPq55tmmbno4bP4ZVQiotIq2PdoftN8t9py4Gkzl062i0TTtUMv9dGL5+l5geWHW3abOW/vG/g5NaznBDOXSiKvH3nFzJ3RtUMf9S9v/4F+LuvE9T54ian72IkoevweTeEUPNFk20amRJPN4zv+S72wZ2XGRJPt8zKtg2V4rW+Xoaqh/436uRfsJ9YplXwTjRw/ExVRcEw0hZPIazSPNv+7TiB2wkNPCAVKJkBFjUQXV5IoH9t2j1lCRFR8iUs0z7zxMz1M984+E80Sb5Js0JtCryCu0GODOCdMIoq3kicaVIDoomaaUNH79dK+x9Pe65UgXj34nO69+BlKqj9njH7cdmi9fgT3/iJx2dALyvQ6hg4zvQbubR9p2WdeOSPb59vL8Ih10aPB1OwcNxFRKZQ80UwYcHPakJV7ekfva/V6PTr114/ZIHnY7/W6voLkc16XwWYuuy61PfTjWy179SOumSDxjek7VW8fiejp1x/UCQ6QSOSaCF7Ho0DFj56U7BuOC+9FMgAkB2wb28TrGPZyJ8pcny9e3POYeZbS56yL9GfHuWdGRPFVdkNn0iMQR07uN8/yJxWtJJCgcGEevaHR5/+Tnj/3rHr9uNlZjsoelTkqf+kt4RHrSjKRpAlIsCA9DfS0YNR5H9GPSJLSoxLZPt92ae/36UQkn3F2h176cf/x1/QjEVExFTzRyPCNPWW6q8sPGT7y6p1ETZLcBWdf0pakkEzkuNCbEAdP/F0/ZktiUuHbsL3UY6rXlOk4c32+TRIREVE5KFii+UDdF9uGhdxTkNuZ3fBebCMTqXTtSXoQblKJ++kloWKXHoR9Pcd9dxomHLvItm0ZgrNhezZ7eMvrGk2uz/fi9blERMVS0qEzuWhtT1LRupdjwjUQIQnIa5IhIy9INm8e2WzmMpNe2KS6O/Qj3ocJPQr7Arysh14EkgCSk1wzwSPWlf2xb2yQZIhhLqjrPko/bty7Wj/ivXbSyfX52ew+ulU/+rkBgoio0Ir6HzbDfGEzH6iEUVnbx4VKGkNO8iVGvO5VWeP6iFdPQbYp3F+GxDHKcBjgwr0MZUmchf0ayBc0BfYB8/ZnZPv8TH9HLM90PETkrVj1YhIkLtEAluOaiPxUTSVDzwk9KcQcPSIi8oeJpnCKOnSGir1YSQZkeM0Ny+WieiWTJIOeE5MMEZVKUXs0RERxwXqxcMruezRERFRZmGiIiChSTDRERBQpJhoiIooUEw0REUWKiYaIiCLFRENERJEqyPdojp48qLYcxG98/V4dPL5LHTt1WJ08fcK8Gi81VR1Up9quqkenfmpYzytVffcx6qza7ubVwms5fUz/zMzL+9aovce3qeOn3lInTh01r8ZPbXVH1bmmq+rVeVAqfueMUR2qO5tXC49lLzzGLjt+j6Zw8ko0+KXiZ974qXpp7+POlqpVS4wrSC8dapwKsrVVDeuF/ytzXej/Y+PlVGuLeub1n6kNu3/txK7Kid0x80rlqK3ppE6fPqWG9/mAuuL8j6pONWebV/LHshceY+cPE03hhE402w9tUA83f8v5e57WlWYlQ2upykkGkwbdoeq6v8ssDW/PsW1q2eY7nd7M0di2IINALwcx/FD9nQX5KRyWvfAYO/+YaAon1DWal/atUQ9v/aZTSR6v+MIKOEYkhBWvLnB6IL8xS8N57fCL+kc9j548kIgkAzhODAkua/qKemX/WrM0HJa98Bg7KpXAiQYtose3/6c62ZqMStLW4pygf3r9f9J+yj8I9GR+vXWuvi6TRCgzq52yY/+bgyBY9sKXPcYufOwof4ESDcZ20e1OYmEVJ04dU6u2fU8dOL7LLPEHravUcFkyk4zA8f9qyzd0DycIlr3wZY+xCx87KoxAiQYXEDG2m3SnW0+qP76xxMz5gwv/uCZDTvzUSScePzVz/rDspYQqe4ydFiZ2VBi+Ew1uhcRdKkkY283llFNgmw88qw637DFLskMrHneXJeWaTC64w+6ve1boMuUHy94ZQcseY3dG0NhR4fhONLjfHrdCUkpra6tq2v+UmctOjw1XVZk5gprq2lSZ8oFlL12gssfYpQkSOyoc3yUQX+qqtPvt83Gy9bh6ed8TZi47fBmzEr8nkw98KXXj3tVmLjuWvXRByh5jly5I7KhwfCcafHOY0h1q2W2eZYdv/FN7+4/vNM+yY9lrz2/ZY+za8xs7KhzfiQY/T0Hpjp30F5Ogd1glhd+f2mHZa89v2WPs2vMbOyoc34mGF7Lba1X+7uSJ82+XRcnvBWqWvfb8lj3Grj2/saPC4VVCIiKKFBMNERFFiomGiIgixURDRESRYqIhIqJIMdEQEVGkmGiIiChSTDRERBQpJhoiIooUEw0REUWKiYaIiCLFRENERJFioiEiokgx0RARUaSYaIiIKFJMNEREFCkmGiIiihQTDRERRaqq1XHvhkY9c+vwZfrRi6xTCFf2/7S6rM/71V92P6J+/9oPzNKUD170JXVR99FmTqnXj7yiXtm3Ro3v/ymzJN2a136oLux2Wdt7Hmr6str51t/0c7C3V8hjENliJgr9uR+9eJ7q22Vou2OFjw9bqHp0usDMKR3jbh37pMXUhn2T9xw7eUj94MVPmFdS5LX9x3epn7w03SwtnGLHj2UvvOF9PqBjsfXgM+rhrd8yS1MkrgLl5Xfbv6/+cchdZkk6xB/kPYjlht2/0c/B3p5XOc9XkNj5WZeyK6seDQoXTkwUOvyRUcAABRDzmFAZYpJ5u3Bi+eXnfdjMpVzQ5W3mWeVDAkJSkNjIyYxKAfMST1Sgso5A7DrXdtOVieh39tvTklYlY9kLD2VGkrfEBpAcZB6JB2TeTvKI3dCe481cysBuI80zqgRllWjO6zJEP8oJjIL6802z9HM/9p/YlXZy4wRA5SmFvNL16JjqeQiczO5WeybHTh1ud8Kj4pTKtdKx7IWHXh1s2v8H/QhBer+IHXroQho4STlvk6CsEs3Rk/v1I4ZrwnjzSFNaqxyVJgrrsVPFqygbGxvV+vXrzVxxIVngBMWQTVCda7qqXUc2pp3wqDixrFhKGbtKKHtQihgeOrFbP36g7ov6MSjEDtCrBGng7Du+Xc9T/JVVosEQD05OVJYYF8VQUBBoUdmtclSa2w6t08+LZfny5WrkyJElOeGlFYkhIMRPTly/nn/zl/oR75MWuSwrhlLGrhLKHpQihug1YzgW5QWxC5qs9x/fqWMvw2XFbuBQ9MrurjNUljLGi5M16Akv3XCpZP0OHYVVVVWVNgn7hC8mxA7XGQDj5kGSDYaLUFnihJcWeaEvwrrlil0xE07cy55XDIsFw4yIHcoPknXQZIPeC96H2BW7gUPRqy52RegXCq1uYXYMdjEadwnB0B4N+v2l1tzcbJ4Vj1zABrn24BdakjjhEfdSD10gdqWIX6WUvVLAXYvo3aAMBSGJBbFDsoq6gUPFVV2KEzkTtCBxIVDgugGuOwSBShYFFa2iYgxdtLa2pk1ixIgRatmyZWrduuINn9itSImjXHvwS074YrUqc8Vu8uTJZmm0KqHsecWwGHBNUK5NQeeabjoOQSCxIDkjdhw2qzzVxawI3TC0gzFde0wc993LMgjz3Q0pqFEPXXgpZiVpx0oPOTiVo8zjNbQs3d93yEVOeEzFblUWM3aVWPagGDGUa4CYpHGD79fIMvRm3N/H8kOSM4fNKk9JvrBZSbLFTDB2mTF+4TF24QWJnZ91KbuyuxmAiIgqCxMNERFFiomGiIgixURDRESRYqIhIqJIMdEQEVGkmGiIiChSTDRERBQpJhoiIooUEw0REUWKiYaIiCLFRENERJFioiEiokgx0RARUaSYaIiIKFJMNEREFCkmGiIiihQTDRERRcp3oqmp6mCekajyGb7a6o7mGdn8limWvfb8lj3Grj2/saPC8R3xTrVdzTMSnWu7mWfZda5h7Lx0rDnLPMuOZa89v2WPsWvPb+yocHwnmh6d+plnJLp26G2eZder8yDzjGzdOp5vnmXHstee37LH2LXnN3ZUOL4TzbCeV6oONZ3NHCEWw3q+18xlh9jVVjN2tg5Ob+aSHg1mLjuWvXRByx5jd0aQ2FHh+E409d3HKNXaauao1YnFkB5jzVx29ec4sVOnUzOknTrd4j9+LHtpApU9xi5NkNhR4fhONGfVdlfDek3gxUVHlapRdd1H+e6Cd3B6M+/sPYk3BRi11Z3U25yy5Dd+LHtnBC17jN0ZQWNHheM70cDo869T1VU1Zi65qquq1di+HzNz/lxx/kd5shvVzgn/7r7Xmzl/WPZSwpQ9xi4lTOyoMAIlmi61PdQ1A29zWuidzJLkQe/k6gs/q87pdIFZ4k+nmrPVh+rvVLVVye7VoOx88KIv6bIUBMte+LLH2IWPHRVGoEQDuN4wxmmNdkzgBUYU1jF9p6pLel5plgTTt8tQdfXAWxObbFDRXXXhLap/13eYJcGw7IUve4xd+NhR/gInGhh57ofU+y5EC6mzqqmqNUsrF8Z2Mew1YcDN+tjzMbTHOPXhwV9zejhdE3M3EK7JdKzuov7hotn6+PPBshceY0elUtXquHdDo565dfgy/ejXgeO71B/fWKKaDzyr7+Y42XrcvFIZkAhwXLiAiLHdQna7j596S/3p9SXqxT2PqWrnpG85fdS8UjlwCzPuLsOFf1yTCTpclg3LXniMnT9h60VqL69EIw637FFN+59SL+97Qh1q2a2OnTysWmN6Oy9+ngLfHMadKbjfHrdCRnmXypGT+9XWg39WG/euVvuP71QnTh1Vp1pbzKvxgxYkvvGPL2PiezJRx49lL7xix+61lw+oPTuPqMsmFP46SRSxY6IpnIIkmlJav369am5uVpMnTzZLyC/GLhzGLZzGxkYdt3Xr1pkl5Y2JpnBCXaMpJ3PmzNETBcfYhcO4BYfkvHz58rZHSpZYJxoW3vAYu3AYt3DsxMwknTyxTjQsvOExduEwbsG5kzKTdPLENtGw8IbH2IXDuIXjlZCZpJMl7WaAOPnh7U+rvzy+y8yl9L/kHDXrpxPMHGXC2IXDuOXncyNSSfme9fG6iYI3A+QvtolGxLXwlgPGLhzGLRwmmuTSicY8j6Wqqir9GPPDKAnGLhzGLRzGLblif3szERGVNyYaIiKKFBMNERFFiomGiIgixURDRESRCnzX2frXDqsvPbxV/Xn7QbXnrZNmaTL07FKrrriwu5ozaZB696DuZmkwjF/4+DF2jF0Y+cSuee8xVdcref8srtACJZqFf9ipZv5qi3rrxCmzJJm6dapRt105QH19Up1Z4g/jlxImfoxdCmMXXtDY7T96Uo1c8JxaN2OU6nFW5f+juCj5HjpDi4iFNeXQ8VPqu7/foZ5o2m+W5Mb4nRE0fozdGYxdeEFj9z1nXfRo8Ej58Z1o0O1mYT3jsFNov77qVTOXG+OXLkj8GLt0jF14fmOH3sx/rHlNP8cj5ik834kGY7uUDq1Fvxi/9vzGj7Frj7ELz0/s0IuR5IJH9mry4zvRJO0Coh/7jviPCePXnt/4MXbtMXbh5Yqd3ZsR7NXkh7c3ExFZcF3m8+P7q69eO0jP4xHzWE7hMNEQEVlG9O+qvjaxTk8gz7GcwmGiISIqEvkF6zAyvVeWL126VM2fP18/z8fEiRPV2rVrzVxhlEWi+ZzTLX3l30abuWB+eeM79ORmL989d6x+rFSZYuAH4o74u8nyhvpzQv9t4oCxCy+f2GV6r7280s/bQpsyZYqaOXOmmSsviejR9Jn9lHlGQT255YAa+s1nzBwFwdjlh+dt5eDQGRFRQBim6tWrl57wHDDkhAlDWXi877779OtDhgxRW7Zs0evAl7/8Zb2Ovdxre3v27Enbnsi0HMNmMnSG16ZPn95uHfuzMeUaIsO6sj72FduX/QPZtr3dDRs26GV4Db0sLC9oohnUs7MeKmj97pX6EfMg83j87bTL9CTLxFkdqtPeK2RddKOvG3muXnbvR4aot+Y16OnyAWcu0GVajvcDuuQb7nhX2zryUxTYLrYvn21/fjHZ+4HjBgzBYBkm7PNP/nlYW5zsoQf8HpO8V5Zn+nvYMe3V5cxPa3gtx+dLPPD4x9tGtq2DoSGw4779q+9O269iYezC8zrWKGJXKectKvpvf/vbatOmTXpavHixXgaNjY36P4j26NFDPf744/r1O+64Q/3iF7/Qr8OkSZP0OnPnzlULFizIuL3vfOc76qqrrtLrzp4927xbZVzudv3117ftCxLKI488ojZv3qx2796tVq1aZdbKDu/HhGN47LHH1I033qj3D7A97Ace9+3bp9fDdmfNmqVfhwkTJujlBU00D0wZqla+tFdV3f57tXTdm3pefOXRraru60/rOzfeOnFar3O05bQu0IBEc+PSl/XyPUdadOFDgTq7U41e1vijF9U3Jl2kC+3Uy89TE+/7i3r73X9We8098ZmWe8Hrn1z6kpoy8jw9v+BDg/W3hfE5G984opeVAo7vsw9t0vsB9m8yjfr283qfP/au89X3176mY/mei9J/IBDrYDlO1ganIvP6e2Cb53btqNdDTEWm5W5vHmrR23t04141Y8KFgeIeJcYuPK9jFYWKXSWdtxs3blTPPvus6tOnj55Wrlypl8Gll16qHy+//HI99e7du22ZGDdunH5Eax+9hEzbe+6559quuch7INNyN3kN+wEvvPCCuuWWW/Q+1dfX695HNkh26BVdccUV6uabb1YHDhzQ7x01apROXPfcc49OPNjuokWLdM9l8ODBev+FHHtBE81Ap9Dc0tBftzC+cs0gPS9+uu7v6tV9x3RB+t2mfXqZfV86lmNMGx58/k393uH9uupfW8X21tw6Ql187lnqw+/srf6w9aBeF9uTbWRa7obleB37I5Dw7jFf0JJ9KwUc39JPvF0f79VDe+rjB8TG3mfsq8RSyHFhen7HYTXSOem9/h7Y5gNP79LrIVayjUzL3SQ+f95+SD/6jXvUGLvwvI4VChm7Sjpv+/Xrp6677jrdM0BrHVO2Ct9NhqswDIVKO9P2kAxkmMoe4sq0PJeBAweqJUuW6OdIcE1NTfp5Jg888ICqq6tTK1asUNOmTTNLlU4ut99+u94PJB5sd968eW37jsmtoIkGPZHPL2vSLQxMQS6EYrihwWkNoYVz45gL1Iadh9U2p2D96oU9bdvDtM4pzGg5YT1M8hPemZb7gd6UDMtdfXFP/VgKm/5+VI2/d33bsX74gRfMK7nheHHciCFanL/86x7PvwdiihYkYF0Z5sm0PJd84l5IjF14+Zy3fmNXSectKtjJkyeriy++uO0aRhAYMsN7Vq9erb7whS9k3N6MGTP00BjmH330Ub0MMi3PRXpQeB96Kj179tRJLpOxY8equ+++W+/X8OHDzdLU8SPB3HTTTXoe20VvR/bdviYkCppopvz3RnXLuFRLBlPQ8eZln7xUNd85Rv398Al156PN6tb/a1Jnd6xu2x7GYNHyQcsJ6z33r6kuIWRa7sePn3ld/WjKMD3+e163DvpEKQUMLyIGcrwyrOgHKre/ffEKteKmy3SPEK0/r78HYtq7Swc9/8CUS3SrEDItz8Ur7qh4i42xCy+f89Zv7CrtvEXlunfv3rQWPFr+0rPB0JY9vCXPsS7Wk0dU2OC1PVTo6HVg/q677sq53P5MWQZYJvsln71w4UJ9XQXbcpPjwIR9woSkItuWi/128pH9wIT3g2wHfP8/GrRMkgAXdXcdPKE+/r8vmSXZ4UTyIwnxQ4v0MSd+qPTtIY5s/MSPsfPG2J0R5Xnrd91yh2G2hoYG/RzXUnCN5f3vf7+e9wvDfffff7966KGH0hJNLkw0DvSUMMYPf3r1oG6RoWXmBxPNmRgcOXFa/faVfYGGrZJeWTJ24RXrvPW7LmXGRJOnIAWW2kt6ZZkPxi48JpriKug1GiIiIjcmGiIiy0tvHlFfW9GsJ5DnGJ6jcJhoiIgsuMX6vqd2qTkrU//yGY/zfre9ZLefVwLfiaanz+8GJEnXjjXmWW6MX3t+48fYtcfYhZcrdp1rq9Wsqy80cyk3v6ef6tuto5mjoHwnmisuTP/ZCVJq1MBu5llujF97fuPH2LXH2IXnJ3Z2YvFKPBSM70QzZ9Ig1a2T/xZ8peveuVbNeO8AM5cb45cuSPwYu3SMXXh+Y2cnF/Zm8uc70eA3x267coDqykKrC+unxvRVH7w09a1ePxi/M4LGj7E7g7ELL2jskGBwXYa9mfz5/h6NeKJpv/7F1Oe3H1YHjnn/eGClwtguut1oEQVJMjbGL3z8GDvGLox8Yocf8+RNAPkLnGiIiIiC4O3NREQUKSYaIiKKFBMNERFFiomGiIgiFftEs379erV8+XIzR0EwduExduEwbskU+7vOGhsbVXNzs1q3bp1ZQn4xduExduEwbskU6x6NtI7YSgqOsQuPsQuHcUuuWCeaOXPmmGfpzyk3xi48xi4cxi25Ypto3K0itpL8Y+zCY+zCYdySLbaJxqtFxFaSP4xdeIxdOIxbssX+ZoCqqir9yF/SCY6xC4+xC4dxS6bY395MRETljYmGiIgixURDRESRYqIhIqJIMdEQEVGkmGiIiChSTDRERBQpJhoiIooUEw0REUWKiYaIiCLFRENERJFioiEiokgx0RARUaSYaIiIKFKx/zcBRERU3tijISKiSDHREBFRpJhoiIgoUkw0REQUKSYaIiKKFBMNERXV0qVLVVVVlZmjJGCiSZC1a9fqExyPVBiI5/z5881c/E2fPl1NnDjRzBUH4sfEU9mYaIhIQwNk1apVasWKFWYJUWEw0STIuHHjFL6fi0cqDMRz5syZZi7ebrjhBp1o3DDUNWTIEDNXeIhfFN8bj3q/yT8mGiJSW7Zs0Ummvr7eLCEqnJInGrQ63GPCci1BJpwENhnTlcmr1YJtyusYd7ZhvpLG1f1CHBEPdzwRP4kVpiSxy5JdjjJdsMY6dtnBPNYV8j6Z4gDlYfDgwXrCPtvnI45v6tSpavPmzWnHhBjY8RJ43Y4H2Oei13uwvnu5+xx3n69YhnrC3raf/abSKFmikRMShcEeE0ahb2hoUE8++aTuTmNasGCBeTWVJO6///621zChUNkFVRKLvD5+/Pi0C+ALFy5Uq1ev1p/vLsBJg7h95jOfaYvVvHnzEnNSogzOmjWr7djRos+nPKCMSeUm23Q3csqNJBn7fAOptJuamtSDDz6o17Ff90vOS3kvYowYZSPnqrwH+4a/k30OA4b6Zs+erddBzFeuXNn298t3v6mwip5opKWyePFizwKwc+dO/WhfR0BiABS0RYsWtRtHRqJCQZOWFE6eq666Sj+HKVOmtLsugffgs5ubm/X+lHuFEAX8LVAR2NcY8Bwnp/ukrkTbtm1T1157rZlTetgon+stO3bs0LGzh5+k7JYrlHs0LtznGyrtfOF8xHlpNyQRGySAbLAv9t8B84gr4mubO3du235ju9OmTdMNSCo/RUs0qLhQoaNiRwWf6c4WFByc/F6Vv9eJLPAeVByAVg5aQNiGe5jIDScV9ge9HqwvySoJ8LdAhYLjtidUDu6TuhKhMpPjL8TfHQ0awPbi0lNGy1/OFZlwjkGucycXdyIPAp9t7xPKpJzfYsCAAeZZSl1dnXlG5aZoiQYJRHovKDj2eKqb9DbQe8G6QU9a+Sy01HDSYBuZuHtYUlkkBVqBOG73lJQ44FgxNIPhHJSDfHtyqLhRKUrlHYeGC3oY7r8/Jq8GXTGggYnz1t4XSX4UT0UfOpMeBIa2cCLa11bcsJ6MzwJaMDiJvVpaaJkOHDjQzKWgxYptoFXlTlZyETFXD6uSoQXodTtr0tgNEwzHgLu1LFD+ckEFje2hAkfvupzh/FuzZo2Z8899roH7vMQ6SLxu7p6JGxqYOO9tfuJO5atkNwNIEsCJbfdu0KK0W4EyXAYyrHbNNdfoeYEWEJZLK9w95IbCbp8YeB2JDp9f7mPoUcLfACewO17ZepuVBI0Pu3JEo0Na8ShrYDdQ3HFyQ7m1e0SoULM1pMoBEiEqdvucQ0zsY5UGnm306NF6mf0+d3xwPrrLF+IjDcdMcL4/9dRTZi533DPx2m8qEaeyLTtO0sAYW9vkNm3atLTXMW9zWkNprzstVfNKsjknnY4HHm12rLxer2T2caPc2dzlyOmhtDqVYFp5wjyWC8zL+ngeB+7j9Npv+5wUOG77fbIdOx5gr4PtyPuExFVIOZVJXrfjjuX4PBted/8Nvfabio//yjlB0FJ1TljdyivV+DsRJY1S/w8meI0fgs0CPAAAAABJRU5ErkJggg==)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qqw7uAtF09D"
      },
      "source": [
        "* decoder는 encoder와 마찬가지로 여러 개의 RNN cell로 이루어짐\n",
        "* decoder의 처음 입력은 시작을 나타내는 토큰을 입력하며, 이 토큰 다음에 등장할 단어를 예측\n",
        "* 처음 셀에서 단어를 예측한 결과는 다시 다음 시점의 decoder 입력으로 사용\n",
        "* 위 과정을 반복해, 끝을 나타내는 토큰이 예측될 때까지 반복\n",
        "* 요약하자면 decoder는 encoder에서 넘겨받은 컨텍스트 벡터를 활용해 시퀀스를 만들어 냄\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8unfQ0tBBPWZ"
      },
      "source": [
        "### 교사 강요(Teacher Forcing)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jTuXNNhmBXml"
      },
      "source": [
        "* 앞서 설명한 seq2seq 모델을 잘 살펴보면 디코더의 입력이 필요하지 않음을 알 수 있음\n",
        "* 예측이 잘못됐을 경우, 잘못된 예측이 다음 시점으로 입력돼 연쇄적으로 잘못된 예측을 함\n",
        "* 이를 해결하기 위해 디코더의 다음 시점의 입력으로 이전 시점의 출력이 아닌, 정답을 주어 이를 방지함\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEjNhInwBYJQ"
      },
      "source": [
        "## 디코더 모델 구성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FH1CNOj99iG5"
      },
      "source": [
        "\n",
        "\n",
        "*   모델의 구성은 encoder과 거의 유사함\n",
        "*   lstm의 return_sequences는 출력을 시퀀스로 반환할 때 사용\n",
        "*   decoder_lstm을 사용할 때는 initial_state를 인코더의 은닉 상태로 설정\n",
        "*   마지막으로 Dense layer와 softmax를 통과해 예측 글자에 해당하는 인덱스를 반환\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xuv70KtuEbu6"
      },
      "source": [
        "decoder_inputs = Input(shape=(None, tar_vocab_size))\n",
        "decoder_lstm = LSTM(units=256, return_sequences=True, return_state=True)\n",
        "\n",
        "# 디코더에게 인코더의 은닉 상태, 셀 상태를 전달.\n",
        "decoder_outputs, _, _= decoder_lstm(decoder_inputs, initial_state=encoder_states)\n",
        "\n",
        "decoder_softmax_layer = Dense(tar_vocab_size, activation='softmax')\n",
        "decoder_outputs = decoder_softmax_layer(decoder_outputs)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QleDp9A7PjHk"
      },
      "source": [
        "## Seq2Seq 모델"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b7x6_ifRPoec"
      },
      "source": [
        "\n",
        "\n",
        "*   앞서 구성한 encoder와 decoder를 결합해 seq2seq model을 구성\n",
        "*   구성한 모델과 준비한 데이터를 사용해 기계 번역 학습\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Pty9yEAPlmb"
      },
      "source": [
        "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
        "model.compile(optimizer=\"rmsprop\", loss=\"categorical_crossentropy\")"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QwWT2xjrZmoj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4f49faeb-8885-4c30-a8ab-374c5916f48f"
      },
      "source": [
        "model.fit(x=[encoder_input, decoder_input], y=decoder_target, batch_size=64, epochs=20, validation_split=0.2)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/20\n",
            "750/750 [==============================] - 22s 16ms/step - loss: 0.8508 - val_loss: 0.7710\n",
            "Epoch 2/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.5741 - val_loss: 0.6671\n",
            "Epoch 3/20\n",
            "750/750 [==============================] - 11s 14ms/step - loss: 0.5048 - val_loss: 0.6005\n",
            "Epoch 4/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.4592 - val_loss: 0.5630\n",
            "Epoch 5/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.4243 - val_loss: 0.5245\n",
            "Epoch 6/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3972 - val_loss: 0.4997\n",
            "Epoch 7/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3764 - val_loss: 0.4750\n",
            "Epoch 8/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3597 - val_loss: 0.4561\n",
            "Epoch 9/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3459 - val_loss: 0.4463\n",
            "Epoch 10/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3342 - val_loss: 0.4325\n",
            "Epoch 11/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3241 - val_loss: 0.4208\n",
            "Epoch 12/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3149 - val_loss: 0.4135\n",
            "Epoch 13/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.3069 - val_loss: 0.4065\n",
            "Epoch 14/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2995 - val_loss: 0.4009\n",
            "Epoch 15/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2927 - val_loss: 0.3949\n",
            "Epoch 16/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2866 - val_loss: 0.3859\n",
            "Epoch 17/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2808 - val_loss: 0.3822\n",
            "Epoch 18/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2755 - val_loss: 0.3775\n",
            "Epoch 19/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2706 - val_loss: 0.3747\n",
            "Epoch 20/20\n",
            "750/750 [==============================] - 10s 13ms/step - loss: 0.2660 - val_loss: 0.3704\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7e9605736230>"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyDzPgiMaf5_"
      },
      "source": [
        "## 예측"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GZbbHaDm_aRX"
      },
      "source": [
        "\n",
        "\n",
        "*   일반 모델과는 다르게, seq2seq 모델은 모델 예측 프로세스가 다름\n",
        "*   예측할 때는 인덱스 하나씩을 예측하게 되며, 예측한 인덱스를 저장하고 이를 다시 입력으로 사용해 종료 토큰이 나올때까지 반복\n",
        "*   마지막으로 예측한 인덱스들을 사전을 통해 글자들로 변환해 최종 예측을 얻음\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4i6nyNCbamz_"
      },
      "source": [
        "encoder_model = Model(inputs=encoder_inputs, outputs=encoder_states)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YvQNB_ljairY"
      },
      "source": [
        "# 이전 시점의 상태들을 저장하는 텐서\n",
        "decoder_state_input_h = Input(shape=(256,))\n",
        "decoder_state_input_c = Input(shape=(256,))\n",
        "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
        "\n",
        "# 문장의 다음 단어를 예측하기 위해서 초기 상태(initial_state)를 이전 시점의 상태로 사용.\n",
        "# 뒤의 함수 decode_sequence()에 동작을 구현 예정\n",
        "decoder_outputs, state_h, state_c = decoder_lstm(decoder_inputs, initial_state=decoder_states_inputs)\n",
        "\n",
        "# 훈련 과정에서와 달리 LSTM의 리턴하는 은닉 상태와 셀 상태를 버리지 않음.\n",
        "decoder_states = [state_h, state_c]\n",
        "decoder_outputs = decoder_softmax_layer(decoder_outputs)\n",
        "decoder_model = Model(inputs=[decoder_inputs] + decoder_states_inputs, outputs=[decoder_outputs] + decoder_states)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "asYLEiiRap-Y"
      },
      "source": [
        "index_to_src = dict((i, char) for char, i in src_to_index.items())\n",
        "index_to_tar = dict((i, char) for char, i in tar_to_index.items())"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nfpLBKbzajev"
      },
      "source": [
        "def decode_sequence(input_seq):\n",
        "  # 입력으로부터 인코더의 상태를 얻음\n",
        "  states_value = encoder_model.predict(input_seq)\n",
        "\n",
        "  # <SOS>에 해당하는 원-핫 벡터 생성\n",
        "  target_seq = np.zeros((1, 1, tar_vocab_size))\n",
        "  target_seq[0, 0, tar_to_index['\\t']] = 1.\n",
        "\n",
        "  stop_condition = False\n",
        "  decoded_sentence = \"\"\n",
        "\n",
        "  # stop_condition이 True가 될 때까지 루프 반복\n",
        "  while not stop_condition:\n",
        "    # 이점 시점의 상태 states_value를 현 시점의 초기 상태로 사용\n",
        "    output_tokens, h, c = decoder_model.predict([target_seq] + states_value)\n",
        "\n",
        "    # 예측 결과를 문자로 변환\n",
        "    sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
        "    sampled_char = index_to_tar[sampled_token_index]\n",
        "\n",
        "    # 현재 시점의 예측 문자를 예측 문장에 추가\n",
        "    decoded_sentence += sampled_char\n",
        "\n",
        "    # <eos>에 도달하거나 최대 길이를 넘으면 중단.\n",
        "    if (sampled_char == '\\n' or\n",
        "        len(decoded_sentence) > max_tar_len):\n",
        "        stop_condition = True\n",
        "\n",
        "    # 현재 시점의 예측 결과를 다음 시점의 입력으로 사용하기 위해 저장\n",
        "    target_seq = np.zeros((1, 1, tar_vocab_size))\n",
        "    target_seq[0, 0, sampled_token_index] = 1.\n",
        "\n",
        "    # 현재 시점의 상태를 다음 시점의 상태로 사용하기 위해 저장\n",
        "    states_value = [h, c]\n",
        "\n",
        "  return decoded_sentence"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IEwOVfYZaokX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ba94d810-4be3-4c61-9291-64f7fb281c80"
      },
      "source": [
        "for seq_index in [3,50,100,300,1001]: # 입력 문장의 인덱스\n",
        "  input_seq = encoder_input[seq_index:seq_index+1]\n",
        "  decoded_sentence = decode_sequence(input_seq)\n",
        "  print(35 * \"-\")\n",
        "  print('입력 문장:', lines.src[seq_index])\n",
        "  print('정답 문장:', lines.tar[seq_index][2:len(lines.tar[seq_index])-1]) # '\\t'와 '\\n'을 빼고 출력\n",
        "  print('번역 문장:', decoded_sentence[1:len(decoded_sentence)-1]) # '\\n'을 빼고 출력"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 0s 343ms/step\n",
            "1/1 [==============================] - 0s 350ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "1/1 [==============================] - 0s 25ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "1/1 [==============================] - 0s 34ms/step\n",
            "1/1 [==============================] - 0s 32ms/step\n",
            "1/1 [==============================] - 0s 32ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 38ms/step\n",
            "-----------------------------------\n",
            "입력 문장: Go.\n",
            "정답 문장: Bouge ! \n",
            "번역 문장: Continuez. \n",
            "1/1 [==============================] - 0s 33ms/step\n",
            "1/1 [==============================] - 0s 30ms/step\n",
            "1/1 [==============================] - 0s 28ms/step\n",
            "1/1 [==============================] - 0s 30ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 28ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 31ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 28ms/step\n",
            "1/1 [==============================] - 0s 29ms/step\n",
            "1/1 [==============================] - 0s 32ms/step\n",
            "1/1 [==============================] - 0s 37ms/step\n",
            "1/1 [==============================] - 0s 32ms/step\n",
            "1/1 [==============================] - 0s 36ms/step\n",
            "1/1 [==============================] - 0s 32ms/step\n",
            "1/1 [==============================] - 0s 31ms/step\n",
            "1/1 [==============================] - 0s 32ms/step\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 24ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "-----------------------------------\n",
            "입력 문장: Hello!\n",
            "정답 문장: Bonjour ! \n",
            "번역 문장: Son sur le coup de fait ! \n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 28ms/step\n",
            "1/1 [==============================] - 0s 18ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 24ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "-----------------------------------\n",
            "입력 문장: Got it!\n",
            "정답 문장: J'ai pigé ! \n",
            "번역 문장: Conne-toi. \n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 25ms/step\n",
            "1/1 [==============================] - 0s 24ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 26ms/step\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 25ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "-----------------------------------\n",
            "입력 문장: Go home.\n",
            "정답 문장: Rentre à la maison. \n",
            "번역 문장: Parsonne ! \n",
            "1/1 [==============================] - 0s 17ms/step\n",
            "1/1 [==============================] - 0s 22ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 19ms/step\n",
            "1/1 [==============================] - 0s 20ms/step\n",
            "1/1 [==============================] - 0s 23ms/step\n",
            "1/1 [==============================] - 0s 21ms/step\n",
            "-----------------------------------\n",
            "입력 문장: Get going.\n",
            "정답 문장: En avant. \n",
            "번역 문장: Décume-toi. \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g73IOlKD3ppK"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}