2. Seq2Seq - keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Seq2Seq\n",
    "\n",
    "- https://medium.com/@dev.elect.iitd/neural-machine-translation-using-word-level-seq2seq-model-47538cba8cd7\n",
    "- https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py\n",
    "- https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/\n",
    "- https://machinelearningmastery.com/develop-encoder-decoder-model-sequence-sequence-prediction-keras/\n",
    "- https://blog.keras.io/a-ten-minute-introduction-to-sequence-to-sequence-learning-in-keras.html\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "- **`encoder`: it processes the input sequence and returns its own internal state.** \n",
    "\n",
    "    - encoder RNN의 아웃풋은 버린다! only recovering the state.\n",
    "    - This state will serve as the \"context\", or \"conditioning\"      \n",
    "    \n",
    "    \n",
    "- **`decoder`: it is trained to predict the next characters of the target sequence, given previous characters of the target sequence.** \n",
    "    - it is trained to **turn the target sequences into the same sequences** \n",
    "    - but offset by one timestep in the future, \n",
    "    - a training process called \"teacher forcing\" in this context. \n",
    "    \n",
    "    - `Importantly, the encoder uses as initial state the state vectors from the encoder, which is how the decoder obtains information about what it is supposed to generate.` \n",
    "    - Effectively, the decoder learns to generate targets[t+1...] given targets[...t], conditioned on the input sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:21.206430Z",
     "start_time": "2020-09-16T09:48:20.495357Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import string\n",
    "from itertools import chain\n",
    "import pickle\n",
    "import re\n",
    "\n",
    "from string import digits\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:21.302173Z",
     "start_time": "2020-09-16T09:48:21.240339Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before:  (145116, 4)\n",
      "after:  (30000, 2)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Desc</th>\n",
       "      <th>Story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>and its magnificent trunk , larger than life i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a old curvy tree in the sun light .</td>\n",
       "      <td>and its magnificent trunk , larger than life i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a person is taking a picture of a large tree a...</td>\n",
       "      <td>and its magnificent trunk , larger than life i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>large tree with many outstretching branches an...</td>\n",
       "      <td>we found this tree when we were walking in a n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a green sign is describing a historic tree and...</td>\n",
       "      <td>it turns out it is a popular attraction here .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>a large tree with roots that look like crocodi...</td>\n",
       "      <td>the tree is very unusual , with its roots expo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>the trunk was really wide , as much as 12 feet !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>huge brown tree roots rose above the ground .</td>\n",
       "      <td>you can see how big these roots are - pretty a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>a large tree with many branches coming out</td>\n",
       "      <td>we found this tree when we were walking in a n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>a plaque describes an historical tree and advi...</td>\n",
       "      <td>it turns out it is a popular attraction here .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>a tree has spread its roots high and wide acro...</td>\n",
       "      <td>the tree is very unusual , with its roots expo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>a old curvy tree in the sun light .</td>\n",
       "      <td>the trunk was really wide , as much as 12 feet !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>some trees that are surrounded by apartment bu...</td>\n",
       "      <td>you can see how big these roots are - pretty a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>the tree has very long and dated branches .</td>\n",
       "      <td>we found this tree when we were walking in a n...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>a plaque on a stand surround by died leaves of...</td>\n",
       "      <td>it turns out it is a popular attraction here .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 Desc  \\\n",
       "0      big old tree being photographed on a sunny day   \n",
       "1                 a old curvy tree in the sun light .   \n",
       "2   a person is taking a picture of a large tree a...   \n",
       "3   large tree with many outstretching branches an...   \n",
       "4   a green sign is describing a historic tree and...   \n",
       "5   a large tree with roots that look like crocodi...   \n",
       "6      big old tree being photographed on a sunny day   \n",
       "7       huge brown tree roots rose above the ground .   \n",
       "8          a large tree with many branches coming out   \n",
       "9   a plaque describes an historical tree and advi...   \n",
       "10  a tree has spread its roots high and wide acro...   \n",
       "11                a old curvy tree in the sun light .   \n",
       "12  some trees that are surrounded by apartment bu...   \n",
       "13        the tree has very long and dated branches .   \n",
       "14  a plaque on a stand surround by died leaves of...   \n",
       "\n",
       "                                                Story  \n",
       "0   and its magnificent trunk , larger than life i...  \n",
       "1   and its magnificent trunk , larger than life i...  \n",
       "2   and its magnificent trunk , larger than life i...  \n",
       "3   we found this tree when we were walking in a n...  \n",
       "4      it turns out it is a popular attraction here .  \n",
       "5   the tree is very unusual , with its roots expo...  \n",
       "6    the trunk was really wide , as much as 12 feet !  \n",
       "7   you can see how big these roots are - pretty a...  \n",
       "8   we found this tree when we were walking in a n...  \n",
       "9      it turns out it is a popular attraction here .  \n",
       "10  the tree is very unusual , with its roots expo...  \n",
       "11   the trunk was really wide , as much as 12 feet !  \n",
       "12  you can see how big these roots are - pretty a...  \n",
       "13  we found this tree when we were walking in a n...  \n",
       "14     it turns out it is a popular attraction here .  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load\n",
    "with open('df_story_desc_final.pickle', 'rb') as f:\n",
    "    df_story_and_desc = pickle.load(f)\n",
    "\n",
    "print (\"before: \", df_story_and_desc.shape)\n",
    "df_story_and_desc = df_story_and_desc[:30000]\n",
    "\n",
    "df_story_and_desc_id = df_story_and_desc['Story_Photo_id']\n",
    "df_story_and_desc_text = df_story_and_desc[['Desc', 'Story']]\n",
    "\n",
    "print (\"after: \", df_story_and_desc_text.shape)\n",
    "df_story_and_desc_text.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "==> 15개 단위로 같은 사진 set에 대한 내용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:21.350046Z",
     "start_time": "2020-09-16T09:48:21.335085Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Desc</th>\n",
       "      <th>Story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>and its magnificent trunk , larger than life i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>the trunk was really wide , as much as 12 feet !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>some more different parts of the tree .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>the trunk was incredibly thick and rigid .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>i was dwarfed by the tree 's size .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Desc  \\\n",
       "0   big old tree being photographed on a sunny day   \n",
       "6   big old tree being photographed on a sunny day   \n",
       "18  big old tree being photographed on a sunny day   \n",
       "24  big old tree being photographed on a sunny day   \n",
       "39  big old tree being photographed on a sunny day   \n",
       "\n",
       "                                                Story  \n",
       "0   and its magnificent trunk , larger than life i...  \n",
       "6    the trunk was really wide , as much as 12 feet !  \n",
       "18            some more different parts of the tree .  \n",
       "24         the trunk was incredibly thick and rigid .  \n",
       "39                i was dwarfed by the tree 's size .  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_story_and_desc_text[df_story_and_desc_text.Desc=='big old tree being photographed on a sunny day']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:21.413875Z",
     "start_time": "2020-09-16T09:48:21.405898Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pair 0 : big old tree being photographed on a sunny day ==> and its magnificent trunk , larger than life itself .\n",
      "pair 6 : big old tree being photographed on a sunny day ==> the trunk was really wide , as much as 12 feet !\n",
      "pair 18 : big old tree being photographed on a sunny day ==> some more different parts of the tree .\n",
      "pair 24 : big old tree being photographed on a sunny day ==> the trunk was incredibly thick and rigid .\n",
      "pair 39 : big old tree being photographed on a sunny day ==> i was dwarfed by the tree 's size .\n"
     ]
    }
   ],
   "source": [
    "for i,row in df_story_and_desc_text[df_story_and_desc_text.Desc=='big old tree being photographed on a sunny day'].iterrows():\n",
    "    print(\"pair\",i,\":\",row['Desc']+\" ==> \"+row['Story'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "==> 하나의 Description Text에 5개의 Story가 대응됨"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:21.968231Z",
     "start_time": "2020-09-16T09:48:21.954237Z"
    }
   },
   "outputs": [],
   "source": [
    "def re_sub(item):\n",
    "    re_sentence = []\n",
    "    for sentence in item:\n",
    "        sentence = re.sub('[^a-zA-Z]+', ' ', sentence)\n",
    "        re_sentence.append(sentence)\n",
    "    return re_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:22.538078Z",
     "start_time": "2020-09-16T09:48:22.139819Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>in_desc</th>\n",
       "      <th>out_story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>&lt;sos&gt; and its magnificent trunk larger than li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a old curvy tree in the sun light</td>\n",
       "      <td>&lt;sos&gt; and its magnificent trunk larger than li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a person is taking a picture of a large tree a...</td>\n",
       "      <td>&lt;sos&gt; and its magnificent trunk larger than li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>large tree with many outstretching branches an...</td>\n",
       "      <td>&lt;sos&gt; we found this tree when we were walking ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a green sign is describing a historic tree and...</td>\n",
       "      <td>&lt;sos&gt; it turns out it is a popular attraction ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29995</th>\n",
       "      <td>a crowd of people in a village square three of...</td>\n",
       "      <td>&lt;sos&gt; we went to organization last summer for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29996</th>\n",
       "      <td>people on a safari truck watching as they expl...</td>\n",
       "      <td>&lt;sos&gt; we got to ride so many different rides  ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29997</th>\n",
       "      <td>children pose for a photograph on steps in a p...</td>\n",
       "      <td>&lt;sos&gt; the kids were having so much fun  &lt;eos&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29998</th>\n",
       "      <td>kids wearing pirate hats are brandishing toy s...</td>\n",
       "      <td>&lt;sos&gt; they loved going around to all the diffe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29999</th>\n",
       "      <td>bronze statue outdoors in foreground large cas...</td>\n",
       "      <td>&lt;sos&gt; we had such a good time and ca n t wait ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 in_desc  \\\n",
       "0         big old tree being photographed on a sunny day   \n",
       "1                     a old curvy tree in the sun light    \n",
       "2      a person is taking a picture of a large tree a...   \n",
       "3      large tree with many outstretching branches an...   \n",
       "4      a green sign is describing a historic tree and...   \n",
       "...                                                  ...   \n",
       "29995  a crowd of people in a village square three of...   \n",
       "29996  people on a safari truck watching as they expl...   \n",
       "29997  children pose for a photograph on steps in a p...   \n",
       "29998  kids wearing pirate hats are brandishing toy s...   \n",
       "29999  bronze statue outdoors in foreground large cas...   \n",
       "\n",
       "                                               out_story  \n",
       "0      <sos> and its magnificent trunk larger than li...  \n",
       "1      <sos> and its magnificent trunk larger than li...  \n",
       "2      <sos> and its magnificent trunk larger than li...  \n",
       "3      <sos> we found this tree when we were walking ...  \n",
       "4      <sos> it turns out it is a popular attraction ...  \n",
       "...                                                  ...  \n",
       "29995  <sos> we went to organization last summer for ...  \n",
       "29996  <sos> we got to ride so many different rides  ...  \n",
       "29997      <sos> the kids were having so much fun  <eos>  \n",
       "29998  <sos> they loved going around to all the diffe...  \n",
       "29999  <sos> we had such a good time and ca n t wait ...  \n",
       "\n",
       "[30000 rows x 2 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_data = df_story_and_desc_text.apply(lambda x: re_sub(x))\n",
    "clean_data.columns = ['in_desc','out_story']\n",
    "clean_data['out_story'] = clean_data['out_story'].apply(lambda x : '<sos> '+ x + ' <eos>')\n",
    "clean_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>in_desc</th>\n",
       "      <th>out_story</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>big old tree being photographed on a sunny day</td>\n",
       "      <td>&lt;sos&gt; and its magnificent trunk larger than li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a old curvy tree in the sun light</td>\n",
       "      <td>&lt;sos&gt; and its magnificent trunk larger than li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a person is taking a picture of a large tree a...</td>\n",
       "      <td>&lt;sos&gt; and its magnificent trunk larger than li...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>large tree with many outstretching branches an...</td>\n",
       "      <td>&lt;sos&gt; we found this tree when we were walking ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a green sign is describing a historic tree and...</td>\n",
       "      <td>&lt;sos&gt; it turns out it is a popular attraction ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8250</th>\n",
       "      <td>a crowd of people in a village square three of...</td>\n",
       "      <td>&lt;sos&gt; we went to organization last summer for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8251</th>\n",
       "      <td>people on a safari truck watching as they expl...</td>\n",
       "      <td>&lt;sos&gt; we got to ride so many different rides  ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8252</th>\n",
       "      <td>children pose for a photograph on steps in a p...</td>\n",
       "      <td>&lt;sos&gt; the kids were having so much fun  &lt;eos&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8253</th>\n",
       "      <td>kids wearing pirate hats are brandishing toy s...</td>\n",
       "      <td>&lt;sos&gt; they loved going around to all the diffe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8254</th>\n",
       "      <td>bronze statue outdoors in foreground large cas...</td>\n",
       "      <td>&lt;sos&gt; we had such a good time and ca n t wait ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8255 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                in_desc  \\\n",
       "0        big old tree being photographed on a sunny day   \n",
       "1                    a old curvy tree in the sun light    \n",
       "2     a person is taking a picture of a large tree a...   \n",
       "3     large tree with many outstretching branches an...   \n",
       "4     a green sign is describing a historic tree and...   \n",
       "...                                                 ...   \n",
       "8250  a crowd of people in a village square three of...   \n",
       "8251  people on a safari truck watching as they expl...   \n",
       "8252  children pose for a photograph on steps in a p...   \n",
       "8253  kids wearing pirate hats are brandishing toy s...   \n",
       "8254  bronze statue outdoors in foreground large cas...   \n",
       "\n",
       "                                              out_story  \n",
       "0     <sos> and its magnificent trunk larger than li...  \n",
       "1     <sos> and its magnificent trunk larger than li...  \n",
       "2     <sos> and its magnificent trunk larger than li...  \n",
       "3     <sos> we found this tree when we were walking ...  \n",
       "4     <sos> it turns out it is a popular attraction ...  \n",
       "...                                                 ...  \n",
       "8250  <sos> we went to organization last summer for ...  \n",
       "8251  <sos> we got to ride so many different rides  ...  \n",
       "8252      <sos> the kids were having so much fun  <eos>  \n",
       "8253  <sos> they loved going around to all the diffe...  \n",
       "8254  <sos> we had such a good time and ca n t wait ...  \n",
       "\n",
       "[8255 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clean_data = clean_data.drop_duplicates(\"in_desc\")\n",
    "clean_data = clean_data.reset_index(drop=True)\n",
    "clean_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectorize the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "0907 DONE\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:23.166348Z",
     "start_time": "2020-09-16T09:48:23.043677Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5654, 3463)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desc_words = set()\n",
    "for desc_paragraph in clean_data.in_desc:\n",
    "    for word in desc_paragraph.split():\n",
    "        if word not in desc_words:\n",
    "            desc_words.add(word)\n",
    "            \n",
    "story_words = set()\n",
    "for story_parapgraph in clean_data.out_story:\n",
    "    for word in story_parapgraph.split():\n",
    "        if word not in story_words:\n",
    "            story_words.add(word)\n",
    "    \n",
    "len(desc_words), len(story_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T09:48:23.433578Z",
     "start_time": "2020-09-16T09:48:23.386677Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples: 5654\n",
      "Number of unique input tokens: 5654\n",
      "Number of unique output tokens: 3463\n",
      "Max sequence length for inputs: 53\n",
      "Max sequence length for outputs: 81\n"
     ]
    }
   ],
   "source": [
    "input_words = sorted(list(desc_words))\n",
    "target_words = sorted(list(story_words))\n",
    "\n",
    "num_encoder_tokens = len(desc_words)\n",
    "num_decoder_tokens = len(story_words)\n",
    "\n",
    "max_encoder_seq_length = max([len(txt.split(\" \")) for txt in clean_data['in_desc']])\n",
    "max_decoder_seq_length = max([len(txt.split(\" \")) for txt in clean_data['out_story']])\n",
    "\n",
    "print('Number of samples:', len(input_words))\n",
    "print('Number of unique input tokens:', num_encoder_tokens)\n",
    "print('Number of unique output tokens:', num_decoder_tokens)\n",
    "print('Max sequence length for inputs:', max_encoder_seq_length)\n",
    "print('Max sequence length for outputs:', max_decoder_seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T10:03:28.557593Z",
     "start_time": "2020-09-16T10:03:28.553630Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8255, 81, 3463)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(clean_data.out_story), max_decoder_seq_length, num_decoder_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-09-16T10:04:04.921972Z",
     "start_time": "2020-09-16T10:04:04.905020Z"
    }
   },
   "outputs": [],
   "source": [
    "encoder_input_data = np.zeros(\n",
    "    (len(clean_data.in_desc), max_encoder_seq_length),\n",
    "    dtype='float32')\n",
    "decoder_input_data = np.zeros(\n",
    "    (len(clean_data.out_story), max_decoder_seq_length),\n",
    "    dtype='float32')\n",
    "decoder_target_data = np.zeros(\n",
    "    (len(clean_data.out_story), max_decoder_seq_length, num_decoder_tokens),\n",
    "    dtype='float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "메모리 뻑남..\n",
    "\n",
    "MemoryError: Unable to allocate 59.8 GiB for an array with shape (30000, 82, 6524) and data type float32\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8255, 53)\n",
      "(8255, 81)\n",
      "(8255, 81, 3463)\n"
     ]
    }
   ],
   "source": [
    "print(encoder_input_data.shape)\n",
    "print(decoder_input_data.shape)\n",
    "print(decoder_target_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**why decoder_target_data.shape is 3d**\n",
    "\n",
    "- 모든 단어에 대하여 이전 단어로부터 다음 단어를 예측하는 소프트맥스 층을 가지기 때문에 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "0908 DONE\n",
    "\n",
    "- 왜 1로 초기화 하는가\n",
    "\n",
    "    - ==> 초기화를 1로 하는 것이 아니라 t=0을 건너뛰는 것임\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_token_index = dict([(word, i) for i, word in enumerate(input_words)])\n",
    "target_token_index = dict([(word, i) for i, word in enumerate(target_words)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, (input_text, target_text) in enumerate(zip(clean_data['in_desc'], clean_data['out_story'])):\n",
    "\n",
    "    # encoder\n",
    "    for t, word in enumerate(input_text.split()):\n",
    "        encoder_input_data[i, t] = input_token_index[word]\n",
    "\n",
    "    # decoder\n",
    "    for t, word in enumerate(target_text.split()):\n",
    "        # decoder_target_data is ahead of decoder_input_data by one timestep\n",
    "        decoder_input_data[i, t] = target_token_index[word]  \n",
    "        if t > 0: \n",
    "            # decoder_target_data will be ahead by one timestep\n",
    "            # and will not include the start character.\n",
    "            decoder_target_data[i, t - 1, target_token_index[word]] = 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8255, 53)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_input_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8255, 81)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[1.000e+00, 9.400e+01, 1.602e+03, ..., 0.000e+00, 0.000e+00,\n",
       "        0.000e+00],\n",
       "       [1.000e+00, 9.400e+01, 1.602e+03, ..., 0.000e+00, 0.000e+00,\n",
       "        0.000e+00],\n",
       "       [1.000e+00, 9.400e+01, 1.602e+03, ..., 0.000e+00, 0.000e+00,\n",
       "        0.000e+00],\n",
       "       ...,\n",
       "       [1.000e+00, 3.047e+03, 1.653e+03, ..., 0.000e+00, 0.000e+00,\n",
       "        0.000e+00],\n",
       "       [1.000e+00, 3.059e+03, 1.788e+03, ..., 0.000e+00, 0.000e+00,\n",
       "        0.000e+00],\n",
       "       [1.000e+00, 3.338e+03, 1.369e+03, ..., 0.000e+00, 0.000e+00,\n",
       "        0.000e+00]], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print (decoder_input_data.shape)\n",
    "decoder_input_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8255, 81, 3463)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]],\n",
       "\n",
       "       [[0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.],\n",
       "        [0., 0., 0., ..., 0., 0., 0.]]], dtype=float32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(decoder_target_data.shape)\n",
    "decoder_target_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "0909 DONE\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build keras encoder-decoder model\n",
    "\n",
    "http://incredible.ai/nlp/2020/02/20/Sequence-To-Sequence-with-Attention/\n",
    "https://docs.chainer.org/en/stable/examples/seq2seq.html\n",
    "\n",
    "https://tykimos.github.io/2018/09/14/ten-minute_introduction_to_sequence-to-sequence_learning_in_Keras/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, LSTM, Dense, Embedding, Bidirectional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64  # Batch size for training.\n",
    "epochs = 50  # Number of epochs to train for.\n",
    "latent_dim = 64  # Latent dimensionality of the encoding space.\n",
    "num_samples = 10000  # Number of samples to train on.\n",
    "embedding_size = 50 # Embedding Dimension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoder_inputs:  (None, None)\n",
      "en_x:  (None, None, 50)\n",
      "encoder_outputs:  (None, 64)\n",
      "state_h:  (None, 64)\n",
      "state_c:  (None, 64)\n"
     ]
    }
   ],
   "source": [
    "# Define an input sequence and process it.\n",
    "encoder_inputs = Input(shape=(None,))\n",
    "print('encoder_inputs: ', encoder_inputs.shape)\n",
    "\n",
    "# English words embedding\n",
    "en_x=  Embedding(num_encoder_tokens, embedding_size)(encoder_inputs)\n",
    "print('en_x: ', en_x.shape)\n",
    "\n",
    "encoder = LSTM(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(en_x)\n",
    "print('encoder_outputs: ', encoder_outputs.shape)\n",
    "print('state_h: ', state_h.shape)\n",
    "print('state_c: ', state_c.shape)\n",
    "\n",
    "# We discard `encoder_outputs` and only keep the states.\n",
    "encoder_states = [state_h, state_c]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "decoder_inputs:  (None, None)\n",
      "decoder_outputs:  (None, None, 3463)\n"
     ]
    }
   ],
   "source": [
    "# Set up the decoder, using `encoder_states` as initial state.\n",
    "decoder_inputs = Input(shape=(None, ))\n",
    "print('decoder_inputs: ', decoder_inputs.shape)\n",
    "\n",
    "# french word embeddings\n",
    "dex=  Embedding(num_decoder_tokens, embedding_size)\n",
    "final_dex = dex(decoder_inputs)\n",
    "\n",
    "# We set up our decoder to return full output sequences, and to return internal states as well. \n",
    "# We don't use the return states in the training model, but we will use them in inference.\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_outputs, _, _ = decoder_lstm(final_dex,\n",
    "                                     initial_state=encoder_states)\n",
    "\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "print('decoder_outputs: ', decoder_outputs.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "\n",
    "0911 DONE\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_2 (InputLayer)            [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding (Embedding)           (None, None, 50)     282700      input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, None, 50)     173150      input_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm (LSTM)                     [(None, 64), (None,  29440       embedding[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "lstm_1 (LSTM)                   [(None, None, 64), ( 29440       embedding_1[0][0]                \n",
      "                                                                 lstm[0][1]                       \n",
      "                                                                 lstm[0][2]                       \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, None, 3463)   225095      lstm_1[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 739,825\n",
      "Trainable params: 739,825\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define the model that will turn\n",
    "# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8255, 53)\n",
      "(8255, 81)\n",
      "(8255, 81, 3463)\n"
     ]
    }
   ],
   "source": [
    "print(encoder_input_data.shape)\n",
    "print(decoder_input_data.shape)\n",
    "print(decoder_target_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run training\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "104/104 [==============================] - 59s 568ms/step - loss: 0.9069 - accuracy: 0.5221 - val_loss: 0.8622 - val_accuracy: 0.8703\n",
      "Epoch 2/50\n",
      "104/104 [==============================] - 57s 548ms/step - loss: 0.8376 - accuracy: 0.8710 - val_loss: 0.8519 - val_accuracy: 0.8723\n",
      "Epoch 3/50\n",
      "104/104 [==============================] - 57s 550ms/step - loss: 0.8169 - accuracy: 0.8714 - val_loss: 0.8479 - val_accuracy: 0.8725\n",
      "Epoch 4/50\n",
      "104/104 [==============================] - 57s 553ms/step - loss: 0.8012 - accuracy: 0.8711 - val_loss: 0.8447 - val_accuracy: 0.8694\n",
      "Epoch 5/50\n",
      "104/104 [==============================] - 58s 553ms/step - loss: 0.7889 - accuracy: 0.8683 - val_loss: 0.8350 - val_accuracy: 0.8701\n",
      "Epoch 6/50\n",
      "104/104 [==============================] - 57s 547ms/step - loss: 0.7701 - accuracy: 0.8664 - val_loss: 0.8286 - val_accuracy: 0.8598\n",
      "Epoch 7/50\n",
      "104/104 [==============================] - 58s 558ms/step - loss: 0.7561 - accuracy: 0.8658 - val_loss: 0.8207 - val_accuracy: 0.8686\n",
      "Epoch 8/50\n",
      "104/104 [==============================] - 57s 551ms/step - loss: 0.7439 - accuracy: 0.8667 - val_loss: 0.8142 - val_accuracy: 0.8697\n",
      "Epoch 9/50\n",
      "104/104 [==============================] - 57s 546ms/step - loss: 0.7321 - accuracy: 0.8698 - val_loss: 0.8102 - val_accuracy: 0.8728\n",
      "Epoch 10/50\n",
      "104/104 [==============================] - 57s 544ms/step - loss: 0.7194 - accuracy: 0.8738 - val_loss: 0.8042 - val_accuracy: 0.8762\n",
      "Epoch 11/50\n",
      "104/104 [==============================] - 57s 552ms/step - loss: 0.7063 - accuracy: 0.8774 - val_loss: 0.7955 - val_accuracy: 0.8793\n",
      "Epoch 12/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.6932 - accuracy: 0.8804 - val_loss: 0.7960 - val_accuracy: 0.8775\n",
      "Epoch 13/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.6808 - accuracy: 0.8818 - val_loss: 0.7847 - val_accuracy: 0.8810\n",
      "Epoch 14/50\n",
      "104/104 [==============================] - 56s 538ms/step - loss: 0.6688 - accuracy: 0.8829 - val_loss: 0.7826 - val_accuracy: 0.8808\n",
      "Epoch 15/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.6574 - accuracy: 0.8838 - val_loss: 0.7789 - val_accuracy: 0.8810\n",
      "Epoch 16/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.6466 - accuracy: 0.8846 - val_loss: 0.7792 - val_accuracy: 0.8800\n",
      "Epoch 17/50\n",
      "104/104 [==============================] - 56s 543ms/step - loss: 0.6364 - accuracy: 0.8852 - val_loss: 0.7780 - val_accuracy: 0.8808\n",
      "Epoch 18/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.6266 - accuracy: 0.8860 - val_loss: 0.7779 - val_accuracy: 0.8815\n",
      "Epoch 19/50\n",
      "104/104 [==============================] - 56s 543ms/step - loss: 0.6172 - accuracy: 0.8868 - val_loss: 0.7786 - val_accuracy: 0.8813\n",
      "Epoch 20/50\n",
      "104/104 [==============================] - 56s 541ms/step - loss: 0.6082 - accuracy: 0.8872 - val_loss: 0.7817 - val_accuracy: 0.8787\n",
      "Epoch 21/50\n",
      "104/104 [==============================] - 56s 542ms/step - loss: 0.5995 - accuracy: 0.8878 - val_loss: 0.7825 - val_accuracy: 0.8787\n",
      "Epoch 22/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.5909 - accuracy: 0.8885 - val_loss: 0.7840 - val_accuracy: 0.8776\n",
      "Epoch 23/50\n",
      "104/104 [==============================] - 56s 541ms/step - loss: 0.5825 - accuracy: 0.8883 - val_loss: 0.7862 - val_accuracy: 0.8800\n",
      "Epoch 24/50\n",
      "104/104 [==============================] - 56s 541ms/step - loss: 0.5743 - accuracy: 0.8869 - val_loss: 0.7885 - val_accuracy: 0.8801\n",
      "Epoch 25/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.5662 - accuracy: 0.8891 - val_loss: 0.7903 - val_accuracy: 0.8690\n",
      "Epoch 26/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.5584 - accuracy: 0.8892 - val_loss: 0.7943 - val_accuracy: 0.8716\n",
      "Epoch 27/50\n",
      "104/104 [==============================] - 57s 553ms/step - loss: 0.5508 - accuracy: 0.8913 - val_loss: 0.7948 - val_accuracy: 0.8714\n",
      "Epoch 28/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.5433 - accuracy: 0.8916 - val_loss: 0.7961 - val_accuracy: 0.8713\n",
      "Epoch 29/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.5360 - accuracy: 0.8929 - val_loss: 0.8001 - val_accuracy: 0.8674\n",
      "Epoch 30/50\n",
      "104/104 [==============================] - 56s 542ms/step - loss: 0.5289 - accuracy: 0.8931 - val_loss: 0.8009 - val_accuracy: 0.8729\n",
      "Epoch 31/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.5217 - accuracy: 0.8933 - val_loss: 0.8047 - val_accuracy: 0.8672\n",
      "Epoch 32/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.5146 - accuracy: 0.8936 - val_loss: 0.8067 - val_accuracy: 0.8583\n",
      "Epoch 33/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.5073 - accuracy: 0.8935 - val_loss: 0.8112 - val_accuracy: 0.8587\n",
      "Epoch 34/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.5003 - accuracy: 0.8936 - val_loss: 0.8137 - val_accuracy: 0.8492\n",
      "Epoch 35/50\n",
      "104/104 [==============================] - 56s 542ms/step - loss: 0.4932 - accuracy: 0.8934 - val_loss: 0.8164 - val_accuracy: 0.8587\n",
      "Epoch 36/50\n",
      "104/104 [==============================] - 56s 542ms/step - loss: 0.4862 - accuracy: 0.8917 - val_loss: 0.8211 - val_accuracy: 0.8563\n",
      "Epoch 37/50\n",
      "104/104 [==============================] - 56s 541ms/step - loss: 0.4791 - accuracy: 0.8910 - val_loss: 0.8242 - val_accuracy: 0.8596\n",
      "Epoch 38/50\n",
      "104/104 [==============================] - 56s 542ms/step - loss: 0.4725 - accuracy: 0.8906 - val_loss: 0.8283 - val_accuracy: 0.8376\n",
      "Epoch 39/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.4656 - accuracy: 0.8883 - val_loss: 0.8288 - val_accuracy: 0.8414\n",
      "Epoch 40/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.4588 - accuracy: 0.8838 - val_loss: 0.8313 - val_accuracy: 0.8195\n",
      "Epoch 41/50\n",
      "104/104 [==============================] - 56s 542ms/step - loss: 0.4522 - accuracy: 0.8822 - val_loss: 0.8349 - val_accuracy: 0.8366\n",
      "Epoch 42/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.4454 - accuracy: 0.8766 - val_loss: 0.8400 - val_accuracy: 0.8180\n",
      "Epoch 43/50\n",
      "104/104 [==============================] - 57s 548ms/step - loss: 0.4388 - accuracy: 0.8679 - val_loss: 0.8419 - val_accuracy: 0.8015\n",
      "Epoch 44/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.4324 - accuracy: 0.8689 - val_loss: 0.8452 - val_accuracy: 0.7947\n",
      "Epoch 45/50\n",
      "104/104 [==============================] - 56s 537ms/step - loss: 0.4260 - accuracy: 0.8668 - val_loss: 0.8508 - val_accuracy: 0.7939\n",
      "Epoch 46/50\n",
      "104/104 [==============================] - 56s 541ms/step - loss: 0.4196 - accuracy: 0.8692 - val_loss: 0.8527 - val_accuracy: 0.7975\n",
      "Epoch 47/50\n",
      "104/104 [==============================] - 56s 540ms/step - loss: 0.4137 - accuracy: 0.8627 - val_loss: 0.8557 - val_accuracy: 0.7715\n",
      "Epoch 48/50\n",
      "104/104 [==============================] - 56s 539ms/step - loss: 0.4077 - accuracy: 0.8676 - val_loss: 0.8623 - val_accuracy: 0.8149\n",
      "Epoch 49/50\n",
      "104/104 [==============================] - 57s 544ms/step - loss: 0.4018 - accuracy: 0.8642 - val_loss: 0.8663 - val_accuracy: 0.7772\n",
      "Epoch 50/50\n",
      "104/104 [==============================] - 56s 538ms/step - loss: 0.3961 - accuracy: 0.8610 - val_loss: 0.8682 - val_accuracy: 0.7707\n"
     ]
    }
   ],
   "source": [
    "with tf.device('/gpu:0'):\n",
    "    model.fit([encoder_input_data, decoder_input_data], decoder_target_data,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "결과가.. 이상.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model\n",
    "model.save('s2s.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Unknown layer: Functional",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-5da83cb64786>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m's2s.h5'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniconda/envs/python_workspace/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py\u001b[0m in \u001b[0;36mload_model\u001b[0;34m(filepath, custom_objects, compile)\u001b[0m\n\u001b[1;32m    182\u001b[0m     if (h5py is not None and (\n\u001b[1;32m    183\u001b[0m         isinstance(filepath, h5py.File) or h5py.is_hdf5(filepath))):\n\u001b[0;32m--> 184\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mhdf5_format\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_model_from_hdf5\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcustom_objects\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcompile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    185\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mversion_info\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m4\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpathlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda/envs/python_workspace/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py\u001b[0m in \u001b[0;36mload_model_from_hdf5\u001b[0;34m(filepath, custom_objects, compile)\u001b[0m\n\u001b[1;32m    176\u001b[0m     \u001b[0mmodel_config\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_config\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    177\u001b[0m     model = model_config_lib.model_from_config(model_config,\n\u001b[0;32m--> 178\u001b[0;31m                                                custom_objects=custom_objects)\n\u001b[0m\u001b[1;32m    179\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    180\u001b[0m     \u001b[0;31m# set weights\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda/envs/python_workspace/lib/python3.7/site-packages/tensorflow/python/keras/saving/model_config.py\u001b[0m in \u001b[0;36mmodel_from_config\u001b[0;34m(config, custom_objects)\u001b[0m\n\u001b[1;32m     53\u001b[0m                     '`Sequential.from_config(config)`?')\n\u001b[1;32m     54\u001b[0m   \u001b[0;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpython\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdeserialize\u001b[0m  \u001b[0;31m# pylint: disable=g-import-not-at-top\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m   \u001b[0;32mreturn\u001b[0m \u001b[0mdeserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcustom_objects\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcustom_objects\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda/envs/python_workspace/lib/python3.7/site-packages/tensorflow/python/keras/layers/serialization.py\u001b[0m in \u001b[0;36mdeserialize\u001b[0;34m(config, custom_objects)\u001b[0m\n\u001b[1;32m    107\u001b[0m       \u001b[0mmodule_objects\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mglobs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m       \u001b[0mcustom_objects\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcustom_objects\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 109\u001b[0;31m       printable_module_name='layer')\n\u001b[0m",
      "\u001b[0;32m~/miniconda/envs/python_workspace/lib/python3.7/site-packages/tensorflow/python/keras/utils/generic_utils.py\u001b[0m in \u001b[0;36mdeserialize_keras_object\u001b[0;34m(identifier, module_objects, custom_objects, printable_module_name)\u001b[0m\n\u001b[1;32m    360\u001b[0m     \u001b[0mconfig\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0midentifier\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    361\u001b[0m     (cls, cls_config) = class_and_config_for_serialized_keras_object(\n\u001b[0;32m--> 362\u001b[0;31m         config, module_objects, custom_objects, printable_module_name)\n\u001b[0m\u001b[1;32m    363\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    364\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'from_config'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda/envs/python_workspace/lib/python3.7/site-packages/tensorflow/python/keras/utils/generic_utils.py\u001b[0m in \u001b[0;36mclass_and_config_for_serialized_keras_object\u001b[0;34m(config, module_objects, custom_objects, printable_module_name)\u001b[0m\n\u001b[1;32m    319\u001b[0m   \u001b[0mcls\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_registered_object\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mclass_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcustom_objects\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodule_objects\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    320\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0mcls\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 321\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Unknown '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mprintable_module_name\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m': '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mclass_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    322\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    323\u001b[0m   \u001b[0mcls_config\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'config'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Unknown layer: Functional"
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "model = keras.models.load_model('s2s.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentence Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"functional_25\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_35 (InputLayer)        [(None, None)]            0         \n",
      "_________________________________________________________________\n",
      "embedding_12 (Embedding)     (None, None, 50)          282700    \n",
      "_________________________________________________________________\n",
      "lstm_16 (LSTM)               [(None, 64), (None, 64),  29440     \n",
      "=================================================================\n",
      "Total params: 312,140\n",
      "Trainable params: 312,140\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "encoder_model = Model(encoder_inputs, encoder_states)\n",
    "encoder_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"embedding_13/embedding_lookup_1/Identity_1:0\", shape=(None, None, 50), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Redefine the decoder model with decoder will be getting below inputs from encoder while in prediction\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
    "\n",
    "final_dex2= dex(decoder_inputs)\n",
    "print(final_dex2)\n",
    "decoder_outputs2, state_h2, state_c2 = decoder_lstm(final_dex2, initial_state=decoder_states_inputs)\n",
    "decoder_states2 = [state_h2, state_c2]\n",
    "decoder_outputs2 = decoder_dense(decoder_outputs2)\n",
    "\n",
    "# sampling model will take encoder states and decoder_input(seed initially) and output the predictions(french word index) We dont care about decoder_states2\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + decoder_states_inputs,\n",
    "    [decoder_outputs2] + decoder_states2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reverse-lookup token index to decode sequences back to something readable.\n",
    "reverse_input_char_index = dict(\n",
    "    (i, char) for char, i in input_token_index.items())\n",
    "reverse_target_char_index = dict(\n",
    "    (i, char) for char, i in target_token_index.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    states_value = encoder_model.predict(input_seq)\n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1,1))\n",
    "    # Populate the first character of target sequence with the start character.\n",
    "    target_seq[0, 0] = target_token_index['<sos>']\n",
    "    # Sampling loop for a batch of sequences\n",
    "    # (to simplify, here we assume a batch of size 1).\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    \n",
    "    while not stop_condition:\n",
    "        output_tokens, h, c = decoder_model.predict(\n",
    "            [target_seq] + states_value)\n",
    "       \n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_char = reverse_target_char_index[sampled_token_index]\n",
    "        decoded_sentence += ' '+sampled_char\n",
    "        # Exit condition: either hit max length or find stop character.\n",
    "        if (sampled_char == '<eos>' or\n",
    "           len(decoded_sentence) > 52):\n",
    "            stop_condition = True\n",
    "        \n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1,1))\n",
    "        target_seq[0, 0] = sampled_token_index\n",
    "        \n",
    "        # Update states\n",
    "        states_value = [h, c]\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "Input sentence: big old tree being photographed on a sunny day\n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: a old curvy tree in the sun light \n",
      "Decoded sentence:  the dog was a perfect way to talk <eos>\n",
      "-\n",
      "Input sentence: a person is taking a picture of a large tree and you can see their shadow\n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: large tree with many outstretching branches and leaves \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a green sign is describing a historic tree and the rules for visiting it \n",
      "Decoded sentence:  here is the first thing we saw the sun tree <eos>\n",
      "-\n",
      "Input sentence: a large tree with roots that look like crocodile tails \n",
      "Decoded sentence:  the family was excited for the field <eos>\n",
      "-\n",
      "Input sentence: huge brown tree roots rose above the ground \n",
      "Decoded sentence:  the dog was a perfect way to talk <eos>\n",
      "-\n",
      "Input sentence: a large tree with many branches coming out\n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a plaque describes an historical tree and advises visitors to not climb it \n",
      "Decoded sentence:  male is so proud of the drummer so he <eos>\n",
      "-\n",
      "Input sentence: a tree has spread its roots high and wide across the ground \n",
      "Decoded sentence:  the family was excited for the field <eos>\n",
      "-\n",
      "Input sentence: some trees that are surrounded by apartment buildings \n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: the tree has very long and dated branches \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a plaque on a stand surround by died leaves off a tree \n",
      "Decoded sentence:  male is so proud of the drummer so he <eos>\n",
      "-\n",
      "Input sentence: a huge tree sits outside with several large roots stemming from the trunk \n",
      "Decoded sentence:  the family was excited for the field <eos>\n",
      "-\n",
      "Input sentence: various parts of the tree are growing out from the ground \n",
      "Decoded sentence:  we were so excited for the candles was going to begin\n",
      "-\n",
      "Input sentence: a sculpture of two people sitting at the base of a light is outside near some foliage \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: an old brick church taken from the ground looking up \n",
      "Decoded sentence:  the kids were excited for the water of the beach <eos>\n",
      "-\n",
      "Input sentence: the colorful flowers are the showcase of the foliage\n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a bunch of floating heads in the middle of a building \n",
      "Decoded sentence:  the family had a great time for a party <eos>\n",
      "-\n",
      "Input sentence: a delicious looking pastry with gravy cauliflower and roasted carrots \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: large grey statue of two people sitting outside \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a castle type structure with lots of windows \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a close up shot of pink flowers with bright backlighting from the sun \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: many model heads hang from strings in a large hall \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a plate of biscuits and gravy with cauliflower and carrots \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a statue of two ancient figures sitting under a light\n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: the outside of a large brown gothic building \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a bumble bee in a pink flower with blue sky in the background \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: several white heads hanging from the ceiling in a building \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a breaded food sits next to potatoes and vegetables on a plate \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: two people wearing t shirts and one with an umbrella in front of a large monument \n",
      "Decoded sentence:  the family took a great time to the park <eos>\n",
      "-\n",
      "Input sentence: a large group of people standing on the steps of the organization organization \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: two men sit down in front of the camera \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a large group of people are surrounded near downtown location location \n",
      "Decoded sentence:  the kids were excited for the fireworks <eos>\n",
      "-\n",
      "Input sentence: the fireworks are shooting off in the sky \n",
      "Decoded sentence:  the fireworks were very excited for the fireworks <eos>\n",
      "-\n",
      "Input sentence: two men are standing outside on a lawn and the location location is in the distance \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: several people sitting and walking around the location monument \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: two men sitting on the ground and smiling \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: the sun is going down and the people are all sitting \n",
      "Decoded sentence:  the kids were excited for the fireworks <eos>\n",
      "-\n",
      "Input sentence: the fireworks display was vivid purple and silhouetted everyone \n",
      "Decoded sentence:  the fireworks were very excited in the fireworks <eos>\n",
      "-\n",
      "Input sentence: two guys at a public event on bright day \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: the location location is full of people on this day\n",
      "Decoded sentence:  the family took a great time to remember <eos>\n",
      "-\n",
      "Input sentence: two young men sit on the ground as others in the background do the same \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: people gathered at night in location location with a monument in the background \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: several bright multicolored fireworks bursting in the night sky \n",
      "Decoded sentence:  the fireworks were beautiful <eos>\n",
      "-\n",
      "Input sentence: a black and white picture of a wooden bridge over a body of water \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: construction on the bridge is not finished yet\n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: water flows over the side of a dam in an image with a description of the scene below \n",
      "Decoded sentence:  the fireworks were very excited for the fireworks <eos>\n",
      "-\n",
      "Input sentence: a group of men doing work in on a ditch \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: the man is pouring liquid into the gallon \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a black and white photograph showing a wooden trestle bridge \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: black and white image of metal crane and truck \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a general view looking north of water going over the spillway of the gerber dam for the very first time \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a group of men working together to dig a large ditch \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a man pours water into a mysterious contraption \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: photo of a flume over a river with debris at the base \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: an old picture shows a crane and building\n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a black and white photograph from female shows a view of the gerber dam \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a bunch of men working together in a trench \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a man pouring some liquid into a machine \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: this kid is not shy about having picture taken \n",
      "Decoded sentence:  the family took a great time <eos>\n",
      "-\n",
      "Input sentence: chandelier lit in an open area inside an open room \n",
      "Decoded sentence:  we were so excited for the candles was going to begin\n",
      "-\n",
      "Input sentence: the large group of people are observing an object \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a woman in a headscarf is holding her hand out to camera\n",
      "Decoded sentence:  the family took a great time to remember <eos>\n",
      "-\n",
      "Input sentence: a very intricate picture of people fighting on a brightly designed wall \n",
      "Decoded sentence:  the dog was a perfect way to talk <eos>\n",
      "-\n",
      "Input sentence: a man with a professional camera is taking a closeup shot of a boy who is crouched over and looking into the camera while another boy looks on \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: lights are liten up as they are taken inside the church\n",
      "Decoded sentence:  the dog was a perfect way to talk <eos>\n",
      "-\n",
      "Input sentence: a group of people with cameras poses on the ground for a group photo \n",
      "Decoded sentence:  the kids were excited for the fireworks <eos>\n",
      "-\n",
      "Input sentence: a woman wearing a headscarf holding her hand up \n",
      "Decoded sentence:  the family took a great time to remember <eos>\n",
      "-\n",
      "Input sentence: a painting of a whole bunch of people on a wall \n",
      "Decoded sentence:  the dog was a perfect company <eos>\n",
      "-\n",
      "Input sentence: a man is taking a picture of a boy awkwardly close to his face\n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a view of a church sanctuary from the upper balcony \n",
      "Decoded sentence:  the dog was a perfect way to talk <eos>\n",
      "-\n",
      "Input sentence: several people sitting and standing around a large concrete object \n",
      "Decoded sentence:  the family took a great time to remember <eos>\n",
      "-\n",
      "Input sentence: a woman in a headscarf attempts to block the camera with her hand \n",
      "Decoded sentence:  the family took a great time to remember <eos>\n",
      "-\n",
      "Input sentence: a very beautiful mosaic of a church wall\n",
      "Decoded sentence:  i m glad we went to the streets <eos>\n",
      "-\n",
      "Input sentence: a picture showing the base of very large trees \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a young deer in a wooded area of a woods\n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a chubby squirrel stands on the ground eating something \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a tan and black striped snake sets atop some twigs \n",
      "Decoded sentence:  i m glad we went to the streets <eos>\n",
      "-\n",
      "Input sentence: a huge waterfall cascades off of a cliff \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: hundreds of years old trees standing tall in the forest \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: a deer stands in an opening in the forest \n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: a squirrel sitting on sticks eating a nut \n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: a snake is looking at something angrily and hungrily\n",
      "Decoded sentence:  i m glad we went to the streets <eos>\n",
      "-\n",
      "Input sentence: water spills over a fall between rocky mountain crevices \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: large tree trunks are shown with smaller forest trees in the back ground \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: brown deer standing in a wooded area outside \n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: chipmunk standing on a brown ground with sun shining on it \n",
      "Decoded sentence:  the family had a great time together <eos>\n",
      "-\n",
      "Input sentence: a yellow and black striped snake is slithering on the ground \n",
      "Decoded sentence:  i m glad we went to the streets <eos>\n",
      "-\n",
      "Input sentence: water rushes over a waterfall in the mountains \n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a woman takes her child on his first picnic\n",
      "Decoded sentence:  the kids were excited to celebrate the th of july <eos>\n",
      "-\n",
      "Input sentence: a young child holding a cup and wearing an american flag image\n",
      "Decoded sentence:  the kids were excited to the beach for the first view\n",
      "-\n",
      "Input sentence: three people playing and laughing at a bouncing castle\n",
      "Decoded sentence:  the family was excited to celebrate the first day <eos>\n",
      "-\n",
      "Input sentence: a girl jumps inside of a moon bounce \n",
      "Decoded sentence:  the dog was a perfect way to talk <eos>\n",
      "-\n",
      "Input sentence: a family picture of a man woman and small child outside \n",
      "Decoded sentence:  the family took a great time <eos>\n",
      "-\n",
      "Input sentence: smiling woman sitting with child in red white and blue top \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: the baby is holding the cup drinking out of it\n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: the mom is playing with her kids in the bounce house \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: kids playing around in the bouncy house together \n",
      "Decoded sentence:  we had a great time at the park <eos>\n",
      "-\n",
      "Input sentence: parents posing for a picture with their young baby boy \n",
      "Decoded sentence:  the family took a great time to the park <eos>\n"
     ]
    }
   ],
   "source": [
    "for seq_index in range(100):\n",
    "    # Take one sequence (part of the training set)\n",
    "    # for trying out decoding.\n",
    "    input_seq = encoder_input_data[seq_index: seq_index + 1]\n",
    "    decoded_sentence = decode_sequence(input_seq)\n",
    "    print('-')\n",
    "    print('Input sentence:', clean_data['in_desc'][seq_index])\n",
    "    print('Decoded sentence:', decoded_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "- epoch 를 줄여야 어느정도 학습 가능\n",
    "- input에 대응하는 output pair 가 너무 많아서 성능이 안좋은게아닐까? => input : output 1대1 대응만 남기기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1.0,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16.0,
    "lenType": 16.0,
    "lenVar": 40.0
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}