Chapter09/ch9_nb3_train_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"border: 1px solid #e7692c; border-left: 15px solid #e7692c; padding: 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c\">Tip.</strong> <a style=\"color: #000000;\" href=\"https://nbviewer.jupyter.org/github/PacktPublishing/Hands-On-Computer-Vision-with-TensorFlow-2/blob/master/Chapter09/ch9_nb3_train_model.ipynb\" title=\"View with Jupyter Online\">Click here to view this notebook on <code>nbviewer.jupyter.org</code></a>. \n",
    "    <br/>These notebooks are better read there, as Github default viewer ignores some of the formatting and interactive content.\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T22:38:46.581850Z",
     "start_time": "2019-04-12T22:38:46.573807Z"
    }
   },
   "source": [
    "<table style=\"font-size: 1em; padding: 0; margin: 0;\">\n",
    "    <tr style=\"vertical-align: top; padding: 0; margin: 0;background-color: #ffffff\">\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; padding-right: 15px;\">\n",
    "    <p style=\"background: #363636; color:#ffffff; text-align:justify; padding: 10px 25px;\">\n",
    "        <strong style=\"font-size: 1.0em;\"><span style=\"font-size: 1.2em;\"><span style=\"color: #e7692c;\">Hands-on</span> Computer Vision with TensorFlow 2</span><br/>by <em>Eliot Andres</em> & <em>Benjamin Planche</em> (Packt Pub.)</strong><br/><br/>\n",
    "        <strong>> Chapter 9: Performance and running on mobile</strong><br/>\n",
    "    </p>\n",
    "\n",
    "<h1 style=\"width: 100%; text-align: left; padding: 0px 25px;\"><small style=\"color: #e7692c;\">Notebook 3:</small><br/>Training a model and converting it for mobile devices</h1>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #363636; text-align:justify; padding: 0 10px;\">\n",
    "    In this chapter, we covered how to convert and run a model on mobile.\n",
    "<br/><br/>\n",
    "    This notebooks trains a model to recognize face expressions and converts it to CoreML, TFLite and TensorFlow.js\n",
    "</p>\n",
    "<br/>\n",
    "\n",
    "<p style=\"border-left: 15px solid #363636; text-align:justify; padding: 0 10px;\">\n",
    "    <strong> Requirements </strong>\n",
    "<br/><br/>\n",
    "    To run this notebook, you need to download the <a href=\"https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data\">FER dataset</a> and extract it. When done, change the `BASE_PATH` variable to point to the dataset folder.\n",
    "</p>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #e7692c; padding: 0 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c;\">Tip.</strong> The notebooks shared on this git repository illustrate some notions from the book \"<em><strong>Hands-on Computer Vision with TensorFlow 2</strong></em>\" written by Eliot Andres and Benjamin Planche, published by Packt. If you enjoyed the insights shared here, <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Learn more about the book!\"><strong>please consider acquiring the book!</strong></a>\n",
    "<br/><br/>\n",
    "The book provides further guidance for those eager to learn about computer vision and to harness the power of TensorFlow 2 and Keras to build efficient recognition systems for object detection, segmentation, video processing, smartphone applications, and more.</p>\n",
    "        </td>\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; width: 280px;\">\n",
    "    <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Learn more about the book!\" target=\"_blank\">\n",
    "        <img src=\"../banner_images/book_cover.png\" width=280>\n",
    "    </a>\n",
    "    <p style=\"background: #e7692c; color:#ffffff; padding: 10px; text-align:justify;\"><strong>Leverage deep learning to create powerful image processing apps with TensorFlow 2 and Keras. <br/></strong>Get the book for more insights!</p>\n",
    "    <ul style=\"height: 32px; white-space: nowrap; text-align: center; margin: 0px; padding: 0px; padding-top: 10px;\">\n",
    "    <li style=\"display: block;height: 100%;float: left;vertical-align: middle;margin: 0 25px 10px;padding: 0px;\">\n",
    "        <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Get the book on Amazon (paperback or Kindle version)!\" target=\"_blank\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_amazon.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    <li style=\"display: inline-block;height: 100%;vertical-align: middle;float: right;margin: -5px 25px 10px;padding: 0px;\">\n",
    "        <a href=\"https://www.packtpub.com/application-development/hands-computer-vision-tensorflow-2\" title=\"Get your Packt book (paperback, PDF, ePUB, or MOBI version)!\" target=\"_blank\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_packt.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    </ul>\n",
    "        </td>\n",
    "        </tr>\n",
    "        </table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:32:47.307854Z",
     "start_time": "2019-05-26T14:32:46.208824Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.0-dev20190428'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, TensorBoard\n",
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "from tensorflow.keras.applications.mobilenet import MobileNet\n",
    "from tensorflow.keras.layers import Input\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import cv2\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set constants and parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:32:47.631723Z",
     "start_time": "2019-05-26T14:32:47.624298Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Download data here: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data\n",
    "DATASET_PATH = '../data/fer2013/fer2013.csv'\n",
    "\n",
    "IMAGE_SIZE = (48, 48)\n",
    "INPUT_SHAPE = IMAGE_SIZE + (1,)\n",
    "EMOTIONS = [\"angry\", \"disgust\", \"scared\",\n",
    "            \"happy\", \"sad\", \"surprised\", \"neutral\"]\n",
    "CALLBACK_PATIENCE = 50\n",
    "BATCH_SIZE = 32\n",
    "NUM_EPOCHS = 50\n",
    "VALIDATION_SPLIT = .2\n",
    "NUM_CLASSES = len(EMOTIONS)\n",
    "L2_REGULARIZATION = 0.01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:32:51.437882Z",
     "start_time": "2019-05-26T14:32:49.688747Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 48, 48, 1)]       0         \n",
      "_________________________________________________________________\n",
      "conv1_pad (ZeroPadding2D)    (None, 49, 49, 1)         0         \n",
      "_________________________________________________________________\n",
      "conv1 (Conv2D)               (None, 24, 24, 32)        288       \n",
      "_________________________________________________________________\n",
      "conv1_bn (BatchNormalization (None, 24, 24, 32)        128       \n",
      "_________________________________________________________________\n",
      "conv1_relu (ReLU)            (None, 24, 24, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv_dw_1 (DepthwiseConv2D)  (None, 24, 24, 32)        288       \n",
      "_________________________________________________________________\n",
      "conv_dw_1_bn (BatchNormaliza (None, 24, 24, 32)        128       \n",
      "_________________________________________________________________\n",
      "conv_dw_1_relu (ReLU)        (None, 24, 24, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv_pw_1 (Conv2D)           (None, 24, 24, 64)        2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_1_bn (BatchNormaliza (None, 24, 24, 64)        256       \n",
      "_________________________________________________________________\n",
      "conv_pw_1_relu (ReLU)        (None, 24, 24, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv_pad_2 (ZeroPadding2D)   (None, 25, 25, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv_dw_2 (DepthwiseConv2D)  (None, 12, 12, 64)        576       \n",
      "_________________________________________________________________\n",
      "conv_dw_2_bn (BatchNormaliza (None, 12, 12, 64)        256       \n",
      "_________________________________________________________________\n",
      "conv_dw_2_relu (ReLU)        (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv_pw_2 (Conv2D)           (None, 12, 12, 128)       8192      \n",
      "_________________________________________________________________\n",
      "conv_pw_2_bn (BatchNormaliza (None, 12, 12, 128)       512       \n",
      "_________________________________________________________________\n",
      "conv_pw_2_relu (ReLU)        (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv_dw_3 (DepthwiseConv2D)  (None, 12, 12, 128)       1152      \n",
      "_________________________________________________________________\n",
      "conv_dw_3_bn (BatchNormaliza (None, 12, 12, 128)       512       \n",
      "_________________________________________________________________\n",
      "conv_dw_3_relu (ReLU)        (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv_pw_3 (Conv2D)           (None, 12, 12, 128)       16384     \n",
      "_________________________________________________________________\n",
      "conv_pw_3_bn (BatchNormaliza (None, 12, 12, 128)       512       \n",
      "_________________________________________________________________\n",
      "conv_pw_3_relu (ReLU)        (None, 12, 12, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv_pad_4 (ZeroPadding2D)   (None, 13, 13, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv_dw_4 (DepthwiseConv2D)  (None, 6, 6, 128)         1152      \n",
      "_________________________________________________________________\n",
      "conv_dw_4_bn (BatchNormaliza (None, 6, 6, 128)         512       \n",
      "_________________________________________________________________\n",
      "conv_dw_4_relu (ReLU)        (None, 6, 6, 128)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_4 (Conv2D)           (None, 6, 6, 256)         32768     \n",
      "_________________________________________________________________\n",
      "conv_pw_4_bn (BatchNormaliza (None, 6, 6, 256)         1024      \n",
      "_________________________________________________________________\n",
      "conv_pw_4_relu (ReLU)        (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_5 (DepthwiseConv2D)  (None, 6, 6, 256)         2304      \n",
      "_________________________________________________________________\n",
      "conv_dw_5_bn (BatchNormaliza (None, 6, 6, 256)         1024      \n",
      "_________________________________________________________________\n",
      "conv_dw_5_relu (ReLU)        (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_5 (Conv2D)           (None, 6, 6, 256)         65536     \n",
      "_________________________________________________________________\n",
      "conv_pw_5_bn (BatchNormaliza (None, 6, 6, 256)         1024      \n",
      "_________________________________________________________________\n",
      "conv_pw_5_relu (ReLU)        (None, 6, 6, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv_pad_6 (ZeroPadding2D)   (None, 7, 7, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_6 (DepthwiseConv2D)  (None, 3, 3, 256)         2304      \n",
      "_________________________________________________________________\n",
      "conv_dw_6_bn (BatchNormaliza (None, 3, 3, 256)         1024      \n",
      "_________________________________________________________________\n",
      "conv_dw_6_relu (ReLU)        (None, 3, 3, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_6 (Conv2D)           (None, 3, 3, 512)         131072    \n",
      "_________________________________________________________________\n",
      "conv_pw_6_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_6_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_7 (DepthwiseConv2D)  (None, 3, 3, 512)         4608      \n",
      "_________________________________________________________________\n",
      "conv_dw_7_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_dw_7_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_7 (Conv2D)           (None, 3, 3, 512)         262144    \n",
      "_________________________________________________________________\n",
      "conv_pw_7_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_7_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_8 (DepthwiseConv2D)  (None, 3, 3, 512)         4608      \n",
      "_________________________________________________________________\n",
      "conv_dw_8_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_dw_8_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_8 (Conv2D)           (None, 3, 3, 512)         262144    \n",
      "_________________________________________________________________\n",
      "conv_pw_8_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_8_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_9 (DepthwiseConv2D)  (None, 3, 3, 512)         4608      \n",
      "_________________________________________________________________\n",
      "conv_dw_9_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_dw_9_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_9 (Conv2D)           (None, 3, 3, 512)         262144    \n",
      "_________________________________________________________________\n",
      "conv_pw_9_bn (BatchNormaliza (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_9_relu (ReLU)        (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_10 (DepthwiseConv2D) (None, 3, 3, 512)         4608      \n",
      "_________________________________________________________________\n",
      "conv_dw_10_bn (BatchNormaliz (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_dw_10_relu (ReLU)       (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_10 (Conv2D)          (None, 3, 3, 512)         262144    \n",
      "_________________________________________________________________\n",
      "conv_pw_10_bn (BatchNormaliz (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_10_relu (ReLU)       (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_11 (DepthwiseConv2D) (None, 3, 3, 512)         4608      \n",
      "_________________________________________________________________\n",
      "conv_dw_11_bn (BatchNormaliz (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_dw_11_relu (ReLU)       (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_11 (Conv2D)          (None, 3, 3, 512)         262144    \n",
      "_________________________________________________________________\n",
      "conv_pw_11_bn (BatchNormaliz (None, 3, 3, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_pw_11_relu (ReLU)       (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pad_12 (ZeroPadding2D)  (None, 4, 4, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_dw_12 (DepthwiseConv2D) (None, 1, 1, 512)         4608      \n",
      "_________________________________________________________________\n",
      "conv_dw_12_bn (BatchNormaliz (None, 1, 1, 512)         2048      \n",
      "_________________________________________________________________\n",
      "conv_dw_12_relu (ReLU)       (None, 1, 1, 512)         0         \n",
      "_________________________________________________________________\n",
      "conv_pw_12 (Conv2D)          (None, 1, 1, 1024)        524288    \n",
      "_________________________________________________________________\n",
      "conv_pw_12_bn (BatchNormaliz (None, 1, 1, 1024)        4096      \n",
      "_________________________________________________________________\n",
      "conv_pw_12_relu (ReLU)       (None, 1, 1, 1024)        0         \n",
      "_________________________________________________________________\n",
      "conv_dw_13 (DepthwiseConv2D) (None, 1, 1, 1024)        9216      \n",
      "_________________________________________________________________\n",
      "conv_dw_13_bn (BatchNormaliz (None, 1, 1, 1024)        4096      \n",
      "_________________________________________________________________\n",
      "conv_dw_13_relu (ReLU)       (None, 1, 1, 1024)        0         \n",
      "_________________________________________________________________\n",
      "conv_pw_13 (Conv2D)          (None, 1, 1, 1024)        1048576   \n",
      "_________________________________________________________________\n",
      "conv_pw_13_bn (BatchNormaliz (None, 1, 1, 1024)        4096      \n",
      "_________________________________________________________________\n",
      "conv_pw_13_relu (ReLU)       (None, 1, 1, 1024)        0         \n",
      "_________________________________________________________________\n",
      "reshape (Reshape)            (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 7)                 7175      \n",
      "=================================================================\n",
      "Total params: 3,235,463\n",
      "Trainable params: 3,213,575\n",
      "Non-trainable params: 21,888\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "input_tensor = Input(shape=INPUT_SHAPE)\n",
    "model = MobileNet(input_tensor=input_tensor, alpha=1.0,\n",
    "                    include_top=False, weights=None)\n",
    "\n",
    "output = tf.keras.layers.Reshape((1024,))(model.output)\n",
    "output = tf.keras.layers.Dense(7, activation='softmax')(output)\n",
    "model = tf.keras.Model(model.input, output)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:33:26.174937Z",
     "start_time": "2019-05-26T14:33:08.169194Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def load_fer2013():\n",
    "    data = pd.read_csv(DATASET_PATH)\n",
    "    pixels = data['pixels'].tolist()\n",
    "    width, height = 48, 48\n",
    "    faces = []\n",
    "    for pixel_sequence in pixels:\n",
    "        face = [int(pixel) for pixel in pixel_sequence.split(' ')]\n",
    "        face = np.asarray(face).reshape(width, height)\n",
    "        face = cv2.resize(face.astype('uint8'), IMAGE_SIZE)\n",
    "        faces.append(face.astype('float32'))\n",
    "    faces = np.asarray(faces)\n",
    "    faces = np.expand_dims(faces, -1)\n",
    "    emotions = pd.get_dummies(data['emotion']).values\n",
    "    return faces, emotions\n",
    "\n",
    "def preprocess_input(x, v2=True):\n",
    "    x = x.astype('float32')\n",
    "    x = x / 255.0\n",
    "    x = x - 0.5\n",
    "    x = x * 2.0\n",
    "    return x\n",
    "\n",
    "data_generator = ImageDataGenerator(\n",
    "    featurewise_center=False,\n",
    "    featurewise_std_normalization=False,\n",
    "    rotation_range=10,\n",
    "    width_shift_range=0.1,\n",
    "    height_shift_range=0.1,\n",
    "    zoom_range=.1,\n",
    "    horizontal_flip=True)\n",
    "\n",
    "# loading dataset\n",
    "faces, emotions = load_fer2013()\n",
    "faces = preprocess_input(faces)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:29:36.416888Z",
     "start_time": "2019-05-26T14:11:17.714535Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "898/897 [==============================] - 37s 41ms/step - loss: 1.8936 - accuracy: 0.2319 - val_loss: 1.9043 - val_accuracy: 0.2530\n",
      "Epoch 2/50\n",
      "898/897 [==============================] - 21s 24ms/step - loss: 1.8200 - accuracy: 0.2649 - val_loss: 1.7767 - val_accuracy: 0.2768\n",
      "Epoch 3/50\n",
      "898/897 [==============================] - 21s 24ms/step - loss: 1.7820 - accuracy: 0.2988 - val_loss: 1.8422 - val_accuracy: 0.3034\n",
      "Epoch 4/50\n",
      "898/897 [==============================] - 21s 24ms/step - loss: 1.7192 - accuracy: 0.3265 - val_loss: 1.9029 - val_accuracy: 0.2336\n",
      "Epoch 5/50\n",
      "898/897 [==============================] - 21s 24ms/step - loss: 1.6862 - accuracy: 0.3394 - val_loss: 1.6631 - val_accuracy: 0.3416\n",
      "Epoch 6/50\n",
      "898/897 [==============================] - 21s 24ms/step - loss: 1.6721 - accuracy: 0.3502 - val_loss: 1.6184 - val_accuracy: 0.3767\n",
      "Epoch 7/50\n",
      "898/897 [==============================] - 21s 24ms/step - loss: 1.5909 - accuracy: 0.3841 - val_loss: 1.6010 - val_accuracy: 0.3938\n",
      "Epoch 8/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.5573 - accuracy: 0.3980 - val_loss: 1.4983 - val_accuracy: 0.4234\n",
      "Epoch 9/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.5219 - accuracy: 0.4153 - val_loss: 1.5643 - val_accuracy: 0.3969\n",
      "Epoch 10/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.4756 - accuracy: 0.4304 - val_loss: 1.3906 - val_accuracy: 0.4673\n",
      "Epoch 11/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.4226 - accuracy: 0.4564 - val_loss: 1.3008 - val_accuracy: 0.4996\n",
      "Epoch 12/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.3726 - accuracy: 0.4771 - val_loss: 1.3063 - val_accuracy: 0.4990\n",
      "Epoch 13/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.3459 - accuracy: 0.4913 - val_loss: 1.4578 - val_accuracy: 0.4411\n",
      "Epoch 14/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.3270 - accuracy: 0.4988 - val_loss: 1.4089 - val_accuracy: 0.4536\n",
      "Epoch 15/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.2788 - accuracy: 0.5149 - val_loss: 1.2692 - val_accuracy: 0.5210\n",
      "Epoch 16/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.2438 - accuracy: 0.5274 - val_loss: 1.1983 - val_accuracy: 0.5507\n",
      "Epoch 17/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.2278 - accuracy: 0.5352 - val_loss: 1.2243 - val_accuracy: 0.5352\n",
      "Epoch 18/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.2062 - accuracy: 0.5438 - val_loss: 1.1963 - val_accuracy: 0.5534\n",
      "Epoch 19/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1937 - accuracy: 0.5510 - val_loss: 1.2240 - val_accuracy: 0.5391\n",
      "Epoch 20/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1781 - accuracy: 0.5520 - val_loss: 1.1659 - val_accuracy: 0.5614\n",
      "Epoch 21/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1829 - accuracy: 0.5578 - val_loss: 1.1816 - val_accuracy: 0.5613\n",
      "Epoch 22/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1432 - accuracy: 0.5717 - val_loss: 1.1201 - val_accuracy: 0.5730\n",
      "Epoch 23/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1655 - accuracy: 0.5610 - val_loss: 1.1035 - val_accuracy: 0.5800\n",
      "Epoch 24/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1140 - accuracy: 0.5810 - val_loss: 1.0714 - val_accuracy: 0.5965\n",
      "Epoch 25/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.1056 - accuracy: 0.5801 - val_loss: 1.0932 - val_accuracy: 0.5828\n",
      "Epoch 26/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0887 - accuracy: 0.5920 - val_loss: 1.0800 - val_accuracy: 0.5946\n",
      "Epoch 27/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0832 - accuracy: 0.5963 - val_loss: 1.1731 - val_accuracy: 0.5577\n",
      "Epoch 28/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0701 - accuracy: 0.5991 - val_loss: 1.0919 - val_accuracy: 0.5949\n",
      "Epoch 29/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0593 - accuracy: 0.6032 - val_loss: 1.0946 - val_accuracy: 0.5879\n",
      "Epoch 30/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0639 - accuracy: 0.6013 - val_loss: 1.1607 - val_accuracy: 0.5564\n",
      "Epoch 31/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0415 - accuracy: 0.6091 - val_loss: 1.0763 - val_accuracy: 0.5918\n",
      "Epoch 32/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0785 - accuracy: 0.5972 - val_loss: 1.0458 - val_accuracy: 0.6039\n",
      "Epoch 33/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0312 - accuracy: 0.6121 - val_loss: 1.0386 - val_accuracy: 0.6053\n",
      "Epoch 34/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0193 - accuracy: 0.6193 - val_loss: 1.0594 - val_accuracy: 0.5991\n",
      "Epoch 35/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0303 - accuracy: 0.6160 - val_loss: 1.0548 - val_accuracy: 0.6057\n",
      "Epoch 36/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0091 - accuracy: 0.6236 - val_loss: 1.0402 - val_accuracy: 0.6115\n",
      "Epoch 37/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9995 - accuracy: 0.6245 - val_loss: 1.0429 - val_accuracy: 0.6050\n",
      "Epoch 38/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 1.0011 - accuracy: 0.6283 - val_loss: 1.0650 - val_accuracy: 0.5956\n",
      "Epoch 39/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9881 - accuracy: 0.6300 - val_loss: 1.0668 - val_accuracy: 0.6048\n",
      "Epoch 40/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9707 - accuracy: 0.6382 - val_loss: 1.0477 - val_accuracy: 0.6076\n",
      "Epoch 41/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9635 - accuracy: 0.6387 - val_loss: 1.0177 - val_accuracy: 0.6234\n",
      "Epoch 42/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9649 - accuracy: 0.6399 - val_loss: 1.0189 - val_accuracy: 0.6227\n",
      "Epoch 43/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9596 - accuracy: 0.6407 - val_loss: 1.0014 - val_accuracy: 0.6271\n",
      "Epoch 44/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9500 - accuracy: 0.6433 - val_loss: 1.0129 - val_accuracy: 0.6284\n",
      "Epoch 45/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9321 - accuracy: 0.6527 - val_loss: 1.0122 - val_accuracy: 0.6193\n",
      "Epoch 46/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9544 - accuracy: 0.6438 - val_loss: 1.0115 - val_accuracy: 0.6241\n",
      "Epoch 47/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9465 - accuracy: 0.6453 - val_loss: 1.0028 - val_accuracy: 0.6220\n",
      "Epoch 48/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9173 - accuracy: 0.6576 - val_loss: 1.0142 - val_accuracy: 0.6262\n",
      "Epoch 49/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9120 - accuracy: 0.6603 - val_loss: 1.0119 - val_accuracy: 0.6296\n",
      "Epoch 50/50\n",
      "898/897 [==============================] - 22s 24ms/step - loss: 0.9025 - accuracy: 0.6608 - val_loss: 1.0203 - val_accuracy: 0.6308\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fa79e5e3eb8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regularization = tf.keras.regularizers.l2(L2_REGULARIZATION)\n",
    "\n",
    "early_stop = EarlyStopping('val_loss', patience=CALLBACK_PATIENCE)\n",
    "reduce_lr = ReduceLROnPlateau(\n",
    "    'val_loss', factor=0.1, patience=int(CALLBACK_PATIENCE/4), verbose=1)\n",
    "tensorboard = TensorBoard('./logs')\n",
    "callbacks = [early_stop, reduce_lr, tensorboard]\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "xtrain, xtest, ytrain, ytest = train_test_split(faces, emotions, test_size=0.2)\n",
    "\n",
    "model.fit_generator(data_generator.flow(xtrain, ytrain, BATCH_SIZE),\n",
    "                    steps_per_epoch=len(xtrain) / BATCH_SIZE,\n",
    "                    epochs=NUM_EPOCHS, verbose=1, callbacks=callbacks,\n",
    "                    validation_data=(xtest, ytest))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to CoreML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:34:41.620005Z",
     "start_time": "2019-05-26T14:34:27.856889Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0526 16:34:32.567987 140384533387008 tf_logging.py:161] Export includes no default signature!\n",
      "W0526 16:34:35.210698 140384533387008 tf_logging.py:161] Export includes no default signature!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loading the TF graph...\n",
      "Graph Loaded.\n",
      "Now finding ops in the TF graph that can be dropped for inference\n",
      "Collecting all the 'Const' ops from the graph, by running it....\n",
      "Done.\n",
      "Now starting translation to CoreML graph.\n",
      "Automatic shape interpretation succeeded for input blob input_1:0\n",
      "1/377: Analysing op name: dense/bias ( type:  Const )\n",
      "2/377: Analysing op name: dense/BiasAdd/ReadVariableOp ( type:  Identity )\n",
      "3/377: Analysing op name: dense/kernel ( type:  Const )\n",
      "4/377: Analysing op name: dense/MatMul/ReadVariableOp ( type:  Identity )\n",
      "5/377: Analysing op name: reshape/Reshape/shape/1 ( type:  Const )\n",
      "6/377: Analysing op name: reshape/strided_slice/stack_2 ( type:  Const )\n",
      "7/377: Analysing op name: reshape/strided_slice/stack_1 ( type:  Const )\n",
      "8/377: Analysing op name: reshape/strided_slice/stack ( type:  Const )\n",
      "9/377: Analysing op name: conv_pw_13_bn/moving_variance ( type:  Const )\n",
      "10/377: Analysing op name: conv_pw_13_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "11/377: Analysing op name: conv_pw_13_bn/moving_mean ( type:  Const )\n",
      "12/377: Analysing op name: conv_pw_13_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "13/377: Analysing op name: conv_pw_13_bn/beta ( type:  Const )\n",
      "14/377: Analysing op name: conv_pw_13_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "15/377: Analysing op name: conv_pw_13_bn/gamma ( type:  Const )\n",
      "16/377: Analysing op name: conv_pw_13_bn/ReadVariableOp ( type:  Identity )\n",
      "17/377: Analysing op name: conv_pw_13/kernel ( type:  Const )\n",
      "18/377: Analysing op name: conv_pw_13/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "19/377: Analysing op name: conv_dw_13_bn/moving_variance ( type:  Const )\n",
      "20/377: Analysing op name: conv_dw_13_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "21/377: Analysing op name: conv_dw_13_bn/moving_mean ( type:  Const )\n",
      "22/377: Analysing op name: conv_dw_13_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "23/377: Analysing op name: conv_dw_13_bn/beta ( type:  Const )\n",
      "24/377: Analysing op name: conv_dw_13_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "25/377: Analysing op name: conv_dw_13_bn/gamma ( type:  Const )\n",
      "26/377: Analysing op name: conv_dw_13_bn/ReadVariableOp ( type:  Identity )\n",
      "27/377: Analysing op name: conv_dw_13/depthwise_kernel ( type:  Const )\n",
      "28/377: Analysing op name: conv_dw_13/depthwise/ReadVariableOp ( type:  Identity )\n",
      "29/377: Analysing op name: conv_pw_12_bn/moving_variance ( type:  Const )\n",
      "30/377: Analysing op name: conv_pw_12_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "31/377: Analysing op name: conv_pw_12_bn/moving_mean ( type:  Const )\n",
      "32/377: Analysing op name: conv_pw_12_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "33/377: Analysing op name: conv_pw_12_bn/beta ( type:  Const )\n",
      "34/377: Analysing op name: conv_pw_12_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "35/377: Analysing op name: conv_pw_12_bn/gamma ( type:  Const )\n",
      "36/377: Analysing op name: conv_pw_12_bn/ReadVariableOp ( type:  Identity )\n",
      "37/377: Analysing op name: conv_pw_12/kernel ( type:  Const )\n",
      "38/377: Analysing op name: conv_pw_12/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "39/377: Analysing op name: conv_dw_12_bn/moving_variance ( type:  Const )\n",
      "40/377: Analysing op name: conv_dw_12_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "41/377: Analysing op name: conv_dw_12_bn/moving_mean ( type:  Const )\n",
      "42/377: Analysing op name: conv_dw_12_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "43/377: Analysing op name: conv_dw_12_bn/beta ( type:  Const )\n",
      "44/377: Analysing op name: conv_dw_12_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "45/377: Analysing op name: conv_dw_12_bn/gamma ( type:  Const )\n",
      "46/377: Analysing op name: conv_dw_12_bn/ReadVariableOp ( type:  Identity )\n",
      "47/377: Analysing op name: conv_dw_12/depthwise_kernel ( type:  Const )\n",
      "48/377: Analysing op name: conv_dw_12/depthwise/ReadVariableOp ( type:  Identity )\n",
      "49/377: Analysing op name: conv_pad_12/Pad/paddings ( type:  Const )\n",
      "50/377: Analysing op name: conv_pw_11_bn/moving_variance ( type:  Const )\n",
      "51/377: Analysing op name: conv_pw_11_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "52/377: Analysing op name: conv_pw_11_bn/moving_mean ( type:  Const )\n",
      "53/377: Analysing op name: conv_pw_11_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "54/377: Analysing op name: conv_pw_11_bn/beta ( type:  Const )\n",
      "55/377: Analysing op name: conv_pw_11_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "56/377: Analysing op name: conv_pw_11_bn/gamma ( type:  Const )\n",
      "57/377: Analysing op name: conv_pw_11_bn/ReadVariableOp ( type:  Identity )\n",
      "58/377: Analysing op name: conv_pw_11/kernel ( type:  Const )\n",
      "59/377: Analysing op name: conv_pw_11/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "60/377: Analysing op name: conv_dw_11_bn/moving_variance ( type:  Const )\n",
      "61/377: Analysing op name: conv_dw_11_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "62/377: Analysing op name: conv_dw_11_bn/moving_mean ( type:  Const )\n",
      "63/377: Analysing op name: conv_dw_11_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "64/377: Analysing op name: conv_dw_11_bn/beta ( type:  Const )\n",
      "65/377: Analysing op name: conv_dw_11_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "66/377: Analysing op name: conv_dw_11_bn/gamma ( type:  Const )\n",
      "67/377: Analysing op name: conv_dw_11_bn/ReadVariableOp ( type:  Identity )\n",
      "68/377: Analysing op name: conv_dw_11/depthwise_kernel ( type:  Const )\n",
      "69/377: Analysing op name: conv_dw_11/depthwise/ReadVariableOp ( type:  Identity )\n",
      "70/377: Analysing op name: conv_pw_10_bn/moving_variance ( type:  Const )\n",
      "71/377: Analysing op name: conv_pw_10_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "72/377: Analysing op name: conv_pw_10_bn/moving_mean ( type:  Const )\n",
      "73/377: Analysing op name: conv_pw_10_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "74/377: Analysing op name: conv_pw_10_bn/beta ( type:  Const )\n",
      "75/377: Analysing op name: conv_pw_10_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "76/377: Analysing op name: conv_pw_10_bn/gamma ( type:  Const )\n",
      "77/377: Analysing op name: conv_pw_10_bn/ReadVariableOp ( type:  Identity )\n",
      "78/377: Analysing op name: conv_pw_10/kernel ( type:  Const )\n",
      "79/377: Analysing op name: conv_pw_10/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "80/377: Analysing op name: conv_dw_10_bn/moving_variance ( type:  Const )\n",
      "81/377: Analysing op name: conv_dw_10_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "82/377: Analysing op name: conv_dw_10_bn/moving_mean ( type:  Const )\n",
      "83/377: Analysing op name: conv_dw_10_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "84/377: Analysing op name: conv_dw_10_bn/beta ( type:  Const )\n",
      "85/377: Analysing op name: conv_dw_10_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "86/377: Analysing op name: conv_dw_10_bn/gamma ( type:  Const )\n",
      "87/377: Analysing op name: conv_dw_10_bn/ReadVariableOp ( type:  Identity )\n",
      "88/377: Analysing op name: conv_dw_10/depthwise_kernel ( type:  Const )\n",
      "89/377: Analysing op name: conv_dw_10/depthwise/ReadVariableOp ( type:  Identity )\n",
      "90/377: Analysing op name: conv_pw_9_bn/moving_variance ( type:  Const )\n",
      "91/377: Analysing op name: conv_pw_9_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "92/377: Analysing op name: conv_pw_9_bn/moving_mean ( type:  Const )\n",
      "93/377: Analysing op name: conv_pw_9_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "94/377: Analysing op name: conv_pw_9_bn/beta ( type:  Const )\n",
      "95/377: Analysing op name: conv_pw_9_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "96/377: Analysing op name: conv_pw_9_bn/gamma ( type:  Const )\n",
      "97/377: Analysing op name: conv_pw_9_bn/ReadVariableOp ( type:  Identity )\n",
      "98/377: Analysing op name: conv_pw_9/kernel ( type:  Const )\n",
      "99/377: Analysing op name: conv_pw_9/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "100/377: Analysing op name: conv_dw_9_bn/moving_variance ( type:  Const )\n",
      "101/377: Analysing op name: conv_dw_9_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "102/377: Analysing op name: conv_dw_9_bn/moving_mean ( type:  Const )\n",
      "103/377: Analysing op name: conv_dw_9_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "104/377: Analysing op name: conv_dw_9_bn/beta ( type:  Const )\n",
      "105/377: Analysing op name: conv_dw_9_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "106/377: Analysing op name: conv_dw_9_bn/gamma ( type:  Const )\n",
      "107/377: Analysing op name: conv_dw_9_bn/ReadVariableOp ( type:  Identity )\n",
      "108/377: Analysing op name: conv_dw_9/depthwise_kernel ( type:  Const )\n",
      "109/377: Analysing op name: conv_dw_9/depthwise/ReadVariableOp ( type:  Identity )\n",
      "110/377: Analysing op name: conv_pw_8_bn/moving_variance ( type:  Const )\n",
      "111/377: Analysing op name: conv_pw_8_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "112/377: Analysing op name: conv_pw_8_bn/moving_mean ( type:  Const )\n",
      "113/377: Analysing op name: conv_pw_8_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "114/377: Analysing op name: conv_pw_8_bn/beta ( type:  Const )\n",
      "115/377: Analysing op name: conv_pw_8_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "116/377: Analysing op name: conv_pw_8_bn/gamma ( type:  Const )\n",
      "117/377: Analysing op name: conv_pw_8_bn/ReadVariableOp ( type:  Identity )\n",
      "118/377: Analysing op name: conv_pw_8/kernel ( type:  Const )\n",
      "119/377: Analysing op name: conv_pw_8/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "120/377: Analysing op name: conv_dw_8_bn/moving_variance ( type:  Const )\n",
      "121/377: Analysing op name: conv_dw_8_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "122/377: Analysing op name: conv_dw_8_bn/moving_mean ( type:  Const )\n",
      "123/377: Analysing op name: conv_dw_8_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "124/377: Analysing op name: conv_dw_8_bn/beta ( type:  Const )\n",
      "125/377: Analysing op name: conv_dw_8_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "126/377: Analysing op name: conv_dw_8_bn/gamma ( type:  Const )\n",
      "127/377: Analysing op name: conv_dw_8_bn/ReadVariableOp ( type:  Identity )\n",
      "128/377: Analysing op name: conv_dw_8/depthwise_kernel ( type:  Const )\n",
      "129/377: Analysing op name: conv_dw_8/depthwise/ReadVariableOp ( type:  Identity )\n",
      "130/377: Analysing op name: conv_pw_7_bn/moving_variance ( type:  Const )\n",
      "131/377: Analysing op name: conv_pw_7_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "132/377: Analysing op name: conv_pw_7_bn/moving_mean ( type:  Const )\n",
      "133/377: Analysing op name: conv_pw_7_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "134/377: Analysing op name: conv_pw_7_bn/beta ( type:  Const )\n",
      "135/377: Analysing op name: conv_pw_7_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "136/377: Analysing op name: conv_pw_7_bn/gamma ( type:  Const )\n",
      "137/377: Analysing op name: conv_pw_7_bn/ReadVariableOp ( type:  Identity )\n",
      "138/377: Analysing op name: conv_pw_7/kernel ( type:  Const )\n",
      "139/377: Analysing op name: conv_pw_7/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "140/377: Analysing op name: conv_dw_7_bn/moving_variance ( type:  Const )\n",
      "141/377: Analysing op name: conv_dw_7_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "142/377: Analysing op name: conv_dw_7_bn/moving_mean ( type:  Const )\n",
      "143/377: Analysing op name: conv_dw_7_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "144/377: Analysing op name: conv_dw_7_bn/beta ( type:  Const )\n",
      "145/377: Analysing op name: conv_dw_7_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "146/377: Analysing op name: conv_dw_7_bn/gamma ( type:  Const )\n",
      "147/377: Analysing op name: conv_dw_7_bn/ReadVariableOp ( type:  Identity )\n",
      "148/377: Analysing op name: conv_dw_7/depthwise_kernel ( type:  Const )\n",
      "149/377: Analysing op name: conv_dw_7/depthwise/ReadVariableOp ( type:  Identity )\n",
      "150/377: Analysing op name: conv_pw_6_bn/moving_variance ( type:  Const )\n",
      "151/377: Analysing op name: conv_pw_6_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "152/377: Analysing op name: conv_pw_6_bn/moving_mean ( type:  Const )\n",
      "153/377: Analysing op name: conv_pw_6_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "154/377: Analysing op name: conv_pw_6_bn/beta ( type:  Const )\n",
      "155/377: Analysing op name: conv_pw_6_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "156/377: Analysing op name: conv_pw_6_bn/gamma ( type:  Const )\n",
      "157/377: Analysing op name: conv_pw_6_bn/ReadVariableOp ( type:  Identity )\n",
      "158/377: Analysing op name: conv_pw_6/kernel ( type:  Const )\n",
      "159/377: Analysing op name: conv_pw_6/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "160/377: Analysing op name: conv_dw_6_bn/moving_variance ( type:  Const )\n",
      "161/377: Analysing op name: conv_dw_6_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "162/377: Analysing op name: conv_dw_6_bn/moving_mean ( type:  Const )\n",
      "163/377: Analysing op name: conv_dw_6_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "164/377: Analysing op name: conv_dw_6_bn/beta ( type:  Const )\n",
      "165/377: Analysing op name: conv_dw_6_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "166/377: Analysing op name: conv_dw_6_bn/gamma ( type:  Const )\n",
      "167/377: Analysing op name: conv_dw_6_bn/ReadVariableOp ( type:  Identity )\n",
      "168/377: Analysing op name: conv_dw_6/depthwise_kernel ( type:  Const )\n",
      "169/377: Analysing op name: conv_dw_6/depthwise/ReadVariableOp ( type:  Identity )\n",
      "170/377: Analysing op name: conv_pad_6/Pad/paddings ( type:  Const )\n",
      "171/377: Analysing op name: conv_pw_5_bn/moving_variance ( type:  Const )\n",
      "172/377: Analysing op name: conv_pw_5_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "173/377: Analysing op name: conv_pw_5_bn/moving_mean ( type:  Const )\n",
      "174/377: Analysing op name: conv_pw_5_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "175/377: Analysing op name: conv_pw_5_bn/beta ( type:  Const )\n",
      "176/377: Analysing op name: conv_pw_5_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "177/377: Analysing op name: conv_pw_5_bn/gamma ( type:  Const )\n",
      "178/377: Analysing op name: conv_pw_5_bn/ReadVariableOp ( type:  Identity )\n",
      "179/377: Analysing op name: conv_pw_5/kernel ( type:  Const )\n",
      "180/377: Analysing op name: conv_pw_5/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "181/377: Analysing op name: conv_dw_5_bn/moving_variance ( type:  Const )\n",
      "182/377: Analysing op name: conv_dw_5_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "183/377: Analysing op name: conv_dw_5_bn/moving_mean ( type:  Const )\n",
      "184/377: Analysing op name: conv_dw_5_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "185/377: Analysing op name: conv_dw_5_bn/beta ( type:  Const )\n",
      "186/377: Analysing op name: conv_dw_5_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "187/377: Analysing op name: conv_dw_5_bn/gamma ( type:  Const )\n",
      "188/377: Analysing op name: conv_dw_5_bn/ReadVariableOp ( type:  Identity )\n",
      "189/377: Analysing op name: conv_dw_5/depthwise_kernel ( type:  Const )\n",
      "190/377: Analysing op name: conv_dw_5/depthwise/ReadVariableOp ( type:  Identity )\n",
      "191/377: Analysing op name: conv_pw_4_bn/moving_variance ( type:  Const )\n",
      "192/377: Analysing op name: conv_pw_4_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "193/377: Analysing op name: conv_pw_4_bn/moving_mean ( type:  Const )\n",
      "194/377: Analysing op name: conv_pw_4_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "195/377: Analysing op name: conv_pw_4_bn/beta ( type:  Const )\n",
      "196/377: Analysing op name: conv_pw_4_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "197/377: Analysing op name: conv_pw_4_bn/gamma ( type:  Const )\n",
      "198/377: Analysing op name: conv_pw_4_bn/ReadVariableOp ( type:  Identity )\n",
      "199/377: Analysing op name: conv_pw_4/kernel ( type:  Const )\n",
      "200/377: Analysing op name: conv_pw_4/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "201/377: Analysing op name: conv_dw_4_bn/moving_variance ( type:  Const )\n",
      "202/377: Analysing op name: conv_dw_4_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "203/377: Analysing op name: conv_dw_4_bn/moving_mean ( type:  Const )\n",
      "204/377: Analysing op name: conv_dw_4_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "205/377: Analysing op name: conv_dw_4_bn/beta ( type:  Const )\n",
      "206/377: Analysing op name: conv_dw_4_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "207/377: Analysing op name: conv_dw_4_bn/gamma ( type:  Const )\n",
      "208/377: Analysing op name: conv_dw_4_bn/ReadVariableOp ( type:  Identity )\n",
      "209/377: Analysing op name: conv_dw_4/depthwise_kernel ( type:  Const )\n",
      "210/377: Analysing op name: conv_dw_4/depthwise/ReadVariableOp ( type:  Identity )\n",
      "211/377: Analysing op name: conv_pad_4/Pad/paddings ( type:  Const )\n",
      "212/377: Analysing op name: conv_pw_3_bn/moving_variance ( type:  Const )\n",
      "213/377: Analysing op name: conv_pw_3_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "214/377: Analysing op name: conv_pw_3_bn/moving_mean ( type:  Const )\n",
      "215/377: Analysing op name: conv_pw_3_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "216/377: Analysing op name: conv_pw_3_bn/beta ( type:  Const )\n",
      "217/377: Analysing op name: conv_pw_3_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "218/377: Analysing op name: conv_pw_3_bn/gamma ( type:  Const )\n",
      "219/377: Analysing op name: conv_pw_3_bn/ReadVariableOp ( type:  Identity )\n",
      "220/377: Analysing op name: conv_pw_3/kernel ( type:  Const )\n",
      "221/377: Analysing op name: conv_pw_3/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "222/377: Analysing op name: conv_dw_3_bn/moving_variance ( type:  Const )\n",
      "223/377: Analysing op name: conv_dw_3_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "224/377: Analysing op name: conv_dw_3_bn/moving_mean ( type:  Const )\n",
      "225/377: Analysing op name: conv_dw_3_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "226/377: Analysing op name: conv_dw_3_bn/beta ( type:  Const )\n",
      "227/377: Analysing op name: conv_dw_3_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "228/377: Analysing op name: conv_dw_3_bn/gamma ( type:  Const )\n",
      "229/377: Analysing op name: conv_dw_3_bn/ReadVariableOp ( type:  Identity )\n",
      "230/377: Analysing op name: conv_dw_3/depthwise_kernel ( type:  Const )\n",
      "231/377: Analysing op name: conv_dw_3/depthwise/ReadVariableOp ( type:  Identity )\n",
      "232/377: Analysing op name: conv_pw_2_bn/moving_variance ( type:  Const )\n",
      "233/377: Analysing op name: conv_pw_2_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "234/377: Analysing op name: conv_pw_2_bn/moving_mean ( type:  Const )\n",
      "235/377: Analysing op name: conv_pw_2_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "236/377: Analysing op name: conv_pw_2_bn/beta ( type:  Const )\n",
      "237/377: Analysing op name: conv_pw_2_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "238/377: Analysing op name: conv_pw_2_bn/gamma ( type:  Const )\n",
      "239/377: Analysing op name: conv_pw_2_bn/ReadVariableOp ( type:  Identity )\n",
      "240/377: Analysing op name: conv_pw_2/kernel ( type:  Const )\n",
      "241/377: Analysing op name: conv_pw_2/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "242/377: Analysing op name: conv_dw_2_bn/moving_variance ( type:  Const )\n",
      "243/377: Analysing op name: conv_dw_2_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "244/377: Analysing op name: conv_dw_2_bn/moving_mean ( type:  Const )\n",
      "245/377: Analysing op name: conv_dw_2_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "246/377: Analysing op name: conv_dw_2_bn/beta ( type:  Const )\n",
      "247/377: Analysing op name: conv_dw_2_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "248/377: Analysing op name: conv_dw_2_bn/gamma ( type:  Const )\n",
      "249/377: Analysing op name: conv_dw_2_bn/ReadVariableOp ( type:  Identity )\n",
      "250/377: Analysing op name: conv_dw_2/depthwise_kernel ( type:  Const )\n",
      "251/377: Analysing op name: conv_dw_2/depthwise/ReadVariableOp ( type:  Identity )\n",
      "252/377: Analysing op name: conv_pad_2/Pad/paddings ( type:  Const )\n",
      "253/377: Analysing op name: conv_pw_1_bn/moving_variance ( type:  Const )\n",
      "254/377: Analysing op name: conv_pw_1_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "255/377: Analysing op name: conv_pw_1_bn/moving_mean ( type:  Const )\n",
      "256/377: Analysing op name: conv_pw_1_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "257/377: Analysing op name: conv_pw_1_bn/beta ( type:  Const )\n",
      "258/377: Analysing op name: conv_pw_1_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "259/377: Analysing op name: conv_pw_1_bn/gamma ( type:  Const )\n",
      "260/377: Analysing op name: conv_pw_1_bn/ReadVariableOp ( type:  Identity )\n",
      "261/377: Analysing op name: conv_pw_1/kernel ( type:  Const )\n",
      "262/377: Analysing op name: conv_pw_1/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "263/377: Analysing op name: conv_dw_1_bn/moving_variance ( type:  Const )\n",
      "264/377: Analysing op name: conv_dw_1_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "265/377: Analysing op name: conv_dw_1_bn/moving_mean ( type:  Const )\n",
      "266/377: Analysing op name: conv_dw_1_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "267/377: Analysing op name: conv_dw_1_bn/beta ( type:  Const )\n",
      "268/377: Analysing op name: conv_dw_1_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "269/377: Analysing op name: conv_dw_1_bn/gamma ( type:  Const )\n",
      "270/377: Analysing op name: conv_dw_1_bn/ReadVariableOp ( type:  Identity )\n",
      "271/377: Analysing op name: conv_dw_1/depthwise_kernel ( type:  Const )\n",
      "272/377: Analysing op name: conv_dw_1/depthwise/ReadVariableOp ( type:  Identity )\n",
      "273/377: Analysing op name: conv1_bn/moving_variance ( type:  Const )\n",
      "274/377: Analysing op name: conv1_bn/FusedBatchNorm/ReadVariableOp_1 ( type:  Identity )\n",
      "275/377: Analysing op name: conv1_bn/moving_mean ( type:  Const )\n",
      "276/377: Analysing op name: conv1_bn/FusedBatchNorm/ReadVariableOp ( type:  Identity )\n",
      "277/377: Analysing op name: conv1_bn/beta ( type:  Const )\n",
      "278/377: Analysing op name: conv1_bn/ReadVariableOp_1 ( type:  Identity )\n",
      "279/377: Analysing op name: conv1_bn/gamma ( type:  Const )\n",
      "280/377: Analysing op name: conv1_bn/ReadVariableOp ( type:  Identity )\n",
      "281/377: Analysing op name: conv1/kernel ( type:  Const )\n",
      "282/377: Analysing op name: conv1/Conv2D/ReadVariableOp ( type:  Identity )\n",
      "283/377: Analysing op name: conv1_pad/Pad/paddings ( type:  Const )\n",
      "284/377: Analysing op name: input_1 ( type:  Placeholder )\n",
      "Skipping name of placeholder\n",
      "285/377: Analysing op name: conv1_pad/Pad ( type:  Pad )\n",
      "286/377: Analysing op name: conv1/Conv2D ( type:  Conv2D )\n",
      "287/377: Analysing op name: conv1_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "288/377: Analysing op name: conv1_relu/Relu6 ( type:  Relu6 )\n",
      "289/377: Analysing op name: conv_dw_1/depthwise ( type:  DepthwiseConv2dNative )\n",
      "290/377: Analysing op name: conv_dw_1_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "291/377: Analysing op name: conv_dw_1_relu/Relu6 ( type:  Relu6 )\n",
      "292/377: Analysing op name: conv_pw_1/Conv2D ( type:  Conv2D )\n",
      "293/377: Analysing op name: conv_pw_1_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "294/377: Analysing op name: conv_pw_1_relu/Relu6 ( type:  Relu6 )\n",
      "295/377: Analysing op name: conv_pad_2/Pad ( type:  Pad )\n",
      "296/377: Analysing op name: conv_dw_2/depthwise ( type:  DepthwiseConv2dNative )\n",
      "297/377: Analysing op name: conv_dw_2_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "298/377: Analysing op name: conv_dw_2_relu/Relu6 ( type:  Relu6 )\n",
      "299/377: Analysing op name: conv_pw_2/Conv2D ( type:  Conv2D )\n",
      "300/377: Analysing op name: conv_pw_2_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "301/377: Analysing op name: conv_pw_2_relu/Relu6 ( type:  Relu6 )\n",
      "302/377: Analysing op name: conv_dw_3/depthwise ( type:  DepthwiseConv2dNative )\n",
      "303/377: Analysing op name: conv_dw_3_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "304/377: Analysing op name: conv_dw_3_relu/Relu6 ( type:  Relu6 )\n",
      "305/377: Analysing op name: conv_pw_3/Conv2D ( type:  Conv2D )\n",
      "306/377: Analysing op name: conv_pw_3_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "307/377: Analysing op name: conv_pw_3_relu/Relu6 ( type:  Relu6 )\n",
      "308/377: Analysing op name: conv_pad_4/Pad ( type:  Pad )\n",
      "309/377: Analysing op name: conv_dw_4/depthwise ( type:  DepthwiseConv2dNative )\n",
      "310/377: Analysing op name: conv_dw_4_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "311/377: Analysing op name: conv_dw_4_relu/Relu6 ( type:  Relu6 )\n",
      "312/377: Analysing op name: conv_pw_4/Conv2D ( type:  Conv2D )\n",
      "313/377: Analysing op name: conv_pw_4_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "314/377: Analysing op name: conv_pw_4_relu/Relu6 ( type:  Relu6 )\n",
      "315/377: Analysing op name: conv_dw_5/depthwise ( type:  DepthwiseConv2dNative )\n",
      "316/377: Analysing op name: conv_dw_5_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "317/377: Analysing op name: conv_dw_5_relu/Relu6 ( type:  Relu6 )\n",
      "318/377: Analysing op name: conv_pw_5/Conv2D ( type:  Conv2D )\n",
      "319/377: Analysing op name: conv_pw_5_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "320/377: Analysing op name: conv_pw_5_relu/Relu6 ( type:  Relu6 )\n",
      "321/377: Analysing op name: conv_pad_6/Pad ( type:  Pad )\n",
      "322/377: Analysing op name: conv_dw_6/depthwise ( type:  DepthwiseConv2dNative )\n",
      "323/377: Analysing op name: conv_dw_6_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "324/377: Analysing op name: conv_dw_6_relu/Relu6 ( type:  Relu6 )\n",
      "325/377: Analysing op name: conv_pw_6/Conv2D ( type:  Conv2D )\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "326/377: Analysing op name: conv_pw_6_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "327/377: Analysing op name: conv_pw_6_relu/Relu6 ( type:  Relu6 )\n",
      "328/377: Analysing op name: conv_dw_7/depthwise ( type:  DepthwiseConv2dNative )\n",
      "329/377: Analysing op name: conv_dw_7_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "330/377: Analysing op name: conv_dw_7_relu/Relu6 ( type:  Relu6 )\n",
      "331/377: Analysing op name: conv_pw_7/Conv2D ( type:  Conv2D )\n",
      "332/377: Analysing op name: conv_pw_7_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "333/377: Analysing op name: conv_pw_7_relu/Relu6 ( type:  Relu6 )\n",
      "334/377: Analysing op name: conv_dw_8/depthwise ( type:  DepthwiseConv2dNative )\n",
      "335/377: Analysing op name: conv_dw_8_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "336/377: Analysing op name: conv_dw_8_relu/Relu6 ( type:  Relu6 )\n",
      "337/377: Analysing op name: conv_pw_8/Conv2D ( type:  Conv2D )\n",
      "338/377: Analysing op name: conv_pw_8_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "339/377: Analysing op name: conv_pw_8_relu/Relu6 ( type:  Relu6 )\n",
      "340/377: Analysing op name: conv_dw_9/depthwise ( type:  DepthwiseConv2dNative )\n",
      "341/377: Analysing op name: conv_dw_9_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "342/377: Analysing op name: conv_dw_9_relu/Relu6 ( type:  Relu6 )\n",
      "343/377: Analysing op name: conv_pw_9/Conv2D ( type:  Conv2D )\n",
      "344/377: Analysing op name: conv_pw_9_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "345/377: Analysing op name: conv_pw_9_relu/Relu6 ( type:  Relu6 )\n",
      "346/377: Analysing op name: conv_dw_10/depthwise ( type:  DepthwiseConv2dNative )\n",
      "347/377: Analysing op name: conv_dw_10_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "348/377: Analysing op name: conv_dw_10_relu/Relu6 ( type:  Relu6 )\n",
      "349/377: Analysing op name: conv_pw_10/Conv2D ( type:  Conv2D )\n",
      "350/377: Analysing op name: conv_pw_10_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "351/377: Analysing op name: conv_pw_10_relu/Relu6 ( type:  Relu6 )\n",
      "352/377: Analysing op name: conv_dw_11/depthwise ( type:  DepthwiseConv2dNative )\n",
      "353/377: Analysing op name: conv_dw_11_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "354/377: Analysing op name: conv_dw_11_relu/Relu6 ( type:  Relu6 )\n",
      "355/377: Analysing op name: conv_pw_11/Conv2D ( type:  Conv2D )\n",
      "356/377: Analysing op name: conv_pw_11_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "357/377: Analysing op name: conv_pw_11_relu/Relu6 ( type:  Relu6 )\n",
      "358/377: Analysing op name: conv_pad_12/Pad ( type:  Pad )\n",
      "359/377: Analysing op name: conv_dw_12/depthwise ( type:  DepthwiseConv2dNative )\n",
      "360/377: Analysing op name: conv_dw_12_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "361/377: Analysing op name: conv_dw_12_relu/Relu6 ( type:  Relu6 )\n",
      "362/377: Analysing op name: conv_pw_12/Conv2D ( type:  Conv2D )\n",
      "363/377: Analysing op name: conv_pw_12_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "364/377: Analysing op name: conv_pw_12_relu/Relu6 ( type:  Relu6 )\n",
      "365/377: Analysing op name: conv_dw_13/depthwise ( type:  DepthwiseConv2dNative )\n",
      "366/377: Analysing op name: conv_dw_13_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "367/377: Analysing op name: conv_dw_13_relu/Relu6 ( type:  Relu6 )\n",
      "368/377: Analysing op name: conv_pw_13/Conv2D ( type:  Conv2D )\n",
      "369/377: Analysing op name: conv_pw_13_bn/FusedBatchNorm ( type:  FusedBatchNorm )\n",
      "370/377: Analysing op name: conv_pw_13_relu/Relu6 ( type:  Relu6 )\n",
      "371/377: Analysing op name: reshape/Shape ( type:  Shape )\n",
      "372/377: Analysing op name: reshape/strided_slice ( type:  StridedSlice )\n",
      "373/377: Analysing op name: reshape/Reshape/shape ( type:  Pack )\n",
      "374/377: Analysing op name: reshape/Reshape ( type:  Reshape )\n",
      "375/377: Analysing op name: dense/MatMul ( type:  MatMul )\n",
      "376/377: Analysing op name: dense/BiasAdd ( type:  BiasAdd )\n",
      "377/377: Analysing op name: dense/Softmax ( type:  Softmax )\n",
      "Translation to CoreML spec completed. Now compiling and saving the CoreML model.\n",
      "\n",
      " Core ML model generated. Saved at location: mobilenet.mlmodel \n",
      "\n",
      "Core ML input(s): \n",
      " [name: \"input_1__0\"\n",
      "type {\n",
      "  imageType {\n",
      "    width: 48\n",
      "    height: 48\n",
      "    colorSpace: GRAYSCALE\n",
      "  }\n",
      "}\n",
      "]\n",
      "Core ML output(s): \n",
      " [name: \"dense__Softmax__0\"\n",
      "type {\n",
      "  dictionaryType {\n",
      "    stringKeyType {\n",
      "    }\n",
      "  }\n",
      "}\n",
      ", name: \"classLabel\"\n",
      "type {\n",
      "  stringType {\n",
      "  }\n",
      "}\n",
      "]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "input {\n",
       "  name: \"input_1__0\"\n",
       "  type {\n",
       "    imageType {\n",
       "      width: 48\n",
       "      height: 48\n",
       "      colorSpace: GRAYSCALE\n",
       "    }\n",
       "  }\n",
       "}\n",
       "output {\n",
       "  name: \"dense__Softmax__0\"\n",
       "  type {\n",
       "    dictionaryType {\n",
       "      stringKeyType {\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "}\n",
       "output {\n",
       "  name: \"classLabel\"\n",
       "  type {\n",
       "    stringType {\n",
       "    }\n",
       "  }\n",
       "}\n",
       "predictedFeatureName: \"classLabel\"\n",
       "predictedProbabilitiesName: \"dense__Softmax__0\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tfcoreml as tf_converter\n",
    "from tensorflow.python.keras.callbacks import TensorBoard\n",
    "from tensorflow.python.saved_model import tag_constants\n",
    "from tensorflow.python.tools import freeze_graph\n",
    "input_saved_model_dir = \"./saved_model\"\n",
    "\n",
    "tf.compat.v1.keras.experimental.export_saved_model(\n",
    "    model, saved_model_path=input_saved_model_dir, serving_only=False)\n",
    "\n",
    "\n",
    "output_node_name = 'dense/Softmax'\n",
    "input_binary = False\n",
    "input_saver_def_path = False\n",
    "restore_op_name = None\n",
    "filename_tensor_name = None\n",
    "clear_devices = True\n",
    "input_meta_graph = False\n",
    "checkpoint_path = None\n",
    "input_graph_filename = None\n",
    "saved_model_tags = tag_constants.SERVING\n",
    "\n",
    "\n",
    "freeze_graph.freeze_graph(input_graph_filename, input_saver_def_path,\n",
    "                          input_binary, checkpoint_path, output_node_name,\n",
    "                          restore_op_name, filename_tensor_name,\n",
    "                          'frozen_model.pb', clear_devices, \"\", \"\", \"\",\n",
    "                          input_meta_graph, input_saved_model_dir,\n",
    "                          saved_model_tags)\n",
    "\n",
    "\n",
    "tf_converter.convert('frozen_model.pb',\n",
    "                     'mobilenet.mlmodel',\n",
    "                     class_labels=EMOTIONS,\n",
    "                     image_input_names=['input_1:0'],\n",
    "                     output_feature_names=[output_node_name + ':0'],\n",
    "                     red_bias=-1,\n",
    "                     green_bias=-1,\n",
    "                     blue_bias=-1,\n",
    "                     image_scale=1/127.5,\n",
    "                     is_bgr=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to TFLite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:34:55.510418Z",
     "start_time": "2019-05-26T14:34:52.149382Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12784696"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "converter = tf.lite.TFLiteConverter.from_keras_model(model)\n",
    "## Or from a SavedModel\n",
    "# converter = tf.lite.TFLiteConverter('./saved_model')\n",
    "\n",
    "tflite_model = converter.convert()\n",
    "open(\"result.tflite\", \"wb\").write(tflite_model)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert to TFJS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-05-26T14:35:48.595974Z",
     "start_time": "2019-05-26T14:35:48.591625Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert in the current environment\n",
    "import sys\n",
    "!tensorflowjs_converter --input_format=tf_saved_model saved_model my-tfjs --output_format tfjs_graph_model"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}